WO2026002170A1 - 模型和/或功能评估方法、装置、设备、产品及存储介质 - Google Patents

模型和/或功能评估方法、装置、设备、产品及存储介质

Info

Publication number
WO2026002170A1
WO2026002170A1 PCT/CN2025/104096 CN2025104096W WO2026002170A1 WO 2026002170 A1 WO2026002170 A1 WO 2026002170A1 CN 2025104096 W CN2025104096 W CN 2025104096W WO 2026002170 A1 WO2026002170 A1 WO 2026002170A1
Authority
WO
WIPO (PCT)
Prior art keywords
model
time
evaluation
information
functionality
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/104096
Other languages
English (en)
French (fr)
Inventor
陈晶晶
曹昱华
张晓然
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
China Mobile Communications Group Co Ltd
Research Institute of China Mobile Communication Co Ltd
Original Assignee
China Mobile Communications Group Co Ltd
Research Institute of China Mobile Communication Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by China Mobile Communications Group Co Ltd, Research Institute of China Mobile Communication Co Ltd filed Critical China Mobile Communications Group Co Ltd
Publication of WO2026002170A1 publication Critical patent/WO2026002170A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/04—Network management architectures or arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H04L41/0813—Configuration setting characterised by the conditions triggering a change of settings
    • H04L41/0816—Configuration setting characterised by the conditions triggering a change of settings the condition being an adaptation, e.g. in response to network events
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W24/00—Supervisory, monitoring or testing arrangements
    • H04W24/02—Arrangements for optimising operational condition

Definitions

  • This disclosure relates to the field of wireless communication technology, and in particular to a model and/or function evaluation method, apparatus, related equipment, product and storage medium.
  • AI Artificial Intelligence
  • ML Machine Learning
  • embodiments of this disclosure provide a model and/or functional evaluation method, apparatus, related equipment, products, and storage media.
  • This disclosure provides a model and/or functional evaluation method, performed by a first device, the method comprising at least one of the following:
  • the model includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the function includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the first time includes at least one of the time it takes for the service's model to transition to an inactive model, the time it takes for the model to be evaluated, and the time it takes for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the first message indicates at least one of the following:
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any time unit among the at least one time unit; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the method further includes:
  • the second information includes at least one of the following:
  • the method further includes:
  • the third information includes at least one of the following:
  • determining that the performance loss is less than or equal to a first threshold includes one of the following:
  • the loss rate of the ACK and/or NACK characters is less than or equal to the first threshold
  • the loss rate of ACK and/or NACK characters within the third time period is less than or equal to the first threshold.
  • the method further includes:
  • the fourth message includes fourth time information of the first device performing model and/or functional evaluation
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • This disclosure also provides a model and/or functional evaluation method, performed by a second device, the method comprising:
  • the first message indicates at least one of the following:
  • the first time includes at least one of the time it takes for the service's model to transition to an inactive model, the time it takes for the model to be evaluated, and the time it takes for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any time unit among the at least one time unit; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the method further includes:
  • the system receives second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and/or, the second information is used to instruct the first device to complete the evaluation of the function.
  • the second information includes at least one of the following:
  • the method further includes:
  • the third information includes at least one of the following:
  • the method further includes:
  • Receive fourth information sent by the first device includes fourth time information of the first device performing model and/or functional evaluation;
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • This disclosure also provides a model and/or functional evaluation apparatus, disposed on a first device, comprising:
  • the device includes at least one of the following:
  • An evaluation unit is used to perform model and/or functional evaluation in the first instance.
  • Performance unit used to determine if the performance loss is less than or equal to a first threshold.
  • This disclosure also provides a model and/or functional evaluation apparatus, disposed on a second device, the apparatus comprising:
  • the sending unit is configured to send a first message to the first device; the first message indicates at least one of the following:
  • This disclosure also provides a model and/or functional evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, it implements the steps of any of the first device-side methods described above, or implements the steps of any of the second device-side methods described above.
  • This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described above on the first device side, or implements the steps of any of the methods described above on the second device side.
  • This disclosure also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described above for the first device side, or implements the steps of any of the methods described above for the second device side.
  • the model and/or function evaluation method, apparatus, related equipment, products, and storage media provided in this disclosure are applied to a first device.
  • the method includes at least one of the following: performing model and/or function evaluation within a first time period; determining that the performance loss is less than or equal to a first threshold.
  • the first device performs model and/or function evaluation at a first time period. If AI/ML performance degradation occurs within the first time period, the network knows that the degradation is due to AI/ML model evaluation during that period. Therefore, the network can accept the AI performance loss during that period or may not require the terminal to meet AI performance requirements.
  • the network may consider the current AI/ML model unsuitable, potentially triggering model switching, activation, or deactivation prematurely. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to a non-AI mode. Meanwhile, taking into account the throughput loss caused by inactive AI/ML model evaluation, a trade-off between degrees of freedom and throughput loss is achieved by constraining the allowable data loss rate within a specified time period.
  • Figure 1 is a schematic flowchart of a method for evaluating a model and/or function according to an embodiment of this disclosure
  • Figure 2 is a schematic flowchart of another model and/or function evaluation method according to an embodiment of this disclosure.
  • Figure 3 is a schematic diagram of the structure of a model and/or functional evaluation device according to an embodiment of the present disclosure
  • Figure 4 is a schematic diagram of the structure of another model and/or functional evaluation device according to an embodiment of the present disclosure.
  • Figure 5 is a schematic diagram of the structure of the first device according to an embodiment of this disclosure.
  • Figure 6 is a schematic diagram of the structure of the second device in an embodiment of this disclosure.
  • Figure 7 is a schematic diagram of the structure of the model and/or functional evaluation system of the present disclosure.
  • AI or ML models can also be written as AI/ML models. Changes in configuration parameters or the environment may cause the module to become unsuitable for the environment, leading to a deterioration in the performance of the AI/ML model.
  • a dedicated mechanism is needed to detect the model's performance, thereby aiding in model lifecycle management, including model activation/deactivation (or model switching).
  • Activating/deactivating and switching AI/ML models requires assessing whether the performance of the currently applied model is degrading and identifying the target model to be activated/switched. For example, if model A is currently working, and other models B, C, and D are not being used, when model A becomes unsuitable, it's necessary to determine which of models B, C, and D is suitable for the current environment. Models B, C, and D that are not being used are considered inactive models. To ensure timely activation/deactivation and switching of AI/ML models, in addition to monitoring and evaluating the currently used AI/ML models, it's also necessary to monitor the performance of inactive models in real-time or periodically to quickly identify new models to be applied.
  • the problem is that, depending on device capabilities, when two or more models cannot be activated simultaneously, evaluating the performance of the inactive model requires interrupting the currently applied AI/ML model and switching to the inactive model for evaluation. This switching and evaluation process leads to AI service interruption. Furthermore, if the terminal arbitrarily decides when to perform inactive model evaluation, when AI/ML performance degradation occurs, the network cannot determine whether the problem is due to a mismatch between the current AI/ML model and the terminal switching to the inactive model, potentially leading to incorrect network decisions. In summary, the terminal's behavior is uncontrollable, and the network's and terminal's understanding of AI/ML performance degradation are inconsistent, resulting in a decline in system performance.
  • FIG1 is a schematic flowchart of a model and/or function evaluation method according to an embodiment of this disclosure, the method including at least one of the following:
  • Step 101 Conduct model and/or functional evaluation as soon as possible
  • Step 102 Determine if the performance loss is less than or equal to the first threshold.
  • the first device can be determined according to the actual situation, and is not limited here.
  • the first device may include a terminal and/or network-side equipment, such as a base station or a location management function (LMF).
  • a terminal and/or network-side equipment such as a base station or a location management function (LMF).
  • LMF location management function
  • the first time can be determined according to the actual situation, and is not limited here.
  • the first time can also be described as a first duration, which can be understood as a range of time.
  • the model can be determined according to the actual situation and is not limited here.
  • the model can be an artificial intelligence (AI) and/or machine learning (ML) model. It can be simply referred to as model, AI and/or ML model.
  • AI artificial intelligence
  • ML machine learning
  • a functionality may include or correspond to at least one model.
  • a functionality applies to or corresponds to a certain scenario, such as beam management (or described as beam prediction, including temporal prediction and spatial prediction), CSI compression, CSI prediction, and positioning (including direct positioning and assisted positioning).
  • a model applies to or corresponds to a certain configuration, such as different antenna configurations (e.g., the number of antenna ports), different beam counts, and different stream counts will correspond to different models. Evaluation can also be described as monitoring, performance monitoring, or lifecycle management (LCM). Model evaluation includes the evaluation of deactivated models, and functional evaluation includes the evaluation of deactivated models. Deactivation can also be described as not being deployed or not being applied.
  • the performance loss can be determined based on the actual situation and is not limited here.
  • the performance loss can be described as lost ACK/NACK, or as throughput loss.
  • the performance loss indicates the performance loss due to model and/or functionality evaluation. Determining that the performance loss is less than or equal to a first threshold can also be described as the performance loss not exceeding a first threshold within a certain period of time.
  • the first threshold can be a proportion, percentage, or probability.
  • the advantage of the scheme that performs model and/or functional evaluation within the first timeframe is that it specifies the time and location at which the first device performs the evaluation.
  • the first and second devices can have a consistent understanding; for example, the network and the terminal can agree on when to perform inactive AI/ML model evaluation. Evaluation is performed at the time frame configured by the network, transitioning from a serving AI/ML model to an inactive AI/ML model. Even if AI performance degrades, the network can determine that it's due to the model transition and can make corresponding adjustments within that timeframe.
  • the network knows that the degradation is due to AI/ML model evaluation during that period, and can accept the AI loss during that time, or not require the terminal to meet AI performance requirements (meaning the network doesn't need to take additional measures). If AI/ML performance degrades outside the network-configured timeframe, the network may consider the current AI/ML model unsuitable, potentially triggering model switching, activation, or deactivation prematurely. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to a non-AI mode.
  • the gain of the scheme where the performance loss is less than or equal to the first threshold is as follows: Compared to the first-time scheme, this scheme does not specify the exact location for evaluation, giving the terminal more freedom to determine how to perform inactive AI/ML model evaluation. Simultaneously, considering the throughput loss caused by inactive AI/ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a predetermined timeframe, achieving a trade-off between freedom and throughput loss.
  • the first-time scheme and the first-threshold scheme can be used independently or in combination.
  • model and/or functional evaluation is performed within a first time period; and/or, the performance loss is determined to be less than or equal to a first threshold. That is, the first device performs model and/or functional evaluation at the first time period. If AI/ML performance degradation occurs within this first time period, the network knows that the degradation is due to AI/ML model evaluation during that period. Therefore, the network can accept the AI performance loss during that period, or may not require the terminal to meet AI performance requirements. If AI/ML performance degradation occurs outside the first time period, the network may consider the current AI/ML model unsuitable, potentially triggering model switching, activation, or deactivation prematurely. Furthermore, the network may consider the current environment unsuitable for AI and instruct the terminal to revert to a non-AI mode. Simultaneously, considering the throughput loss caused by inactive AI/ML model evaluation, a constraint is imposed by specifying an allowable data loss rate within a predetermined time period, achieving a trade-off between degrees of freedom and throughput loss.
  • the model includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the function includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the artificial intelligence (AI) and/or machine learning (ML) model can also be referred to as an AI and/or ML model, including AI model, ML model, AI model and ML model.
  • the deactivated AI and/or ML model may include a deactivated AI model, a deactivated ML model, and a deactivated AI model and ML model.
  • the artificial intelligence (AI) and/or machine learning (ML) functions may include AI functions, ML functions, and de-AI and/or ML functions.
  • the deactivated AI and/or ML functionality may include deactivated AI functionality, deactivated ML functionality, and deactivated AI and ML functionality.
  • the first time includes at least one of the following: the time for the service's model to transition to the inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the first time includes at least one of the following: the time when the service model is converted to the inactive model, the evaluation time of the model, and the time when the inactive model is converted to the service model.
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the evaluation time of the functionality, and the time when the inactive functionality transitions to the service's functionality.
  • This can be understood as the first time including the time when the service's functionality transitions to an inactive functionality, the evaluation time of the functionality, the time when the inactive functionality transitions to the service's functionality, the time when the service's functionality transitions to an inactive functionality and the evaluation time of the functionality, and the time when the inactive functionality transitions to the service's functionality.
  • the service can also be described as being applied, deployed, or currently applied/deployed.
  • the method further includes:
  • the first message indicates at least one of the following:
  • the first message can be determined according to the actual situation and is not limited here.
  • the first message can be sent by a network-side device, a core network, an LMF, or an OTT (Over-The-Top) server. If the first device is a terminal, the first message can be sent by at least one of the network-side device, the core network, the LMF, and the OTT server; if the first device is a network-side device, the first message can be sent by at least one of the core network, the LMF, and the OTT server.
  • the period of the first time can be determined according to the actual situation and is not limited here. As an example, the period of the first time can also be described as the period of model and/or functional evaluation.
  • the offset of the first time can be determined according to the actual situation and is not limited here.
  • the offset of the first time can also be described as the starting position of the model and/or functional evaluation time, that is, where in the cycle.
  • the length of the first time interval can be determined based on actual circumstances and is not limited here. As an example, the length of the first time interval can also be described as the duration of the first time interval or the duration of the first time interval.
  • the length of the first time interval includes at least the time during which the first device performs model evaluation.
  • Receiving the first message can be understood as the first device receiving the first message sent by the second device.
  • the second device can be determined based on actual circumstances and is not limited here.
  • the second device can be a network-side device, such as a base station or network.
  • the terminal receives the first information sent by the network.
  • the first information includes the time information for the terminal to evaluate the inactive model.
  • the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI/ML model (it can also include the conversion time from the currently served AI/ML model to the inactive AI/ML model, and the conversion time from the inactive AI/ML model to the original service AI/ML model).
  • the time information included in the first information can be periodic.
  • the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI/ML model evaluation.
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • the step of indicating whether or not the first time unit performs model and/or function evaluation within the first time period through bit information can be determined according to the actual situation and is not limited here.
  • indicating whether or not the first time unit performs model and/or function evaluation through bit information can be understood as indicating whether or not the first time unit is used to perform model and/or function evaluation within the first time period.
  • “yes” or “no” can be described using "whether”.
  • the time information included in the first information can also be non-periodic.
  • the first information includes one or more location information for inactive AI/ML model evaluation within a certain time period.
  • a certain duration can be composed of multiple time units.
  • a bit string indicates which locations (time units) within this duration can perform AI/ML inactive model evaluation.
  • One bit corresponds to one time unit.
  • a bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI/ML model that can interrupt the current service).
  • a bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI/ML model that cannot interrupt the current service).
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the specific process of constructing the second time from at least one first time can be determined according to the actual situation and is not limited here.
  • the second time can be understood as a larger time range; that is, the second time can be understood as being composed of N first times, or as indicating the position of the first time that can be evaluated within a length equal to the length of N first times (N is an integer).
  • the first time can be understood as being within a smaller time range; that is, a subset of the first time, which can be understood as being composed of M first time units, or as indicating the position of the first time unit that can be evaluated within a length equal to the length of M first time units.
  • a scheme can be defined as follows:
  • the first time period consists of at least one time unit, and bit information is used to indicate whether or not a model and/or functional evaluation is performed within that first time unit.
  • a second time period can be defined by at least one first time unit, and bit information is used to indicate whether or not a model and/or functional evaluation is performed within that second time unit.
  • the method further includes:
  • the preset duration and the loss information can be determined according to the actual situation, and are not limited here.
  • the method further includes:
  • the system receives an indication message sent by the terminal.
  • the system reverts to the mode corresponding to a non-inactive AI and/or ML model.
  • the indication information can be determined according to the actual situation, and is not limited here.
  • the indication information is used to fall back to a mode that does not correspond to the inactive AI and/or ML model.
  • the method further includes:
  • the second information can be determined according to the actual situation, and is not limited here.
  • the second information includes at least one of the following: indication information for completing the evaluation; model index ID; function index ID.
  • the terminal receives a second message from the network, indicating whether or not the terminal can perform inactive AI/ML model evaluation within the measurement interval. If the second message indicates that the terminal can perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first message does not need to be sent; if the second message indicates that the terminal cannot perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first message needs to be sent again to indicate where the evaluation should be performed.
  • the network can be configured to use measurement intervals for inactive AI/ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI/ML model evaluation.
  • the second information may include a gain of the second information, mainly considering that the evaluation time required for different AI/ML models varies.
  • the network usually configures a longer duration for evaluation.
  • this longer allowed evaluation time is a waste.
  • the network cannot know when the terminal has completed the evaluation of a model.
  • the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI service for the remaining time of the configured duration.
  • the benefit of the second information is that the network does not need to be configured separately for the timing information used for evaluating inactive AI/ML models, which can reduce signaling overhead and reduce throughput loss.
  • the second information includes at least one of the following:
  • the indication information can be determined according to the actual situation, and is not limited here.
  • the indication information indicates that an inactive AI/ML model evaluation should be performed within the measurement interval.
  • model index ID identifies the specific model
  • function index ID identifies the specific function
  • the method further includes:
  • the third information includes at least one of the following:
  • the third information includes at least one of the following: instructing the first device to perform model evaluation within the measurement interval; instructing the first device to perform functionality evaluation within the measurement interval; instructing the first device not to perform model evaluation within the measurement interval; instructing the first device not to perform functionality evaluation within the measurement interval; which can be understood as instructing whether the first device performs model or functionality evaluation within the measurement interval. That is, the third information can instruct whether the first device performs model or functionality evaluation within the measurement interval.
  • the third information indicating whether or not the first device performs model or functionality evaluation within the measurement interval can also be referred to as the third information indicating whether/not the first device performs model or functionality evaluation within the measurement interval.
  • "Whether or not the first device performs model/functionality evaluation within the measurement interval" can also be described as whether/not the measurement interval can be used for model/functionality evaluation.
  • the measurement interval is a measurement interval used for other purposes, such as the measurement of existing measurement targets.
  • the network can configure the terminal to use this measurement interval for model/functionality evaluation, i.e., trading off a smaller throughput loss due to model/functionality evaluation by extending the measurement time.
  • Receiving third information can be the process by which the first device receives third information sent by the second device.
  • the first device can be a terminal; the second device can be a network; the terminal receives third information sent by the network, which includes a configured period, i.e., compared to the first information, the third information indicates a duration, such as X milliseconds or Y seconds.
  • the network does not specify the specific location for the terminal to perform AI/ML model evaluation within this duration, which is determined autonomously by the terminal.
  • the maximum allowable system loss within this time range needs to be pre-defined in the protocol. For example, the maximum allowable ACK/NACK loss rate within X milliseconds or Y seconds is Z%.
  • the third piece of information can be determined based on the actual situation and is not limited here. As an example, compared with the first piece of information, the third piece of information gives the terminal more freedom to determine how to perform inactive AI/ML model evaluation. At the same time, considering the throughput loss caused by inactive AI/ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a specified time period, thus achieving a trade-off between the degree of freedom and the throughput loss.
  • the benefit of the third information is that the network does not need to be configured separately for time information used for model/functionality evaluation, which can reduce signaling overhead and reduce throughput loss.
  • determining that the performance loss is less than or equal to a first threshold includes one of the following:
  • the loss rate of the ACK and/or NACK characters is less than or equal to the first threshold
  • the loss rate of ACK and/or NACK characters within the third time period is less than or equal to the first threshold.
  • both the first threshold and the third time can be determined according to the actual situation, and no limitation is made here.
  • the method further includes:
  • the fourth message includes fourth time information of the first device performing model and/or functional evaluation
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • the offset of the model and/or function evaluation can be determined according to the actual situation, and is not limited here.
  • the offset of the model and/or function evaluation can be understood as the specific evaluation position of the model and/or function evaluation.
  • the duration of the model and/or function evaluation can be determined according to the actual situation and is not limited here. As an example, the duration of the model and/or function evaluation can be simply referred to as the evaluation duration.
  • Sending the fourth information can be understood as the terminal sending the fourth information to the network.
  • the fourth information including the fourth time information of the first device performing model and/or functional evaluation, can be understood as including the time information of the terminal's expected inactive AI/ML model evaluation.
  • the terminal sends a fourth piece of information to the network.
  • This fourth piece of information includes the time information the terminal expects for evaluating inactive AI/ML models, such as the cycle, specific evaluation location, and evaluation duration.
  • the network can refer to this fourth piece of information to configure the first, second, and third pieces of information.
  • the fourth time information can be determined according to the actual situation and is not limited here.
  • the fourth time information may include the gain of the fourth information: Since the second device cannot know the situation of the first device, the configuration is not optimal.
  • the first device can determine how to evaluate according to its own environment and report the information to the second device through the fourth information to assist the second device in configuring the first time, bit information and the first threshold.
  • the benefit of the fourth information lies in the fact that since the network cannot know the terminal status, the configuration is not optimal.
  • the network can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.
  • the method further includes:
  • an inactive AI and/or ML model is evaluated to obtain the evaluation results
  • the evaluation result is sent to the network device; the evaluation result is used by the network device to update the configuration message.
  • the environmental information of the terminal is obtained; wherein, the environmental information can be understood as the environmental conditions in which the terminal is located.
  • the embodiments disclosed herein mainly consider that the network configuration may not be optimal due to the inability to know the terminal's status.
  • the terminal can determine how to evaluate based on its own environment and report this information to the network to assist the network in configuration.
  • FIG2 is a schematic flowchart of another model and/or function evaluation method according to an embodiment of this disclosure, applied to a second device. The method includes:
  • Step 201 Send a first message to the first device; the first message indicates at least one of the following:
  • the first device can be determined according to the actual situation, and is not limited here.
  • the first device may include a terminal and/or network-side equipment, such as a base station or LMF.
  • the second device can be determined according to the actual situation, and is not limited here.
  • the second device can be a network-side device, such as a base station or network.
  • the first message can be determined according to the actual situation, and is not limited here.
  • the first message can be sent by a network-side device, a core network, an LMF, or an OTT server. If the first device is a terminal, then the first message can be sent by at least one of the network-side device, the core network, the LMF, and the OTT server; if the first device is a network-side device, then the first message can be sent by at least one of the core network, the LMF, and the OTT server.
  • the period of the first time can be determined according to the actual situation and is not limited here. As an example, the period of the first time can also be described as the period of model and/or functional evaluation.
  • the offset of the first time can be determined according to the actual situation and is not limited here.
  • the offset of the first time can also be described as the starting position of the model and/or functional evaluation time, that is, where in the cycle.
  • the length of the first time interval can be determined based on actual circumstances and is not limited here. As an example, the length of the first time interval can also be described as the duration of the first time interval or the duration of the first time interval.
  • the length of the first time interval includes at least the time during which the first device performs model evaluation.
  • Receiving the first message can be understood as the first device receiving the first message sent by the second device.
  • the second device can be determined based on actual circumstances and is not limited here.
  • the second device can be a network-side device, such as a base station or network.
  • the terminal receives the first information sent by the network.
  • the first information includes the time information for the terminal to evaluate the inactive model.
  • the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI/ML model (it can also include the conversion time from the currently served AI/ML model to the inactive AI/ML model, and the conversion time from the inactive AI/ML model to the original service AI/ML model).
  • the time information included in the first information can be periodic.
  • the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI/ML model evaluation.
  • the first time includes at least one of the following: the time for the service's model to transition to the inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the first time includes at least one of the following: the time when the service model is converted to the inactive model, the evaluation time of the model, and the time when the inactive model is converted to the service model.
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the evaluation time of the functionality, and the time when the inactive functionality transitions to the service's functionality.
  • This can be understood as the first time including the time when the service's functionality transitions to an inactive functionality, the evaluation time of the functionality, the time when the inactive functionality transitions to the service's functionality, the time when the service's functionality transitions to an inactive functionality and the evaluation time of the functionality, and the time when the inactive functionality transitions to the service's functionality.
  • the service can also be described as being applied, deployed, or currently applied/deployed.
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • the step of indicating whether or not the first time unit performs model and/or function evaluation within the first time period through bit information can be determined according to the actual situation and is not limited here.
  • indicating whether or not the first time unit performs model and/or function evaluation through bit information can be understood as indicating whether or not the first time unit is used to perform model and/or function evaluation within the first time period.
  • “yes” or “no” can be described using "whether”.
  • the time information included in the first information can also be non-periodic.
  • the first information includes one or more location information for inactive AI/ML model evaluation within a certain time period.
  • a certain duration can be composed of multiple time units.
  • a bit string indicates which locations (time units) within this duration can perform AI/ML inactive model evaluation.
  • One bit corresponds to one time unit.
  • a bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI/ML model that can interrupt the current service).
  • a bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI/ML model that cannot interrupt the current service).
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the specific process of constructing the second time from at least one first time can be determined according to the actual situation and is not limited here.
  • the second time can be understood as a larger time range; that is, the second time can be understood as being composed of N first times, or as indicating the position of the first time that can be evaluated within a length equal to the length of N first times (N is an integer).
  • the first time can be understood as being within a smaller time range; that is, a subset of the first time, which can be understood as being composed of M first time units, or as indicating the position of the first time unit that can be evaluated within a length equal to the length of M first time units.
  • a scheme can be defined as follows:
  • the first time period consists of at least one time unit, and bit information is used to indicate whether or not a model and/or functional evaluation is performed within that first time unit.
  • a second time period can be defined by at least one first time unit, and bit information is used to indicate whether or not a model and/or functional evaluation is performed within that second time unit.
  • the method further includes:
  • the system receives second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and/or, the second information is used to instruct the first device to complete the evaluation of the function.
  • the second information can be determined according to the actual situation, and is not limited here.
  • the second information includes at least one of the following: indication information for completing the evaluation; model index ID; function index ID.
  • Receiving the second information sent by the first device can be understood as the second device receiving the second information sent by the first device.
  • the first device can be a terminal; the second device can be a network; the network receives second information sent by the terminal, the second information indicating whether the terminal can perform inactive AI/ML model evaluation within the measurement interval. If the second information indicates that the terminal can perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first information need not be sent; if the second information indicates that the terminal cannot perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first information needs to be sent further to indicate where the terminal should perform the evaluation.
  • the network can be configured to use measurement intervals for inactive AI/ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI/ML model evaluation.
  • the second information may include a gain of the second information, mainly considering that the evaluation time required for different AI/ML models varies.
  • the network usually configures a longer duration for evaluation.
  • this longer allowed evaluation time is a waste.
  • the network cannot know when the terminal has completed the evaluation of a model.
  • the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI service for the remaining time of the configured duration.
  • the benefit of the second information is that the network does not need to be configured separately for the timing information used for evaluating inactive AI/ML models, which can reduce signaling overhead and reduce throughput loss.
  • the second information includes at least one of the following:
  • the indication information can be determined according to the actual situation, and is not limited here.
  • the indication information indicates that an inactive AI/ML model evaluation should be performed within the measurement interval.
  • model index ID identifies the specific model
  • function index ID identifies the specific function
  • the method further includes:
  • the third information includes at least one of the following:
  • the third information includes at least one of the following: instructing the first device to perform model evaluation within the measurement interval; instructing the first device to perform functionality evaluation within the measurement interval; instructing the first device not to perform model evaluation within the measurement interval; instructing the first device not to perform functionality evaluation within the measurement interval; which can be understood as instructing whether the first device performs model or functionality evaluation within the measurement interval. That is, the third information can instruct whether the first device performs model or functionality evaluation within the measurement interval.
  • the third information indicating whether or not the first device performs model or functionality evaluation within the measurement interval can also be referred to as the third information indicating whether/not the first device performs model or functionality evaluation within the measurement interval.
  • the phrase "whether/not the first device performs model/functionality evaluation within the measurement interval" can also be described as whether/not the measurement interval can be used for model/functionality evaluation.
  • the measurement interval refers to a measurement interval used for other purposes, such as the measurement of a target in existing technologies.
  • the network can configure the terminal to use this measurement interval for model/functionality evaluation, i.e., trading off a smaller throughput loss due to model/functionality evaluation by extending the measurement time.
  • Sending third information to the first device can be a process where the second device sends third information to the first device.
  • the first device can be a terminal; the second device can be a network; the network sends third information to the terminal, which includes a configured period, i.e., compared to the first information, the third information indicates a duration, such as X milliseconds or Y seconds.
  • the network does not specify the specific location where the terminal performs AI/ML model evaluation within this duration, which is determined autonomously by the terminal.
  • the maximum system loss allowed within this time range needs to be pre-defined in the protocol. For example, the maximum allowed ACK/NACK loss rate within X milliseconds or Y seconds is Z%.
  • the third piece of information can be determined based on the actual situation and is not limited here. As an example, compared with the first piece of information, the third piece of information gives the terminal more freedom to determine how to perform inactive AI/ML model evaluation. At the same time, considering the throughput loss caused by inactive AI/ML model evaluation, a constraint is imposed by specifying the allowable data loss rate within a specified time period, thus achieving a trade-off between the degree of freedom and the throughput loss.
  • the benefit of the third information is that the network does not need to be configured separately for time information used for model/functionality evaluation, which can reduce signaling overhead and reduce throughput loss.
  • the method further includes:
  • Receive fourth information sent by the first device includes fourth time information of the first device performing model and/or functional evaluation;
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • the offset of the model and/or function evaluation can be determined according to the actual situation, and is not limited here.
  • the offset of the model and/or function evaluation can be understood as the specific evaluation position of the model and/or function evaluation.
  • the duration of the model and/or function evaluation can be determined according to the actual situation and is not limited here. As an example, the duration of the model and/or function evaluation can be simply referred to as the evaluation duration.
  • Receiving the fourth information sent by the first device can be understood as the second device receiving the fourth information sent by the first device.
  • the fourth information includes fourth time information for the first device to perform model and/or functional evaluation, which can be understood as the fourth information including the time information for the terminal to perform inactive AI/ML model evaluation.
  • the first device can be a terminal; the first device can be a network; the network receives the fourth information sent by the terminal, which includes the terminal's desired time information for evaluating inactive AI/ML models, including the period, specific evaluation location, and evaluation duration.
  • the network can refer to this fourth information to configure the first, second, and third information.
  • the fourth time information can be determined according to the actual situation and is not limited here.
  • the fourth time information may include the gain of the fourth information: Since the second device cannot know the situation of the first device, the configuration is not optimal.
  • the first device can determine how to evaluate according to its own environment and report the information to the second device through the fourth information to assist the second device in configuring the first time, bit information and the first threshold.
  • the benefit of the fourth information lies in the fact that since the network cannot know the terminal status, the configuration is not optimal. Finally, it can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.
  • this disclosure provides a specific scheme for example models and/or functional evaluation methods.
  • the terminal receives the first information sent by the network.
  • the first information includes the time information for the terminal to evaluate the inactive model.
  • the time information can be a period of time within a certain duration, during which the terminal only evaluates the inactive AI/ML model (or it can include the conversion time from the currently served AI/ML model to the inactive AI/ML model, and the conversion time from the inactive AI/ML model to the original service AI/ML model).
  • the time information included in the first information can be periodic.
  • the first information includes the period, offset, and duration (or can be described as the evaluation duration) of the inactive AI/ML model evaluation.
  • the time information included in the first information can also be aperiodic.
  • the first information includes one or more location information for inactive AI/ML model evaluation within a certain time period.
  • a certain duration can be composed of multiple time units.
  • a bit string indicates which locations (time units) within this duration can perform AI/ML inactive model evaluation.
  • One bit corresponds to one time unit.
  • a bit value of 1 or TRUE indicates that the time unit can be used for inactive model evaluation (or can be described as AI/ML model that can interrupt the current service), while a bit value of 0 or FAULSE indicates that the time unit cannot be used for inactive model evaluation (or can be described as AI/ML model that cannot interrupt the current service).
  • the terminal sends a fifth message to the network, indicating that the model evaluation is complete.
  • the fifth message may also include the model ID or functionality identifier that the evaluation is complete.
  • the benefit of introducing the fifth piece of information lies in the fact that, if the first piece of information includes the duration of evaluation, considering the different evaluation times required for different AI/ML models, and to cover as many scenarios as possible, the network usually configures a longer duration for evaluation. However, for models that do not require a long evaluation time, this longer allowed evaluation time is a waste. Furthermore, the network cannot know when the terminal has completed the evaluation of a model. By introducing the fifth piece of information, when the AI/ML model evaluation is completed, the terminal is allowed to notify the network. Then the network knows that the terminal has resumed AI services for the remaining time of the configured duration.
  • the first information gain is that the network and the terminal have a consistent understanding of when to perform inactive AI/ML model evaluation.
  • the evaluation is performed at the time point configured by the network to switch from serving AI/ML model to inactive AI/ML model. Even if the AI performance degrades, the network can determine that it is due to the model switch, and the network can make corresponding adjustments during this period.
  • Network-side behavior If AI/ML performance degradation occurs within the network-configured timeframe, the network knows that this degradation is due to AI/ML model evaluation during that period. Therefore, the network can accept the AI performance loss during this time, or may not require the terminal to meet AI performance demands (meaning the network doesn't need to take additional measures). If AI/ML performance degradation occurs outside the network-configured timeframe, the network may consider the current AI/ML model unsuitable, potentially triggering premature model switching, activation, or deactivation. Furthermore, the network may deem the current environment unsuitable for AI and instruct the terminal to revert to non-AI mode.
  • the terminal receives a second message sent by the network, indicating whether or not the terminal can perform inactive AI/ML model evaluation within the measurement interval. If the second message indicates that the terminal can perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first message need not be sent; if the second message indicates that the terminal cannot perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first message needs to be sent to indicate where the evaluation should be performed.
  • the network can be configured to use measurement intervals for inactive AI/ML model evaluation, that is, to exchange the extended measurement time for a small throughput loss caused by AI/ML model evaluation.
  • the benefit of introducing the second information is that the network does not need to be configured separately for the timing information used for evaluating inactive AI/ML models, which can reduce signaling overhead and reduce throughput loss.
  • Option 2 The terminal receives third information sent by the network.
  • This third information includes a configured period, which, compared to the first information, indicates a duration, such as X milliseconds or Y seconds.
  • the network does not specify the exact location within this duration for the terminal to perform AI/ML model evaluation; this is determined autonomously by the terminal.
  • the maximum allowable system loss within this time frame needs to be pre-defined in the protocol. For example, the maximum allowable ACK/NACK loss rate within X milliseconds or Y seconds is Z%.
  • the gain from introducing third information is that, compared to the first information, it gives the terminal more degrees of freedom to determine how to conduct inactive AI/ML model evaluation.
  • a constraint is imposed by specifying the allowable data loss rate within a predetermined timeframe, achieving a trade-off between degrees of freedom and throughput loss.
  • Option 3 The terminal sends a fourth piece of information to the network.
  • This fourth piece of information includes the terminal's desired time for evaluating inactive AI/ML models, such as the period, specific evaluation location, and evaluation duration.
  • the network can refer to this fourth piece of information to configure the first, second, and third pieces of information.
  • the gain of the fourth information Since the network cannot know the terminal's situation, the configuration is not optimal.
  • the terminal can determine how to evaluate based on its own environment and report this information to the network through the fourth information to assist the network in configuration.
  • the terminal receives first information sent by the network.
  • This first information indicates the time information for the terminal to perform inactive model evaluation.
  • this time information can be a period of time within a certain duration, during which the terminal only performs inactive AI/ML model evaluation.
  • the first information enables the network and the terminal to have a consistent understanding of when to perform inactive AI/ML model evaluation, which can assist the network in making relevant decisions. For example, if the evaluation is performed at the time specified by the network, switching from serving AI/ML model to inactive AI/ML model, even if AI performance degrades, the network can determine that it is due to the AI/ML model evaluation performed within that time period.
  • the network can accept the AI loss during that period, or not require the terminal to meet the AI performance requirements (which can be understood as the network not needing to take additional measures). If AI/ML performance degrades outside the time specified by the network, the network can consider the current AI/ML model unsuitable, which may trigger model switching, activation, deactivation, or even the network may consider the current environment unsuitable for AI and instruct the terminal to fall back to non-AI mode.
  • the terminal receives second information sent by the network, indicating whether or not the terminal can perform inactive AI/ML model evaluation within a measurement interval. If the second information indicates that the terminal can perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first information need not be sent; if the second information indicates that the terminal cannot perform inactive AI/ML model evaluation within the measurement interval, then the aforementioned first information needs to be sent again to indicate where the evaluation should be performed.
  • the second information enables the network to avoid separately configuring time information for inactive AI/ML model evaluation, which can reduce signaling overhead and reduce the losses caused by inactive model evaluation.
  • the terminal receives third information sent by the network.
  • This third information includes a configured period and the maximum allowable loss from inactive model evaluation within a certain timeframe. Compared to the first information, the third information only provides a duration, such as X milliseconds or Y seconds.
  • the network does not specify the exact location where the terminal performs AI/ML model evaluation within this duration; this is determined autonomously by the terminal.
  • it is necessary to specify the maximum allowable system loss within this timeframe e.g., the maximum allowable ACK/NACK loss rate of Z% within X milliseconds or Y seconds). The loss from the terminal performing inactive model evaluation cannot exceed the maximum value in the third information.
  • the third information gives the terminal more freedom to determine how to perform inactive AI/ML model evaluation. Simultaneously, considering the performance loss caused by inactive AI/ML model evaluation, a trade-off between freedom and performance loss is achieved by specifying the maximum allowable loss rate within the agreed duration.
  • the terminal sends a fourth piece of information to the network.
  • This fourth piece of information includes the time information the terminal expects for evaluating inactive AI/ML models, including the period, specific evaluation location, and evaluation duration.
  • the network can refer to this fourth piece of information to configure the first, second, and third pieces of information.
  • Evaluating the performance of the inactive model requires interrupting the currently applied AI/ML model and switching to the inactive model for evaluation. This switching and evaluation process leads to an interruption of the AI service. Furthermore, if the endpoint arbitrarily decides when to perform the inactive model evaluation, when AI/ML performance degrades, the network cannot determine whether the problem is due to a mismatch between the current AI/ML model and the endpoint switching to the inactive model. This could lead to incorrect network decisions and a decrease in system performance.
  • the terminal receives first information from the network.
  • This first information includes time information for the terminal to perform inactive model evaluation.
  • this time information can be a period of time within which the terminal only performs inactive AI/ML model evaluation (it can also include the transition time from the currently served AI/ML model to the inactive AI/ML model, and the transition time from the inactive AI/ML model to the original served AI/ML model).
  • the terminal receives second information from the network, indicating whether/whether the terminal can perform inactive AI/ML model evaluation within a measurement interval.
  • the terminal receives third information configured by the network.
  • This third information only includes the configured period, such as X milliseconds or Y seconds, and the maximum allowed ACK/NACK loss rate within this time range is pre-defined in the protocol.
  • the terminal is allowed to send fourth information to the network, which includes the terminal's desired time information for performing inactive AI/ML model evaluation, including the period, specific evaluation location, and evaluation duration.
  • this disclosure embodiment also provides a model and/or functional evaluation device 300, disposed on a first device, as shown in FIG3, FIG3 being a schematic structural diagram of a model and/or functional evaluation device according to an embodiment of this disclosure; the device 300 includes at least one of the following:
  • Evaluation unit 301 is used to perform model and/or functional evaluation in the first instance
  • Performance unit 302 is used to determine whether the performance loss is less than or equal to a first threshold.
  • the model includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the function includes one of the following:
  • AI Artificial intelligence
  • ML machine learning
  • the first time includes at least one of the following: the time for the service's model to transition to the inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the device 300 further includes a receiving unit for receiving a first message; the first message indicating at least one of the following:
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the device 300 further includes a transmitting unit for transmitting second information; the second information is used to instruct the first device to complete the evaluation of the model; and/or, the second information is used to instruct the first device to complete the evaluation of the function.
  • the second information includes at least one of the following:
  • the receiving unit is further configured to receive third information
  • the third information includes at least one of the following:
  • determining that the performance loss is less than or equal to a first threshold includes one of the following:
  • the loss rate of the ACK and/or NACK characters is less than or equal to the first threshold
  • the loss rate of ACK and/or NACK characters within the third time period is less than or equal to the first threshold.
  • the sending unit is further configured to send fourth information; the fourth information includes fourth time information of the first device performing model and/or functional evaluation;
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • this disclosure embodiment also provides a model and/or function evaluation device, disposed on the second device, as shown in FIG4.
  • FIG4 is a structural schematic diagram of another model and/or function evaluation device according to this disclosure embodiment.
  • the device 400 includes:
  • Sending unit 401 is configured to send a first message to a first device; the first message indicates at least one of the following:
  • the first time includes at least one of the following: the time for the service's model to transition to the inactive model, the model's evaluation time, and the time for the inactive model to transition back to the service's model; and/or,
  • the first time includes at least one of the following: the time when the service's functionality transitions to an inactive functionality, the time when the functionality is evaluated, and the time when the inactive functionality transitions to the service's functionality.
  • the first time period is constituted by at least one time unit, and bit information indicates whether the first time unit within the first time period is used for model and/or functional evaluation; the first time unit is any one of the at least one time units; the bit information includes at least one bit; wherein, one bit corresponds to one time unit; the value of the bit includes a first value and/or a second value; the first value indicates that the first time unit is used for model and/or functional evaluation; the second value indicates that the first time unit is not used for model and/or functional evaluation.
  • a second time is constituted by at least one first time, and bit information is used to indicate whether or not a model and/or function evaluation is performed during the second time; the first time is any one of the at least one first time; the bit information includes at least one bit; wherein, one bit corresponds to one first time; the value of the bit includes a first value and/or a second value; the first value indicates that the first time is used for model and/or function evaluation; the second value indicates that the first time is not used for model and/or function evaluation.
  • the device 400 further includes a receiving unit for receiving second information sent by the first device; the second information is used to instruct the first device to complete the evaluation of the model; and/or, the second information is used to instruct the first device to complete the evaluation of the function.
  • the second information includes at least one of the following:
  • the sending unit 401 is used to send third information to the first device
  • the third information includes at least one of the following:
  • the receiving unit is further configured to receive fourth information sent by the first device; the fourth information includes fourth time information of the first device performing model and/or functional evaluation;
  • the fourth time information includes at least one of the following:
  • the duration of the model and/or functional evaluation is the duration of the model and/or functional evaluation.
  • model and/or function evaluation apparatus provided in the above embodiments is only illustrated by the division of the above program modules when performing model and/or function evaluation.
  • the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above.
  • the model and/or function evaluation apparatus provided in the above embodiments and the model and/or function evaluation method embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.
  • this disclosure also provides a model and/or functional evaluation device, including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • a model and/or functional evaluation device including a memory, a processor, and a computer program stored in the memory and executable on the processor.
  • the processor executes the program, it implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.
  • embodiments of this disclosure provide a computer program product, including a computer program, on which the computer program is stored, wherein when the computer program is executed by a processor, it implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.
  • embodiments of this disclosure provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the methods described in the first device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side; or, when the processor executes the program, it implements the steps of any of the methods described in the second device side.
  • the model and/or functional evaluation device can be a first device;
  • Figure 5 is a schematic diagram of the structure of the first device according to an embodiment of the present disclosure.
  • the first device 500 includes: a first processor 501 and a first memory 503.
  • the first device 500 may also include a first communication interface 502.
  • the first memory 503 can be volatile memory or non-volatile memory, or both.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage.
  • the volatile memory can be random access memory (RAM), which is used as an external cache.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • SSRAM Synchronous Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • ESDRAM Enhanced Synchronous Dynamic Random Access Memory
  • SLDRAM SyncLink Dynamic Random Access Memory
  • DRRAM Direct Rambus Random Access Memory
  • the first memory 503 described in the embodiments of this disclosure is intended to include, but is not limited to, these and any other suitable types of memory.
  • the methods disclosed in the above embodiments of this disclosure can be applied to, or implemented by, the first processor 501.
  • the first processor 501 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in the form of software within the first processor 501.
  • the first processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSP digital signal processor
  • the first processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure.
  • the general-purpose processor may be a microprocessor or any conventional processor, etc.
  • the steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in a storage medium, specifically in the first memory 503.
  • the first processor 501 reads information from the first memory 503 and, in conjunction with its hardware, completes the steps of the aforementioned method.
  • the model and/or functional evaluation device can be a second device;
  • Figure 6 is a schematic diagram of the structure of the second device in this embodiment of the present disclosure.
  • the second device 600 includes: a second processor 601 and a second memory 603.
  • the second device 600 may also include a second communication interface 602.
  • the second memory 603 can be volatile memory or non-volatile memory, or both.
  • the non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be disk storage or magnetic tape storage.
  • the volatile memory can be random access memory (RAM), which is used as an external cache.
  • RAM Random Access Memory
  • SRAM Static Random Access Memory
  • SSRAM Synchronous Static Random Access Memory
  • DRAM Dynamic Random Access Memory
  • SDRAM Synchronous Dynamic Random Access Memory
  • DDRSDRAM Double Data Rate Synchronous Dynamic Random Access Memory
  • ESDRAM Enhanced Synchronous Dynamic Random Access Memory
  • SLDRAM SyncLink Dynamic Random Access Memory
  • DRRAM Direct Rambus Random Access Memory
  • the second memory 603 described in the embodiments of this disclosure is intended to include, but is not limited to, these and any other suitable types of memory.
  • the methods disclosed in the above embodiments of this disclosure can be applied to, or implemented by, the second processor 601.
  • the second processor 601 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware or by instructions in the form of software within the second processor 601.
  • the second processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • DSP digital signal processor
  • the second processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure.
  • the general-purpose processor may be a microprocessor or any conventional processor, etc.
  • the steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor.
  • the software modules may be located in a storage medium, specifically a second memory 603.
  • the second processor 601 reads information from the second memory 603 and, in conjunction with its hardware, completes the steps of the aforementioned method.
  • FIG7 is a schematic diagram of the structure of the model and/or function evaluation system of this disclosure, which includes: a first device 701 and a second device 702.
  • the device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.
  • ASICs application-specific integrated circuits
  • DSPs digital signal processors
  • PLDs programmable logic devices
  • CPLDs complex programmable logic devices
  • FPGAs field-programmable gate arrays
  • general-purpose processors controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned methods.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本公开公开一种模型和/或功能评估方法、装置、相关设备、产品及存储介质。应用于第一设备，所述方法包括以下至少之一：在第一时间内进行模型和/或功能评估；确定性能损失小于或者等于第一门限。

Description

模型和/或功能评估方法、装置、设备、产品及存储介质
相关申请的交叉引用
本公开基于申请号为202410866496.4、申请日为2024年06月28日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本公开作为参考。
技术领域
本公开涉及无线通信技术领域，尤其涉及一种模型和/或功能评估方法、装置、相关设备、产品及存储介质。
背景技术
为了能够及时完成人工智能(Artificial Intelligence，AI)或机器学习(Machine Learning，ML)模型的激活/去激活、切换，除了需要检测评估当前服务的AI/ML模型，还需要实时或周期性的检测去激活(inactive)模型的性能便于快速确定新的被应用的模型。但存在的问题是，部分设备受限于能力，可能不能同时激活2个及以上的模型，在评估去激活模型(inactive model)的性能时，需要中断当前正在应用的AI或ML模型，转换至inactive model评估。那么，当出现AI或ML模型性能下降时，网络将无法确定该问题是由于当前AI或ML模型不匹配导致，还是由于终端切换至inactive model导致，网络可能会做出错误的决策，带来系统性能的下降。
发明内容
为解决现有存在的技术问题，本公开实施例提供一种模型和/或功能评估方法、装置、相关设备、产品及存储介质。
本公开实施例提供了一种模型和/或功能评估方法，由第一设备执行，所述方法包括以下至少之一：
在第一时间内进行模型和/或功能评估；
确定性能损失小于或者等于第一门限。
在一些实施例中，所述模型包括以下之一：
去激活模型inactive model；
人工智能AI和/或机器学习ML模型；
去激活的AI和/或ML模型；和/或，
所述功能包括以下之一：
去激活功能inactive functionality；
人工智能AI和/或机器学习ML功能；
去激活的AI和/或ML functionality。
在一些实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
在一些实施例中，所述方法还包括：
接收第一消息；所述第一消息指示以下至少之一：
所述第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
在一些实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
在一些实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
在一些实施例中，所述方法还包括：
发送第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
在一些实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型索引ID；
功能索引ID。
在一些实施例中，所述方法还包括：
接收第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
在一些实施例中，所述确定性能损失小于或者等于第一门限包括以下之一：
确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限；
在第三时间内的确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限。
在一些实施例中，所述方法还包括：
发送第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
本公开实施例还提供了一种模型和/或功能评估方法，由第二设备执行，所述方法包括：
向第一设备发送第一消息；所述第一消息指示以下至少之一：
第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
在一些实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
在一些实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
在一些实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
在一些实施例中，所述方法还包括：
接收所述第一设备发送的第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
在一些实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型身份标识号ID；
功能身份标识号ID。
在一些实施例中，所述方法还包括：
向所述第一设备发送第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
在一些实施例中，所述方法还包括：
接收所述第一设备发送的第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
本公开实施例还提供了一种模型和/或功能评估装置，设置在第一设备上，包括：
所述装置包括以下至少之一：
评估单元，用于在第一时间内进行模型和/或功能评估；
性能单元，用于确定性能损失小于或者等于第一门限。
本公开实施例还提供了一种模型和/或功能评估装置，设置在第二设备上，所述装置包括：
发送单元，用于向第一设备发送第一消息；所述第一消息指示以下至少之一：
第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
本公开实施例还提供了一种模型和/或功能评估设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述第一设备侧任一方法的步骤，或者实现上述第二设备侧任一方法的步骤。
本公开实施例还提供了一种计算机程序产品，包括计算机程序，所述计算机程序在被处理器执行时实现上述第一设备侧任一方法的步骤，或者实现上述第二设备侧任一方法的步骤。
本公开实施例还提供了一种存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述第一设备侧任一方法的步骤，或者实现上述第二设备侧任一方法的步骤。
本公开实施例提供的模型和/或功能评估方法、装置、相关设备、产品及存储介质，所述方法应用于第一设备，所述方法包括以下至少之一：在第一时间内进行模型和/或功能评估；确定性能损失小于或者等于第一门限。采用本公开的实施例，通过在第一时间内进行模型和/或功能评估；和/或，确定性能损失小于或者等于第一门限。即第一设备在第一时间位置进行模型和/或功能评估，当在第一时间内出现AI/ML性能下降，那么网络可知在该段时间内是由于进行AI/ML模型评估导致，那么网络可以接受该段时间的AI损失，或者不按AI性能需求来要求终端。如果在第一时间之外出现AI/ML性能下降，那么网络可能认为当前AI/ML模型不适用，可能会过早的触发模型切换、激活、去激活，甚至于，网络可能会认为当前环境不适用AI，指示终端回退到非AI模式。同时，还考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的折中。
附图说明
图1为本公开实施例一种模型和/或功能评估的方法流程示意图；
图2为本公开实施例另一种模型和/或功能评估的方法流程示意图；
图3为本公开实施例一种模型和/或功能评估装置的结构示意图；
图4为本公开实施例又一种模型和/或功能评估装置的结构示意图；
图5为本公开实施例第一设备的结构示意图；
图6为本公开实施例中第二设备的结构示意图；
图7为本公开实施例模型和/或功能评估系统结构示意图。
具体实施方式
下面结合附图及实施例对本公开再作进一步详细的描述。
AI或ML模型也可以写成AI/ML模型，可能由于配置的参数或者所处的场景发生变化，导致模块不再匹配所处环境，AI/ML模型的性能变差。需要有专门的机制检测此模型的性能，进而帮助进行模型的生命周期管理，包括模型的激活/去激活(或描述为模型切换)。
AI/ML模型的激活/去激活、切换需要评估当前正在应用的模型是否性能下降，以及确定待激活/切换的目标模型。举例说明，当前正在工作的是模型A，其他未被应用的有模型B,模型C，模型D，在模型A不再适用的时候，需要在模型B、C、D中确定哪一个适用于当前的环境。没有被应用的模型B\C\D属于inactive model。为了能够及时完成AI/ML模型的激活/去激活、切换，除了需要检测评估当前服务的AI/ML模型，还需要实时或周期性的检测inactive模型的性能，便于快速确定新的被应用的模型。
但存在的问题是，取决于设备能力，当不能同时激活2个及以上的模型时，在评估inactive model性能时，需要中断当前正在应用的AI/ML模型，转换至inactive模型进行评估，该转换过程和评估过程会导致AI服务的中断。而且如果由终端任意决定何时进行inactive model评估，那么当出现AI/ML性能下降的时候，网络无法确定该问题是由于当前AI/ML model不匹配导致，还是由于终端切换至inactive model，导致网络可能会做出错误的决策。综上，终端行为不可控，网络与终端对AI/ML性能下降的理解不一致，导致系统性能的下降。
本公开实施例提供了一种模型和/或功能评估方法，应用于第一设备，如图1所示，图1为本公开实施例一种模型和/或功能评估的方法流程示意图，所述方法包括以下至少之一：
步骤101：在第一时间内进行模型和/或功能评估；
步骤102：确定性能损失小于或者等于第一门限。
需要说明的是，所述第一设备可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第一设备可以包括终端和/或网络侧设备，比如基站、位置管理功能(Location Management Function，LMF)。
在步骤101中，所述第一时间可以根据实际情况进行确定，在此不做限定。所述第一时间还可以描述为第一时长，可以理解为一段时间范围。
所述模型可以根据实际情况进行确定，在此不做限定，作为一种示例，所述模型可以为人工智能AI和/或机器学习ML model。可以简称为model、AI和/或ML model。
所述功能可以根据实际情况进行确定，在此不做限定，作为一种示例，所述功能可以描述为functionality,或者AI和/或ML functionality。一个/种functionality可以包括或者对应至少一个/种model。functionality应用于或者对应于一定的场景，比如波束管理(或者描述为波束预测，包括时域预测、空域预测)、CSI压缩、CSI预测、定位(包括直接定位、辅助定位)。Model应用于或者对应于一定的配置，比如不同的天线配置(比如天线端口数)、不同的波束数目、不同的流数会对应不同的model。评估还可以描述为监测(monitoring或monitor)，或者描述为性能监测，或者描述为生命周期管理(Life Cycle Managment，LCM)。模型评估包括去激活模型的评估，功能评估包括去激活的模型的评估。去激活还可以描述为未被部署或未被应用。
在步骤102中，所述性能损失可以根据实际情况进行确定，在此不做限定，作为一种示例，所述性能损失可以描述为丢失的ACK/NACK，或者描述为吞吐率损失。性能损失指示由于model和/或functionality评估导致的性能损失。确定性能损失小于或者等于第一门限还可以描述为一定时间内的性能损失不超过第一门限。第一门限可以是比例、百分比、概率。
在第一时间内进行模型和/或功能评估的方案的增益为：该方案规定了第一设备在什么时间位置进行评估。第一设备和第二设备可以有一致理解，以网络和终端为例，对于何时进行inactive AI/ML model评估有一致的理解，在网络配置的时间位置进行由服务AI/ML model转换至inactive AI/ML model进行评估，即使AI性能下降，网络也可以确定是由于model转换导致，而且网络可以在该段时间内进行相应的调整。当在网络配置的时间内出现AI/ML性能下降，那么网络可知在该段时间内是由于进行AI/ML模型评估导致，那么网络可以接受该段时间的AI损失，或者不按AI性能需求来要求终端(可以理解为网络不需要采用额外措施)。如果在网络配置的时间之外出现AI/ML性能下降，那么网络可能认为当前AI/ML模型不适用，可能会过早的触发模型切换、激活、去激活，甚至于，网络可能会认为当前环境不适用AI，指示终端回退到非AI模式。
确定性能损失小于或者等于第一门限的方案的增益为：相比于第一时间的方案，该方案没有规定进行评估的具体位置，留给终端更多的自由度确定如何进行inactive AI/ML model评估。同时，考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的折中。第一时间方案和第一门限方案，这两个方案可以各自独立使用，也可以结合使用。
本公开实施例，通过在第一时间内进行模型和/或功能评估；和/或，确定性能损失小于或者等于第一门限。即第一设备在第一时间位置进行模型和/或功能评估，当在第一时间内出现AI/ML性能下降，那么网络可知在该段时间内是由于进行AI/ML模型评估导致，那么网络可以接受该段时间的AI损失，或者不按AI性能需求来要求终端。如果在第一时间之外出现AI/ML性能下降，那么网络可能认为当前AI/ML模型不适用，可能会过早的触发模型切换、激活、去激活，甚至于，网络可能会认为当前环境不适用AI，指示终端回退到非AI模式。同时，还考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的折中。
在一实施例中，所述模型包括以下之一：
去激活模型inactive model；
人工智能AI和/或机器学习ML模型；
去激活的AI和/或ML模型；和/或，
所述功能包括以下之一：
去激活功能inactive functionality；
人工智能AI和/或机器学习ML功能；
去激活的AI和/或ML functionality。
本实施例中，所述人工智能AI和/或机器学习ML模型也可以称为AI和/或ML模型，包括AI模型、ML模型、AI模型和ML模型。
所述去激活的AI和/或ML模型可以包括去激活的AI模型、去激活的ML模型、去激活的AI模型和ML模型。
所述人工智能AI和/或机器学习ML功能可以包括AI功能、ML功能、去AI和/或ML功能。
所述去激活的AI和/或ML functionality可以包括去激活的AI functionality、去激活的ML functionality、去激活的AI和ML functionality。
在一实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
本实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一可以理解为所述第一时间可以包括服务的model转换至inactive model的时间、model的评估时间、inactive model转换至服务的model的时间、服务的model转换至inactive model的时间和model的评估时间、服务的model转换至inactive model的时间和model的评估时间以及inactive model转换至服务的model的时间。
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一可以理解为所述第一时间可以包括服务的functionality转换至inactive functionality的时间、functionality的评估时间、以及inactive functionality转换至服务的functionality的时间、服务的functionality转换至inactive functionality的时间和functionality的评估时间、服务的functionality转换至inactive functionality的时间和functionality的评估时间以及inactive functionality转换至服务的functionality的时间。
在实际应用中，所述服务还可以描述为被应用，或者描述为被部署，或者描述为当前应用/部署，正在应用/部署。
在一实施例中，所述方法还包括：
接收第一消息；所述第一消息指示以下至少之一：
所述第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
本实施例，所述第一消息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第一消息可以由网络侧设备发送，也可以由核心网发送，也可以由LMF发送，也可以由OTT(Over-The-Top)server发送。如果第一设备是终端，那么第一消息可以由网络侧设备、核心网、LMF、OTT server中的至少之一发送；如果第一设备是网络侧设备，那么第一消息可以由核心网、LMF、OTT server中的至少之一发送。
所述第一时间的周期可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的周期还可以描述为模型和/或功能评估的周期。
所述第一时间的偏移可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的偏移，还可以描述为模型和/或功能评估的时间的起始位置，即在周期内的什么位置。
所述第一时间的长度可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的长度还可以描述为第一时间的持续长度，或者描述为持续时间。所述第一时间的长度至少包括第一设备进行模型评估的时间。
所述接收第一消息可以理解为第一设备接收第二设备发送的第一消息。其中，所述第二设备可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二设备可以是网络侧设备，比如基站、网络。
在实际应用中，终端接收网络发送的第一信息，第一信息包括终端进行inactive model评估的时间信息，具体的，该时间信息可以是一定时长内的一段时间，终端只在该段时间内进行inactive AI/ML model的评估(也可以包括由当前服务的AI/ML model转换至inactive AI/ML model的转换时间、由inactive AI/ML model转换至原服务的AI/ML model的转换时间)。
进一步的，第一信息包括的时间信息可以是周期性的，在这种情况下，第一信息包括进行inactive AI/ML model评估的周期、偏移、持续时间(或描述为评估时长)。
在一实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
本实施例中，所述通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以根据实际情况进行确定，在此不做限定，作为一种示例，所述通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以理解为所述通过比特信息指示所述第一时间内第一时间单元用于进行model和/或功能的评估或不用于进行model和/或功能的评估。其中，所述是或否可以用whether描述。
在实际应用中，第一信息包括的时间信息也可以是非周期的，在这种情况下，第一信息包括一定时间内进行inactive AI/ML model评估的一个或者多个位置信息。具体的，作为一种实施方式，可以是由多个时间单元构成一定时长，由比特串指示该段时长内哪些位置(时间单元)可以进行AI/ML inactive model评估，1个比特对应1个时间单元，比特取值为1或者为TRUE，代表该时间单元可以用于inactive model评估(或者描述为可以中断当前服务的AI/ML model)，比特取值为0或者为FAULSE，代表该时间单元不可以用于inactive model评估(或者描述为不能中断当前服务的AI/ML model)。
在一实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
本实施例中，由至少一个所述第一时间构成第二时间中的具体构成过程可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二时间可以理解是更大的一个时间范围；即第二时间，可以理解为由N个第一时间构成第二时间，或者理解为第二时间的长度等于N个第一时间的长度(N为整数)内指示可以进行评估的第一时间的位置。所述第一时间可以理解是在在一个更小的时间范围；即第一时间的子集，可以理解为由M个第一时间单元构成第一时间，或者理解为第一时间的长度等于M个第一时间单元的长度内指示可以进行评估的第一时间单元的位置。
在实际应用中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以称为方案一；由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估可以称为方案二；这两个方案可以单独使用，也可以结合使用。这两个方案可以实现非周期的评估。所述比特信息可以通过消息发送给第一设备。
在一实施例中，所述方法还包括：
在预设时长内出现所述inactive AI和/或ML model性能下降的情况下，向第二设备发送在所述预设时长内所述inactive AI和/或ML model性能的损失信息。
本实施例中，所述预设时长和所述损失信息可以根据实际情况进行确定，在此不做限定。
在一实施例中，所述方法还包括：
在所述预设时长外出现所述inactive AI和/或ML model性能下降的情况下，接收所述终端发送的指示信息；
基于所述指示信息回退到非所述inactive AI和/或ML model对应的模式。
本实施例中，所述指示信息可以根据实际情况进行确定，在此不做限定。作为一种示例，所述指示信息用于回退到非所述inactive AI和/或ML model对应的模式。
在一实施例中，所述方法还包括：
发送第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
本实施例中，所述第二信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二信息包括以下至少之一：完成评估的指示信息；模型索引ID；功能索引ID。
在实际应用中，终端接收网络发送的第二信息，第二信息指示终端是/否可以在测量间隔内进行inactive AI/ML model评估。如果第二信息指示终端可以采用测量间隔进行inactive AI/ML model评估，那么可以不发送上述第一信息；如果第二信息指示终端不可以采用测量间隔进行inactive AI/ML model评估，那么需要进一步发送上述第一信息指示终端在哪里进行评估。
具体的，当移动性需求不高时，比如需要测量间隔的待测频点较少时，网络可以配置终端采用测量间隔进行inactive AI/ML model评估，即通过测量时间的延长换取较小的由于AI/ML model评估带来的吞吐率损失。
作为一种示例，所述第二信息可以包括第二信息的增益，主要考虑到不同AI/ML模型的评估需要的时间不同，为了覆盖尽可能多的场景，以网络和终端场景为例，网络通常会配置较长的用于评估的持续时长，但对于不需要较长评估时间的模型，该配置的较长的评估允许时长是一种浪费。但网络无法获知终端何时完成某个模型的评估，通过第二信息的引入，当完成AI/ML模型的评估时，允许终端通知网络，那么网络可知，在配置的持续时间的剩余时间里，终端已恢复AI服务。
在实际应用中，第二信息的增益在于网络可以不单独配置用于inactive AI/ML model评估的时间信息，可以降低信令开销，降低吞吐率损失。
在一实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型索引ID；
功能索引ID。
本实施例中，所述指示信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述指示信息指示在测量间隔内进行inactive AI/ML model评估。
所述模型索引ID和所述功能索引ID均可以根据实际情况进行确定，在此不做限定。所述模型索引ID可以确定具体的模型；所述功能索引ID可以确定具体的功能。
在一实施例中，所述方法还包括：
接收第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
本实施例中，所述第三信息包括以下至少之一：指示所述第一设备在测量间隔内进行model的评估；指示所述第一设备在测量间隔内进行functionality的评估；指示所述第一设备不在测量间隔内进行model的评估；指示所述第一设备不在测量间隔内进行functionality的评估；可以理解为指示所述第一设备是或否在测量间隔内进行model或functionality的评估。也就是说，所述第三信息可以指示所述第一设备是或否在测量间隔内进行model或functionality的评估。
所述第三信息指示所述第一设备是或否在测量间隔内进行model或functionality的评估也可以称为所述第三信息指示所述第一设备是/或否在测量间隔内进行model或functionality的评估。所述第一设备是/否在测量间隔内进行model/functionality的评估”还可以描述为测量间隔是/否可以用于model/functionality的评估。这里的测量间隔是用于其他用途的测量间隔，比如现有技术的测量目标的测量。当移动性需求不高时，如果需要测量间隔的待测频点(频点也可以描述为测量目标)较少时，网络可以配置终端采用该测量间隔进行model/functionality的评估，即通过测量时间的延长换取较小的由于model/functionality评估带来的吞吐率损失。
接收第三信息可以为第一设备接收第二设备发送的第三信息。
在实际应用中，所述第一设备可以为终端；所述第二设备可以为网络；终端接收网络发送的第三信息，第三信息包括配置的周期，即相比于第一信息，第三信息指示给出一个时长，比如X毫秒，或者Y秒。网络不指定终端在该时长内进行AI/ML model评估的具体位置，由终端自主确定。但需要在协议预先规定在该时间范围内允许的最大的系统损失。比如，在X毫秒或Y秒内允许的最大ACK/NACK丢失率为Z％。
所述第三信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第三信息相比于第一信息，留给终端更多的自由度确定如何进行inactive AI/ML model评估。同时，考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的trade-off。
第三信息的增益在于：网络可以不单独配置用于model/functionality评估的时间信息，可以降低信令开销，降低吞吐率损失。
在一实施例中，所述确定性能损失小于或者等于第一门限包括以下之一：
确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限；
在第三时间内的确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限。
本实施例中，所述第一门限和所述第三时间均可以根据实际情况进行确定，在此不做限定。
在一实施例中，所述方法还包括：
发送第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
本实施例中，所述model和/或功能评估的偏移可以根据实际情况进行确定，在此不做限定，作为一种示例，所述model和/或功能评估的偏移可以理解为所述model和/或功能评估的具体评估位置。
所述model和/或功能评估的时长可以根据实际情况进行确定，在此不做限定，作为一种示例，所述model和/或功能评估的时长可以简称为评估时长。
所述发送第四信息可以理解为终端向网络发送第四信息。
所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息可以理解为第四信息包括终端期望的进行inactive AI/ML model评估的时间信息。
在实际应用中，终端向网络发送第四信息，第四信息包括终端期望的进行inactive AI/ML model评估的时间信息，包括周期、具体评估位置、评估时长等。网络可以参考该第四信息进行第一信息、第二信息、第三信息的配置。
所述第四时间信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第四时间信息可以包括第四信息的增益：第二设备由于无法获知第一设备的情况导致配置不是最佳，第一设备可以根据自身所处的环境情况确定如何进行评估，并将该信息通过第四信息上报给第二设备，辅助第二设备进行第一时间、比特信息、第一门限的配置。
在实际应用中，第四信息的增益在于网络由于无法获知终端情况导致配置不是最佳，终可以根据自身所处的环境情况确定如何进行评估，并将该信息通过第四信息上报给网络，辅助网络进行配置。
在一实施例中，所述方法还包括：
获取所述终端所处的环境信息；
基于所述环境信息和第二时间信息进行inactive AI和/或ML model评估，得到评估结果；
向所述网络设备发送所述评估结果；所述评估结果用于所述网络设备更新所述配置消息。
本公开实施例中，获取所述终端所处的环境信息；其中，所述环境信息可以理解为自身所处的环境情况。
本公开实施例主要考虑网络由于无法获知终端情况导致配置不是最佳，终端可以根据自身所处的环境情况确定如何进行评估，并将该信息上报给网络，辅助网络进行配置。
相应地，本公开实施例还提供一种模型和/或功能评估方法，如图2所示，图2为本公开实施例另一种模型和/或功能评估的方法流程示意图，应用于第二设备，所述方法包括：
步骤201：向第一设备发送第一消息；所述第一消息指示以下至少之一：
第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
需要说明的是，需要说明的是，所述第一设备可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第一设备可以包括终端和/或网络侧设备，比如基站、LMF。
所述第二设备可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二设备可以是网络侧设备，比如基站、网络。
在步骤201中，所述第一消息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第一消息可以由网络侧设备发送，也可以由核心网发送，也可以由LMF发送，也可以由OTT server发送。如果第一设备是终端，那么第一消息可以由网络侧设备、核心网、LMF、OTT server中的至少之一发送；如果第一设备是网络侧设备，那么第一消息可以由核心网、LMF、OTT server中的至少之一发送。
所述第一时间的周期可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的周期还可以描述为模型和/或功能评估的周期。
所述第一时间的偏移可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的偏移，还可以描述为模型和/或功能评估的时间的起始位置，即在周期内的什么位置。
所述第一时间的长度可以根据实际情况进行确定，在此不做限定，作为一种示例，第一时间的长度还可以描述为第一时间的持续长度，或者描述为持续时间。所述第一时间的长度至少包括第一设备进行模型评估的时间。
所述接收第一消息可以理解为第一设备接收第二设备发送的第一消息。其中，所述第二设备可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二设备可以是网络侧设备，比如基站、网络。
在实际应用中，终端接收网络发送的第一信息，第一信息包括终端进行inactive model评估的时间信息，具体的，该时间信息可以是一定时长内的一段时间，终端只在该段时间内进行inactive AI/ML model的评估(也可以包括由当前服务的AI/ML model转换至inactive AI/ML model的转换时间、由inactive AI/ML model转换至原服务的AI/ML model的转换时间)。
进一步的，第一信息包括的时间信息可以是周期性的，在这种情况下，第一信息包括进行inactive AI/ML model评估的周期、偏移、持续时间(或描述为评估时长)。
在一实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
本实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一可以理解为所述第一时间可以包括服务的model转换至inactive model的时间、model的评估时间、inactive model转换至服务的model的时间、服务的model转换至inactive model的时间和model的评估时间、服务的model转换至inactive model的时间和model的评估时间以及inactive model转换至服务的model的时间。
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一可以理解为所述第一时间可以包括服务的functionality转换至inactive functionality的时间、functionality的评估时间、以及inactive functionality转换至服务的functionality的时间、服务的functionality转换至inactive functionality的时间和functionality的评估时间、服务的functionality转换至inactive functionality的时间和functionality的评估时间以及inactive functionality转换至服务的functionality的时间。
在实际应用中，所述服务还可以描述为被应用，或者描述为被部署，或者描述为当前应用/部署，正在应用/部署。
在一实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
本实施例中，所述通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以根据实际情况进行确定，在此不做限定，作为一种示例，所述通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以理解为所述通过比特信息指示所述第一时间内第一时间单元用于进行model和/或功能的评估或不用于进行model和/或功能的评估。其中，所述是或否可以用whether描述。
在实际应用中，第一信息包括的时间信息也可以是非周期的，在这种情况下，第一信息包括一定时间内进行inactive AI/ML model评估的一个或者多个位置信息。具体的，作为一种实施方式，可以是由多个时间单元构成一定时长，由比特串指示该段时长内哪些位置(时间单元)可以进行AI/ML inactive model评估，1个比特对应1个时间单元，比特取值为1或者为TRUE，代表该时间单元可以用于inactive model评估(或者描述为可以中断当前服务的AI/ML model)，比特取值为0或者为FAULSE，代表该时间单元不可以用于inactive model评估(或者描述为不能中断当前服务的AI/ML model)。
在一实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
本实施例中，由至少一个所述第一时间构成第二时间中的具体构成过程可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二时间可以理解是更大的一个时间范围；即第二时间，可以理解为由N个第一时间构成第二时间，或者理解为第二时间的长度等于N个第一时间的长度(N为整数)内指示可以进行评估的第一时间的位置。所述第一时间可以理解是在在一个更小的时间范围；即第一时间的子集，可以理解为由M个第一时间单元构成第一时间，或者理解为第一时间的长度等于M个第一时间单元的长度内指示可以进行评估的第一时间单元的位置。
在实际应用中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估可以称为方案一；由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估可以称为方案二；这两个方案可以单独使用，也可以结合使用。这两个方案可以实现非周期的评估。所述比特信息可以通过消息发送给第一设备。
在一实施例中，所述方法还包括：
接收所述第一设备发送的第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
本实施例中，所述第二信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第二信息包括以下至少之一：完成评估的指示信息；模型索引ID；功能索引ID。
接收所述第一设备发送的第二信息可以理解为第二设备接收所述第一设备发送的第二信息。
在实际应用中，所述第一设备可以为终端；所述第二设备可以为网络；网络接收终端发送的第二信息，第二信息指示终端是/否可以在测量间隔内进行inactive AI/ML model评估。如果第二信息指示终端可以采用测量间隔进行inactive AI/ML model评估，那么可以不发送上述第一信息；如果第二信息指示终端不可以采用测量间隔进行inactive AI/ML model评估，那么需要进一步发送上述第一信息指示终端在哪里进行评估。
具体的，当移动性需求不高时，比如需要测量间隔的待测频点较少时，网络可以配置终端采用测量间隔进行inactive AI/ML model评估，即通过测量时间的延长换取较小的由于AI/ML model评估带来的吞吐率损失。
作为一种示例，所述第二信息可以包括第二信息的增益，主要考虑到不同AI/ML模型的评估需要的时间不同，为了覆盖尽可能多的场景，以网络和终端场景为例，网络通常会配置较长的用于评估的持续时长，但对于不需要较长评估时间的模型，该配置的较长的评估允许时长是一种浪费。但网络无法获知终端何时完成某个模型的评估，通过第二信息的引入，当完成AI/ML模型的评估时，允许终端通知网络，那么网络可知，在配置的持续时间的剩余时间里，终端已恢复AI服务。
在实际应用中，第二信息的增益在于网络可以不单独配置用于inactive AI/ML model评估的时间信息，可以降低信令开销，降低吞吐率损失。
在一实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型身份标识号ID；
功能身份标识号ID。
本实施例中，所述指示信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述指示信息指示在测量间隔内进行inactive AI/ML model评估。
所述模型索引ID和所述功能索引ID均可以根据实际情况进行确定，在此不做限定。所述模型索引ID可以确定具体的模型；所述功能索引ID可以确定具体的功能。
在一实施例中，所述方法还包括：
向所述第一设备发送第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
本实施例中，所述第三信息包括以下至少之一：指示所述第一设备在测量间隔内进行model的评估；指示所述第一设备在测量间隔内进行functionality的评估；指示所述第一设备不在测量间隔内进行model的评估；指示所述第一设备不在测量间隔内进行functionality的评估；可以理解为指示所述第一设备是或否在测量间隔内进行model或functionality的评估。也就是说，所述第三信息可以指示所述第一设备是或否在测量间隔内进行model或functionality的评估。
所述第三信息指示所述第一设备是或否在测量间隔内进行model或functionality的评估也可以称为所述第三信息指示所述第一设备是/或否在测量间隔内进行model或functionality的评估。所述“第一设备是/否在测量间隔内进行model/functionality的评估”还可以描述为测量间隔是/否可以用于model/functionality的评估。这里的测量间隔是用于其他用途的测量间隔，比如现有技术的测量目标的测量。当移动性需求不高时，如果需要测量间隔的待测频点(频点也可以描述为测量目标)较少时，网络可以配置终端采用该测量间隔进行model/functionality的评估，即通过测量时间的延长换取较小的由于model/functionality评估带来的吞吐率损失。
向所述第一设备发送第三信息可以为第二设备向第一设备发送第三信息。
在实际应用中，所述第一设备可以为终端；所述第二设备可以为网络；网络向终端发送第三信息，第三信息包括配置的周期，即相比于第一信息，第三信息指示给出一个时长，比如X毫秒，或者Y秒。网络不指定终端在该时长内进行AI/ML model评估的具体位置，由终端自主确定。但需要在协议预先规定在该时间范围内允许的最大的系统损失。比如，在X毫秒或Y秒内允许的最大ACK/NACK丢失率为Z％。
所述第三信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第三信息相比于第一信息，留给终端更多的自由度确定如何进行inactive AI/ML model评估。同时，考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的trade-off。
第三信息的增益在于：网络可以不单独配置用于model/functionality评估的时间信息，可以降低信令开销，降低吞吐率损失。
在一实施例中，所述方法还包括：
接收所述第一设备发送的第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
本实施例中，所述model和/或功能评估的偏移可以根据实际情况进行确定，在此不做限定，作为一种示例，所述model和/或功能评估的偏移可以理解为所述model和/或功能评估的具体评估位置。
所述model和/或功能评估的时长可以根据实际情况进行确定，在此不做限定，作为一种示例，所述model和/或功能评估的时长可以简称为评估时长。
接收所述第一设备发送的第四信息可以理解为第二设备接收所述第一设备发送的第四信息。
所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息可以理解为第四信息包括终端期望的进行inactive AI/ML model评估的时间信息。
在实际应用中，所述第一设备可以为终端；所述第一设备可以为网络；网络接收终端发送的第四信息，第四信息包括终端期望的进行inactive AI/ML model评估的时间信息，包括周期、具体评估位置、评估时长等。网络可以参考该第四信息进行第一信息、第二信息、第三信息的配置。
所述第四时间信息可以根据实际情况进行确定，在此不做限定，作为一种示例，所述第四时间信息可以包括第四信息的增益：第二设备由于无法获知第一设备的情况导致配置不是最佳，第一设备可以根据自身所处的环境情况确定如何进行评估，并将该信息通过第四信息上报给第二设备，辅助第二设备进行第一时间、比特信息、第一门限的配置。
在实际应用中，第四信息的增益在于网络由于无法获知终端情况导致配置不是最佳，终可以根据自身所处的环境情况确定如何进行评估，并将该信息通过第四信息上报给网络，辅助网络进行配置。
为了方便理解，本公开示例模型和/或功能评估方法的具体方案。
方案1：终端接收网络发送的第一信息，第一信息包括终端进行inactive model评估的时间信息，具体的，该时间信息可以是一定时长内的一段时间，终端只在该段时间内进行inactive AI/ML model的评估(也可以包括由当前服务的AI/ML model转换至inactive AI/ML model的转换时间、由inactive AI/ML model转换至原服务的AI/ML model的转换时间)。
进一步的，第一信息包括的时间信息可以是周期性的，在这种情况下，第一信息包括进行inactive AI/ML model评估的周期、偏移、持续时间(或描述为评估时长)。
第一信息包括的时间信息也可以是非周期的，在这种情况下，第一信息包括一定时间内进行inactive AI/ML model评估的一个或者多个位置信息。具体的，作为一种实施方式，可以是由多个时间单元构成一定时长，由比特串指示该段时长内哪些位置(时间单元)可以进行AI/ML inactive model评估，1个比特对应1个时间单元，比特取值为1或者为TRUE，代表该时间单元可以用于inactive model评估(或者描述为可以中断当前服务的AI/ML model)，比特取值为0或者为FAULSE，代表该时间单元不可以用于inactive model评估(或者描述为不能中断当前服务的AI/ML model)
进一步的，终端向网络发送第五信息，第五信息指示完成模型评估，可选的，第五信息还可以包括完成评估的模型ID或者functionality标识。
引入第五信息的增益在于，上述第一信息如果包括用于评估的持续时长，考虑到不同AI/ML模型的评估需要的时间不同，为例覆盖尽可能多的场景，网络通常会配置较长的用于评估的持续时长，但对于不需要较长评估时间的模型，该配置的较长的评估允许时长是一种浪费。但网络无法获知终端何时完成某个模型的评估，通过第五信息的引入，当完成AI/ML模型的评估时，允许终端通知网络，那么网络可知，在配置的持续时间的剩余时间里，终端已恢复AI服务。
第一信息的增益在于：网络和终端对于何时进行inactive AI/ML model评估有一致的理解，在网络配置的时间位置进行由服务AI/ML model转换至inactive AI/ML model进行评估，即使AI性能下降，网络也可以确定是由于model转换导致，而且网络可以在该段时间内进行相应的调整。
网络侧行为：当在网络配置的时间内出现AI/ML性能下降，那么网络可知在该段时间内是由于进行AI/ML模型评估导致，那么网络可以接受该段时间的AI损失，或者不按AI性能需求来要求终端(可以理解为网络不需要采用额外措施)。如果在网络配置的时间之外出现AI/ML性能下降，那么网络可能认为当前AI/ML模型不适用，可能会过早的触发模型切换、激活、去激活，甚至于，网络可能会认为当前环境不适用AI，指示终端回退到非AI模式。
进一步的，终端接收网络发送的第二信息，第二信息指示终端是/否可以在测量间隔内进行inactive AI/ML model评估。如果第二信息指示终端可以采用测量间隔进行inactive AI/ML model评估，那么可以不发送上述第一信息；如果第二信息指示终端不可以采用测量间隔进行inactive AI/ML model评估，那么需要进一步发送上述第一信息指示终端在哪里进行评估。
具体的，当移动性需求不高时，比如需要测量间隔的待测频点较少时，网络可以配置终端采用测量间隔进行inactive AI/ML model评估，即通过测量时间的延长换取较小的由于AI/ML model评估带来的吞吐率损失。
引入第二信息的增益在于：网络可以不单独配置用于inactive AI/ML model评估的时间信息，可以降低信令开销，降低吞吐率损失。
方案2：终端接收网络发送的第三信息，第三信息包括配置的周期，即相比于第一信息，第三信息指示给出一个时长，比如X毫秒，或者Y秒。网络不指定终端在该时长内进行AI/ML model评估的具体位置，由终端自主确定。但需要在协议预先规定在该时间范围内允许的最大的系统损失。比如，在X毫秒或Y秒内允许的最大ACK/NACK丢失率为Z％。
引入第三信息的增益为：相比于第一信息，留给终端更多的自由度确定如何进行inactive AI/ML model评估。同时，考虑到inactive AI/ML model评估带来的吞吐率损失，通过指定约定时长内允许的数据丢失率进行约束，实现自由度和吞吐率损失的trade-off。
方案3：终端向网络发送第四信息，第四信息包括终端期望的进行inactive AI/ML model评估的时间信息，包括周期、具体评估位置、评估时长等。网络可以参考该第四信息进行第一信息、第二信息、第三信息的配置。
第四信息的增益：网络由于无法获知终端情况导致配置不是最佳，终端可以根据自身所处的环境情况确定如何进行评估，并将该信息通过第四信息上报给网络，辅助网络进行配置。
本公开中，终端接收网络发送的第一信息，第一信息指示终端进行inactive model评估的时间信息，具体的，该时间信息可以是一定时长内的一段时间，终端只在该段时间内进行inactive AI/ML model的评估。第一信息使能网络和终端对于何时进行inactive AI/ML model评估有一致的理解，可以辅助网络进行相关决策，比如，在网络配置的时间位置进行由服务AI/ML model转换至inactive AI/ML model进行评估，即使AI性能下降，网络也可以确定是由于在该段时间内进行AI/ML模型评估导致，那么网络可以接受该段时间的AI损失，或者不按AI性能需求来要求终端(可以理解为网络不需要采用额外措施)；如果在网络配置的时间之外出现AI/ML性能下降，那么网络可以认为当前AI/ML模型不适用，可能会触发模型切换、激活、去激活，甚至于，网络可能会认为当前环境不适用AI，指示终端回退到非AI模式。
本公开中，终端接收网络发送的第二信息，第二信息指示终端是/否可以在测量间隔内进行inactive AI/ML model评估。如果第二信息指示终端可以采用测量间隔进行inactive AI/ML model评估，那么可以不发送上述第一信息；如果第二信息指示终端不可以采用测量间隔进行inactive AI/ML model评估，那么需要进一步发送上述第一信息指示终端在哪里进行评估。第二信息使能网络可以不单独配置用于inactive AI/ML model评估的时间信息，可以降低信令开销，降低inactive model评估带来的损失。
本公开中，终端接收网络发送的第三信息，第三信息包括配置的周期、一定时间内允许的inactive model评估带来的损失的最大值。相比于第一信息，第三信息只是给出一个时长，比如X毫秒，或者Y秒，网络不指定终端在该时长内进行AI/ML model评估的具体位置，由终端自主确定。但需要规定在该时间范围内允许的最大的系统损失(比如，在X毫秒或Y秒内允许的最大ACK/NACK丢失率为Z％)，终端进行inactive model评估带来的损失不能超过第三信息中的最大值。相比于第一信息，第三信息留给终端更多的自由度确定如何进行inactive AI/ML model评估。同时，考虑到inactive AI/ML model评估带来的性能损失，通过指定约定时长内允许的最大的丢失率进行约束，实现自由度和性能损失的trade-off。
本公开中，终端向网络发送第四信息，第四信息包括终端期望的进行inactive AI/ML model评估的时间信息，包括周期、具体评估位置、评估时长等。网络可以参考该第四信息进行第一信息、第二信息、第三信息的配置。
在评估inactive model性能时，需要中断当前正在应用的AI/ML模型，转换至inactive模型进行评估，该转换过程和评估过程会导致AI服务的中断。而且如果由终端任意决定何时进行inactive model评估，那么当出现AI/ML性能下降的时候，网络无法确定该问题是由于当前AI/ML model不匹配导致，还是由于终端切换至inactive model导致，网络可能会做出错误的决策，带来系统性能的下降。
为解决上述问题，本公开提出一种解决方法，具体的：终端接收网络发送的第一信息，第一信息包括终端进行inactive model评估的时间信息，具体的，该时间信息可以是一定时长内的一段时间，终端只在该段时间内进行inactive AI/ML model的评估(也可以包括由当前服务的AI/ML model转换至inactive AI/ML model的转换时间、由inactive AI/ML model转换至原服务的AI/ML model的转换时间)。进一步的，终端接收网络发送的第二信息，第二信息指示终端是/否可以在测量间隔内进行inactive AI/ML model评估。可选的，终端接收网络配置的第三信息，第三信息只包括配置的周期，比如X毫秒，或者Y秒，同时在协议预先规定在该时间范围内允许的最大的ACK/NACK丢失率。允许终端向网络发送第四信息，第四信息包括终端期望的进行inactive AI/ML model评估的时间信息，包括周期、具体评估位置、评估时长等。
为了实现本公开实施例的方法，本公开实施例还提供了一种模型和/或功能评估装置300，设置在第一设备上，如图3所示，图3为本公开实施例一种模型和/或功能评估装置的结构示意图；所述装置300包括以下至少之一：
评估单元301，用于在第一时间内进行模型和/或功能评估；
性能单元302，用于确定性能损失小于或者等于第一门限。
在一实施例中，所述模型包括以下之一：
去激活模型inactive model；
人工智能AI和/或机器学习ML模型；
去激活的AI和/或ML模型；和/或，
所述功能包括以下之一：
去激活功能inactive functionality；
人工智能AI和/或机器学习ML功能；
去激活的AI和/或ML functionality。
在一实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
在一实施例中，所述装置300还包括接收单元，用于接收第一消息；所述第一消息指示以下至少之一：
所述第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
在一实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
在一实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
在一实施例中，所述装置300还包括发送单元，用于发送第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
在一实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型索引ID；
功能索引ID。
在一实施例中，所述接收单元，还用于接收第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
在一实施例中，所述确定性能损失小于或者等于第一门限包括以下之一：
确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限；
在第三时间内的确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限。
在一实施例中，所述发送单元，还用于发送第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
为了实现本公开实施例第二设备侧的方法，本公开实施例还提供了一种模型和/或功能评估装置，设置在第二设备上，如图4所示，图4为本公开实施例又一种模型和/或功能评估装置的结构示意图，该装置400包括：
发送单元401，用于向第一设备发送第一消息；所述第一消息指示以下至少之一：
第一时间的周期；
所述第一时间的偏移；
所述第一时间的长度。
在一实施例中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
在一实施例中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
在一实施例中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
在一实施例中，所述装置400还包括接收单元，用于接收所述第一设备发送的第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
在一实施例中，所述第二信息包括以下至少之一：
完成评估的指示信息；
模型身份标识号ID；
功能身份标识号ID。
在一实施例中，所述发送单元401，用于向所述第一设备发送第三信息；
所述第三信息包括以下至少之一：
指示所述第一设备在测量间隔内进行model的评估；
指示所述第一设备在测量间隔内进行functionality的评估；
指示所述第一设备不在测量间隔内进行model的评估；
指示所述第一设备不在测量间隔内进行functionality的评估。
在一实施例中，所述接收单元，还用于接收所述第一设备发送的第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
所述第四时间信息包括以下至少之一：
所述model和/或功能评估的周期；
所述model和/或功能评估的偏移；
所述model和/或功能评估的时长。
需要说明的是：上述实施例提供的模型和/或功能评估装置在进行模型和/或功能评估时，仅以上述各程序模块的划分进行举例说明，实际应用中，可以根据需要而将上述处理分配由不同的程序模块完成，即将装置的内部结构划分成不同的程序模块，以完成以上描述的全部或者部分处理。另外，上述实施例提供的模型和/或功能评估装置与模型和/或功能评估方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
基于上述程序模块的硬件实现，本公开实施例还提供了一种模型和/或功能评估设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述第一设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤。
对应地，本公开实施例提供一种计算机程序产品，包括计算机程序，其上存储有计算机程序，该计算机程序被处理器执行时实现上述第一设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤。
对应地，本公开实施例提供一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被处理器执行时实现上述第一设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤；或者，所述处理器执行所述程序时实现上述第二设备侧任一项所述方法的步骤。
这里需要指出的是：以上存储介质和设备实施例的描述，与上述方法实施例的描述是类似的，具有同方法实施例相似的有益效果。对于本公开存储介质和设备实施例中未披露的技术细节，请参照本公开方法实施例的描述而理解。
需要说明的是，所述模型和/或功能评估设备可以为第一设备；图5为本公开实施例第一设备的结构示意图，如图5所示，该第一设备500包括：第一处理器501和第一存储器503，可选地，所述第一设备500还可以包括第一通信接口502。
可以理解，第一存储器503可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM，ferromagnetic random access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random Access Memory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本公开实施例描述的第一存储器503旨在包括但不限于这些和任意其它适合类型的存储器。
上述本公开实施例揭示的方法可以应用于第一处理器501中，或者由第一处理器501实现。第一处理器501可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过第一处理器501中的硬件的集成逻辑电路或者软件形式的指令完成。上述的第一处理器501可以是通用处理器、数字信号处理器(DSP，Digital Signal Processor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。第一处理器501可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本公开实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于第一存储器503，第一处理器501读取第一存储器503中的信息，结合其硬件完成前述方法的步骤。
需要说明的是，所述模型和/或功能评估设备可以为第二设备；图6为本公开实施例中第二设备的结构示意图，如图6所示，该第二设备600包括：第二处理器601和第二存储器603，可选地，所述第二设备600还可以包括第二通信接口602。
可以理解，第二存储器603可以是易失性存储器或非易失性存储器，也可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(ROM，Read Only Memory)、可编程只读存储器(PROM，Programmable Read-Only Memory)、可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)、电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)、磁性随机存取存储器(FRAM，ferromagnetic random access memory)、快闪存储器(Flash Memory)、磁表面存储器、光盘、或只读光盘(CD-ROM，Compact Disc Read-Only Memory)；磁表面存储器可以是磁盘存储器或磁带存储器。易失性存储器可以是随机存取存储器(RAM，Random Access Memory)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(SRAM，Static Random Access Memory)、同步静态随机存取存储器(SSRAM，Synchronous Static Random Access Memory)、动态随机存取存储器(DRAM，Dynamic Random Access Memory)、同步动态随机存取存储器(SDRAM，Synchronous Dynamic Random Access Memory)、双倍数据速率同步动态随机存取存储器(DDRSDRAM，Double Data Rate Synchronous Dynamic Random Access Memory)、增强型同步动态随机存取存储器(ESDRAM，Enhanced Synchronous Dynamic Random Access Memory)、同步连接动态随机存取存储器(SLDRAM，SyncLink Dynamic Random Access Memory)、直接内存总线随机存取存储器(DRRAM，Direct Rambus Random Access Memory)。本公开实施例描述的第二存储器603旨在包括但不限于这些和任意其它适合类型的存储器。
上述本公开实施例揭示的方法可以应用于第二处理器601中，或者由第二处理器601实现。第二处理器601可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过第二处理器601中的硬件的集成逻辑电路或者软件形式的指令完成。上述的第二处理器601可以是通用处理器、数字信号处理器(DSP，Digital Signal Processor)，或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。第二处理器601可以实现或者执行本公开实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者任何常规的处理器等。结合本公开实施例所公开的方法的步骤，可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于存储介质中，该存储介质位于第二存储器603，第二处理器601读取第二存储器603中的信息，结合其硬件完成前述方法的步骤。
为了实现本公开实施例提供的方法，本公开实施例还提供了一种模型和/或功能评估系统，如图7所示，图7为本公开实施例模型和/或功能评估系统结构示意图，该系统包括：第一设备701、第二设备702。
这里，需要说明的是：第一设备701、第二设备702的具体处理过程已在上文详述，这里不再赘述。
在示例性实施例中，设备可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、DSP、可编程逻辑器件(PLD，Programmable Logic Device)、复杂可编程逻辑器件(CPLD，Complex Programmable Logic Device)、现场可编程门阵列(FPGA，Field-Programmable Gate Array)、通用处理器、控制器、微控制器(MCU，Micro Controller Unit)、微处理器(Microprocessor)、或其他电子元件实现，用于执行前述方法。
应理解，说明书通篇中提到的“一个实施例”或“一实施例”意味着与实施例有关的特定特征、结构或特性包括在本公开的至少一个实施例中。因此，在整个说明书各处出现的“在一个实施例中”或“在一实施例中”未必一定指相同的实施例。此外，这些特定的特征、结构或特性可以任意适合的方式结合在一个或多个实施例中。应理解，在本公开的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本公开实施例的实施过程构成任何限定。上述本公开实施例序号仅仅为了描述，不代表实施例的优劣。
需要说明的是，在本公开中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。
本公开所提供的几个方法实施例中所揭露的方法，在不冲突的情况下可以任意组合，得到新的方法实施例。
本公开所提供的几个产品实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的产品实施例。
本公开所提供的几个方法或设备实施例中所揭露的特征，在不冲突的情况下可以任意组合，得到新的方法实施例或设备实施例。
以上所述，仅为本公开的实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以所述权利要求的保护范围为准。
需要说明的是：“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。
另外，本公开实施例所记载的技术方案之间，在不冲突的情况下，可以任意组合。
以上所述，仅为本公开的较佳实施例而已，并非用于限定本公开的保护范围。

Claims (24)

  1. 一种模型和/或功能评估方法，由第一设备执行，所述方法包括以下至少之一：
    在第一时间内进行模型和/或功能评估；
    确定性能损失小于或者等于第一门限。
  2. 根据权利要求1所述的方法，其中，所述模型包括以下之一：
    去激活模型inactive model；
    人工智能AI和/或机器学习ML模型；
    去激活的AI和/或ML模型；和/或，
    所述功能包括以下之一：
    去激活功能inactive functionality；
    人工智能AI和/或机器学习ML功能；
    去激活的AI和/或ML functionality。
  3. 根据权利要求1或2所述的方法，其中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
    所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
  4. 根据权利要求1所述的方法，其中，所述方法还包括：
    接收第一消息；所述第一消息指示以下至少之一：
    所述第一时间的周期；
    所述第一时间的偏移；
    所述第一时间的长度。
  5. 根据权利要求1或2所述的方法，其中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
  6. 根据权利要求1或2所述的方法，其中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
  7. 根据权利要求1或2所述的方法，其中，所述方法还包括：
    发送第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
  8. 根据权利要求7所述的方法，其中，所述第二信息包括以下至少之一：
    完成评估的指示信息；
    模型索引ID；
    功能索引ID。
  9. 根据权利要求1所述的方法，其中，所述方法还包括：
    接收第三信息；
    所述第三信息包括以下至少之一：
    指示所述第一设备在测量间隔内进行model的评估；
    指示所述第一设备在测量间隔内进行functionality的评估；
    指示所述第一设备不在测量间隔内进行model的评估；
    指示所述第一设备不在测量间隔内进行functionality的评估。
  10. 根据权利要求1所述的方法，其中，所述确定性能损失小于或者等于第一门限包括以下之一：
    确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限；
    在第三时间内的确认字符ACK和/或否定确认字符NACK的丢失率小于或者等于所述第一门限。
  11. 根据权利要求1所述的方法，其中，所述方法还包括：
    发送第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
    所述第四时间信息包括以下至少之一：
    所述model和/或功能评估的周期；
    所述model和/或功能评估的偏移；
    所述model和/或功能评估的时长。
  12. 一种模型和/或功能评估方法，由第二设备执行，所述方法包括：
    向第一设备发送第一消息；所述第一消息指示以下至少之一：
    第一时间的周期；
    所述第一时间的偏移；
    所述第一时间的长度。
  13. 根据权利要求12所述的方法，其中，所述第一时间包括服务的model转换至inactive model的时间，model的评估时间以及inactive model转换至服务的model的时间中的至少之一；和/或，
    所述第一时间包括服务的functionality转换至inactive functionality的时间，functionality的评估时间以及inactive functionality转换至服务的functionality的时间中的至少之一。
  14. 根据权利要求12所述的方法，其中，由至少一个时间单元构成所述第一时间，通过比特信息指示所述第一时间内第一时间单元是或否进行model和/或功能的评估；所述第一时间单元为所述至少一个时间单元中的任意时间单元；所述比特信息包括至少一个比特；其中，一个比特对应一个时间单元；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间单元用于model和/或功能的评估；所述第二值表征所述第一时间单元不用于model和/或功能的评估。
  15. 根据权利要求12所述的方法，其中，由至少一个所述第一时间构成第二时间，采用比特信息指示所述第二时间内第一时间是或否进行model和/或功能的评估；所述第一时间为所述至少一个第一时间中的任意一个；所述比特信息包括至少一个比特；其中，一个比特对应一个第一时间；所述比特的值包括第一值和/或第二值；所述第一值表征所述第一时间用于model和/或功能的评估；所述第二值表征所述第一时间不用于model和/或功能的评估。
  16. 根据权利要求12所述的方法，其中，所述方法还包括：
    接收所述第一设备发送的第二信息；所述第二信息用于指示所述第一设备完成所述模型的评估；和/或，所述第二信息用于指示所述第一设备完成所述功能的评估。
  17. 根据权利要求16所述的方法，其中，所述第二信息包括以下至少之一：
    完成评估的指示信息；
    模型身份标识号ID；
    功能身份标识号ID。
  18. 根据权利要求12所述的方法，其中，所述方法还包括：
    向所述第一设备发送第三信息；
    所述第三信息包括以下至少之一：
    指示所述第一设备在测量间隔内进行model的评估；
    指示所述第一设备在测量间隔内进行functionality的评估；
    指示所述第一设备不在测量间隔内进行model的评估；
    指示所述第一设备不在测量间隔内进行functionality的评估。
  19. 根据权利要求12所述的方法，其中，所述方法还包括：
    接收所述第一设备发送的第四信息；所述第四信息包括所述第一设备进行model和/或功能评估的第四时间信息；
    所述第四时间信息包括以下至少之一：
    所述model和/或功能评估的周期；
    所述model和/或功能评估的偏移；
    所述model和/或功能评估的时长。
  20. 一种模型和/或功能评估装置，设置在第一设备，所述装置包括以下至少之一：
    评估单元，用于在第一时间内进行模型和/或功能评估；
    性能单元，用于确定性能损失小于或者等于第一门限。
  21. 一种模型和/或功能评估装置，设置在第二设备，所述装置包括：
    发送单元，用于向第一设备发送第一消息；所述第一消息指示以下至少之一：
    第一时间的周期；
    所述第一时间的偏移；
    所述第一时间的长度。
  22. 一种模型和/或功能评估设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，其中，所述处理器执行所述程序时实现权利要求1至11任一项所述方法的步骤；或者，所述处理器执行所述程序时实现权利要求12至19任一项所述方法的步骤。
  23. 一种计算机程序产品，包括计算机程序，其中，所述计算机程序在被处理器执行时实现权利要求1至11任一项所述方法的步骤；或者，所述计算机程序在被处理器执行时实现权利要求12至19所述方法的步骤。
  24. 一种存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现权利要求1至11任一项所述方法的步骤，或者实现权利要求12至19所述方法的步骤。
PCT/CN2025/104096 2024-06-28 2025-06-26 模型和/或功能评估方法、装置、设备、产品及存储介质 Pending WO2026002170A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410866496.4 2024-06-28
CN202410866496.4A CN121235143A (zh) 2024-06-28 2024-06-28 模型和/或功能评估方法、装置、设备、产品及存储介质

Publications (1)

Publication Number Publication Date
WO2026002170A1 true WO2026002170A1 (zh) 2026-01-02

Family

ID=98149887

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/104096 Pending WO2026002170A1 (zh) 2024-06-28 2025-06-26 模型和/或功能评估方法、装置、设备、产品及存储介质

Country Status (2)

Country Link
CN (1) CN121235143A (zh)
WO (1) WO2026002170A1 (zh)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023206445A1 (zh) * 2022-04-29 2023-11-02 富士通株式会社 Ai监测装置以及方法
WO2024031605A1 (en) * 2022-08-12 2024-02-15 Qualcomm Incorporated Protocols and signaling for artificial intelligence and machine learning model performance monitoring
CN117744840A (zh) * 2022-09-15 2024-03-22 Oppo广东移动通信有限公司 模型推理性能评估方法、装置、电子设备及存储介质
CN118120327A (zh) * 2022-07-12 2024-05-31 汉阳大学校产学协力团 用于在无线通信网络中使用人工智能/机器学习模型的方法和装置

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023206445A1 (zh) * 2022-04-29 2023-11-02 富士通株式会社 Ai监测装置以及方法
CN118120327A (zh) * 2022-07-12 2024-05-31 汉阳大学校产学协力团 用于在无线通信网络中使用人工智能/机器学习模型的方法和装置
WO2024031605A1 (en) * 2022-08-12 2024-02-15 Qualcomm Incorporated Protocols and signaling for artificial intelligence and machine learning model performance monitoring
CN117744840A (zh) * 2022-09-15 2024-03-22 Oppo广东移动通信有限公司 模型推理性能评估方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN121235143A (zh) 2025-12-30

Similar Documents

Publication Publication Date Title
CN113271632B (zh) 信息的上报、接收方法、终端及网络设备
CN117615395A (zh) 模型处理方法、系统、装置、基站、终端和存储介质
WO2024087217A1 (zh) 一种模型性能监测方法、装置、设备及存储介质
CN115150903B (zh) 波束切换方法、装置及存储介质
WO2021227715A1 (zh) 候选波束测量方法、终端、网络设备、芯片系统及介质
CN112291821B (zh) 信息配置方法、装置、相关设备及存储介质
WO2024008041A1 (zh) 一种通信方法、装置、设备和计算机存储介质
WO2026002170A1 (zh) 模型和/或功能评估方法、装置、设备、产品及存储介质
CN113676929A (zh) 候选波束测量方法、终端、网络设备、芯片系统及介质
WO2023071820A1 (zh) 一种信息传输方法、装置、通信设备和存储介质
KR102434149B1 (ko) 통신 방법, 네트워크 기기 및 단말 기기
JP7766826B2 (ja) 情報報告方法及びその装置、端末、基地局、並びに記憶媒体
CN112543476A (zh) 一种信息处理方法、网络节点和终端
CN115988523A (zh) 一种波束管理方法、装置、通信设备和存储介质
CN121888401A (zh) 模型管理方法、装置、通信设备、存储介质和程序产品
CN118250753A (zh) 测量方法、装置、终端、网络设备及存储介质
WO2025232528A1 (zh) 信息传输方法、装置、相关设备、存储介质及计算机程序产品
WO2025001885A1 (zh) 信息发送和信息接收方法及装置
CN119729529A (zh) 数据处理方法、装置、设备及存储介质
WO2025161983A1 (zh) 通信方法、装置、存储介质、程序产品和通信设备
CN111526521B (zh) 一种数据传输方法和节点、终端
CN121396235A (zh) 信息处理方法、装置、计算机设备、存储介质和程序产品
CN117835354A (zh) 通信方法、装置、网络设备、终端及存储介质
WO2024074132A1 (zh) 一种信息处理方法、装置、通信设备和存储介质
WO2026032155A1 (zh) 通信方法、装置、存储介质和程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25826241

Country of ref document: EP

Kind code of ref document: A1