CN117743796B

CN117743796B - Instruction set automatic quality check method and system based on investment annotation data

Info

Publication number: CN117743796B
Application number: CN202311777746.9A
Authority: CN
Inventors: 武悦娇; 任君翔; 喻滨; 秦久芳; 夏杨
Original assignee: Pacific Ocean Asset Management Co ltd
Current assignee: Pacific Ocean Asset Management Co ltd
Priority date: 2023-12-21
Filing date: 2023-12-21
Publication date: 2024-06-18
Anticipated expiration: 2043-12-21
Also published as: CN117743796A

Abstract

The disclosure provides an instruction set automatic quality check method, an instruction set automatic quality check system, an instruction set automatic quality check equipment and a storage medium based on investment annotation data. An instruction set automatic quality check method based on investment annotation data comprises the following steps: evaluating the instruction set according to a first dimension factor and obtaining a first score; evaluating the instruction set according to a second dimension factor and obtaining a second score; determining weights of different dimension factors through a first function according to the first score and the second score; and determining a second function according to the first function, and obtaining the quality check score of the instruction set through the second function. The method and the device can save the cost of the traditional manual checking instruction set, improve the efficiency and accuracy of the checking process and improve the user experience.

Description

Instruction set automatic quality check method and system based on investment annotation data

Technical Field

The disclosure relates to the technical field of artificial intelligence, in particular to an instruction set automatic quality check method, an instruction set automatic quality check system, an instruction set automatic quality check equipment and a storage medium based on investment annotation data.

Background

In the current technical environment, the end-to-end instruction quality assessment scheme is not fully mature, and still relies on manual or semi-automatic methods for quality assessment. Although significant advances have been made in the areas of artificial intelligence and machine learning, certain challenges remain in the specific task of instruction quality assessment. Mainly because conventional text classification tasks often have difficulty in achieving accurate assessment of the quality of the abstraction, let alone obtaining objective model scores through linear models.

Tag-based methods may encounter difficulties for instruction quality assessment. This is because the quality of instructions tends to be abstract and not easily described by simple labels. Thus, conventional text classification tasks may not accurately evaluate the quality of such abstractions.

Even if some method can be used to define and quantify the quality of the instructions, objective model scoring may not be obtained by a linear model. Since linear models generally assume that there is a direct linear relationship between input and output. However, this assumption may not hold for instruction quality assessment. The quality of an instruction may be affected by a number of factors, including the content of the instruction, the execution environment, the behavior of the user, and so on. The relationship between these factors may be nonlinear and may even be a complex network structure. Thus, evaluating the quality of an instruction by a linear model may yield inaccurate results.

Disclosure of Invention

The present disclosure has been made to solve the above-mentioned problems, and an object of the present disclosure is to provide an instruction set automatic quality verification method, system, device, and storage medium based on investment annotation data, which can obtain more accurate investment instructions in application in investment fields, thereby more accurately understanding and responding to human queries, and realizing interaction effects highly consistent with humans.

The present disclosure provides this summary section to introduce concepts in a simplified form that are further described below in the detailed description section. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In order to solve the above technical problems, an embodiment of the present disclosure provides an instruction set automatic quality verification method based on investment annotation data, which adopts the following technical scheme, including:

evaluating the instruction set according to a first dimension factor and obtaining a first score;

evaluating the instruction set according to a second dimension factor and obtaining a second score;

determining weights of different dimension factors through a first function according to the first score and the second score;

determining a second function according to the first function, and obtaining a quality check score of the instruction set through the second function;

Wherein the second score is inversely proportional to a quality check score of the instruction set and the second function is proportional to a logarithmic loss.

In order to solve the above technical problems, an embodiment of the present disclosure further provides an instruction set automatic quality check system based on investment labeling data, which adopts the following technical scheme, including:

the first score module is used for evaluating the instruction set according to a first dimension factor and obtaining a first score;

The second score module is used for evaluating the instruction set according to a second dimension factor and obtaining a second score;

the first function module is used for determining weights of different dimension factors through a first function according to the first fraction and the second fraction;

the second function module is used for determining a second function according to the first function and obtaining the quality check score of the instruction set through the second function;

In order to solve the above technical problems, an embodiment of the present disclosure further provides a computer device, which adopts the following technical solutions, including:

A memory and a processor, the memory having stored therein a computer program, the processor implementing the method as described above when executing the computer program.

In order to solve the above technical problems, an embodiment of the present disclosure further provides a computer readable storage medium, which adopts the following technical solutions, including:

the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements a method as described in the foregoing.

According to the technical scheme disclosed by the disclosure, compared with the prior art, the method and the device have the advantages that the cost of a traditional manual checking instruction set is remarkably saved, meanwhile, the efficiency and the accuracy of a checking process are improved, and the user experience is improved.

Drawings

FIG. 1 is an exemplary system architecture diagram to which the present disclosure may be applied;

FIG. 2 is a flow chart of one embodiment of an instruction set automated quality check method according to the present disclosure;

FIG. 3 is a schematic diagram of one embodiment of an instruction set automated quality check system in accordance with the present disclosure;

fig. 4 is a structural schematic diagram of one embodiment of a computer device according to the present disclosure.

The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by reference to the following detailed description when taken in conjunction with the accompanying drawings. The same or similar reference numbers will be used throughout the drawings to refer to the same or like elements. It should be understood that the figures are schematic and that elements and components are not necessarily drawn to scale.

Detailed Description

Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure; the terms "comprising" and "having" and any variations thereof in the description and claims of the present disclosure and in the description of the figures above are intended to cover a non-exclusive inclusion. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present disclosure. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.

In order to better understand the present disclosure, a technical solution in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[ System Structure ]

First, a structure of a system of one embodiment of the present disclosure is explained. As shown in fig. 1, the system architecture 100 may include, for example, terminal devices 101, 102, 103, 104, a network 105, and a server 106. The network 105 serves as a medium for providing communication links between the terminal devices 101, 102, 103, 104 and the server 106.

In this embodiment, an electronic device (for example, the terminal device 101, 102, 103, or 104 shown in fig. 1) on which the instruction set automation quality check method operates may perform transmission of various information through the network 105. The network 105 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others. It should be noted that the wireless connection may include, but is not limited to, 3G/4G/5G connections, wi-Fi connections, bluetooth connections, wiMAX connections, zigbee connections, UWB connections, local area networks ("LANs"), wide area networks ("WANs"), the internet (e.g., the internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as other now known or later developed network connections. The network 105 may communicate using any currently known or future developed network protocol, such as HTTP (Hyper Text Transfer Protocol ), and may be interconnected with digital data communications (e.g., communication networks) in any form or medium.

The user may interact with the server 106 via the network 105, for example using the terminal devices 101, 102, 103, 104, to receive or send messages etc. The terminal device 101, 102, 103 or 104 may have various client applications installed thereon, such as a video live and play class application, a web browser application, a shopping class application, a search class application, an instant messaging tool, a mailbox client, social platform software, and the like, for example.

The terminal device 101, 102, 103 or 104 may be, for example, various electronic devices with a touch display screen and/or supporting web browsing, including but not limited to a smart phone, a tablet computer, an electronic book reader, an MP3 (moving picture experts compression standard audio layer 3) player, an MP4 (moving picture experts compression standard audio layer 4) player, a head mounted display device, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PMP (portable multimedia player), a mobile terminal such as a car navigation terminal, etc., a mobile terminal such as a digital TV, a desktop computer, etc.

The server 106 may be, for example, a server providing various services, such as a background server providing support for pages displayed or data transmitted on the terminal device 101, 102, 103, or 104.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

Here, the terminal device may implement the method of the embodiment of the present disclosure, for example, independently or by running applications of various operating systems in cooperation with other electronic terminal devices.

[ Instruction set automated quality check method ]

In the development of modern Large Language Models (LLMs), knowledge of the model is mainly obtained in a pre-training phase, while instructions play a role in guiding the model to follow a specific interaction pattern when interacting with a user. The quality of these instructions is therefore critical to the ability of the guided language model to generate responses in a particular manner. Based on this understanding, the present disclosure proposes a set of hypothetical frameworks for instruction quality assessment.

The evaluation of instruction quality is based, for example, mainly on two dimensions: the first dimension is, for example, characteristics of the instruction itself, such as length, clarity, and relevance to the service of the instruction; the second dimension is, for example, the value of the instruction to the model itself in the Supervisory Fine Tuning (SFT) stage.

In the first dimension, the verification of the quality of the instruction will be based on the sharpness factor and the business relevance factor of the instruction. To accurately define the sharpness factor, consider, for example, the sharpness factor of an instruction from six aspects of question specificity, answerability, disambiguation, clarity, targeting, and conciseness. For the business relevance factor, the evaluation is made, for example, from the two viewpoints of the practicality of the problem in investment business and the relevance of the investment scenario.

In the second dimension, the value of the SFT phase instruction on the model itself, the difficulty and effectiveness of the instruction is evaluated, for example, by calculating the difference between the response capability of the model to the instruction after undergoing a round of fine-tuning and before fine-tuning. Finally, these factors are integrated by employing a linear function to obtain a more accurate command quality check score.

The high-quality instruction obtained by the scheme provided by the invention can obviously improve the accuracy of the SFT model in a downstream task, and further accurately reflect the performance of the large model in a specific field such as the investment business field.

Referring to fig. 2, a flow chart of one embodiment of an instruction set automated quality check method according to the present disclosure is shown. The automatic quality check method of the instruction set comprises the following steps:

s21, evaluating the instruction set according to the first dimension factor and obtaining a first score; here, the first dimension factor comprises at least a sharpness factor and/or a traffic correlation factor, for example.

In order to accurately define the sharpness factor, a set of refined evaluation criteria may be employed, including, for example, the specificity, the answerability, the disambiguation, the clarity, the targeting, and the conciseness of the question. Each indicator aims to evaluate the sharpness and validity of an instruction from a different perspective.

For example, specificity focuses on the degree of detail and level of specificity of the question, whether a answers can be given to the questions by an answer assessment, no ambiguity examines the potential ambiguity in the question presentation, explicitly emphasizes the clarity and certainty of the question, target targeting focuses on whether the question is focused on a specific purpose or target, and conciseness emphasizes the question being presented in a succinctly understood manner.

In the evaluation of the traffic correlation factor, two main aspects are considered, for example: firstly, the practicability of the problem in investment business, namely whether the problem is closely related to actual business scenes such as investment decision, market analysis and the like; and secondly, the relevance of the problem to a specific investment scene, namely whether the problem is related to a specific environment or situation of the investment field or not.

To implement this evaluation framework, for example, specific criteria for each of the above-described attributes first need to be explicitly defined. Subsequently, a series of hints are designed, using a large language model to determine whether the instruction meets a given attribute requirement. This process involves a careful evaluation of each instruction to determine if they meet the criteria of each tag. In this way, the present disclosure may obtain detailed information about the instruction meeting each attribute requirement, thereby providing a solid basis for a comprehensive assessment of instruction quality.

In one or more embodiments, for example, setting the following indexes, evaluating the instruction value from the aspect that whether the instruction has the service correlation of the asset management or investment service and the instruction semantics, further splicing each instruction of quality to be confirmed and the instruction examples described below, inputting the spliced content into a large language model, and then obtaining the score of each instruction of quality to be confirmed on the following indexes, for better integrating the effects of different scores, for example, training a linear function, so that y of the linear function is used as a high quality label in marked data, and each x is the score on each index output by the large model.

S22, evaluating the instruction set Q according to a second dimension factor and obtaining a second score;

In one or more embodiments, evaluating the instruction set according to a second dimension factor and obtaining a second score, for example, includes:

The initial model is trained to obtain a pre-trained model based on a small amount of investment annotation data, where the small amount of investment annotation data refers to a situation where a small amount of data is required to train the model, such as in the field of investment business-related machine learning and artificial intelligence, which may include market quotations, corporate financial data, transaction data, and the like. In one or more embodiments, the initial model is, for example, trained for at least one cycle to obtain a pre-trained model, for example, trained for only 1 cycle to obtain a brief pre-trained model.

Then determining a first loss fraction of the instruction set according to the pre-training model; for samples to be added, using the pre-trained model described above, the loss of predicting the next text unit for a given instruction set Q is calculated to help evaluate the performance of the model and optimize it. For example using a cross entropy loss function. To measure the difference between two probability distributions, which can be used here to measure the difference between the model predicted text unit and the actual text unit:

Where N is the number of text units of the real answer A, this average cross entropy loss is expressed as a conditional answer score. Σ represents summing all possible text units, log represents natural logarithms.

Thirdly, determining a second loss fraction of the instruction set according to the initial model; i.e. further calculate the loss score for a large model without training pre-trained:

the data described above is used to measure the ability of the LLM to generate this answer alone. The index measures the inherent difficulties or challenges presented by the answer alone, without contextual guidance of the corresponding instruction. A higher direct answer score may indicate that the model generated answer is inherently more challenging or complex.

Finally, a second score is determined based on the first and second loss scores, wherein the second score is, for example, a predictive reasoning loss to evaluate instruction quality, which is inversely proportional to the quality check score of the instruction set.

Considering that the way LLM is fine-tuned is relatively single, a set of natural language indicators is used to predict the loss of reasoning. The specific indexes are as follows:

In this case, the effect of the LLM itself on the ability to fit answer strings is partially mitigated. The score measures how well a given instruction favors alignment of the corresponding response. A high S-score indicates that the model cannot align the response with a given corresponding instruction, which in turn indicates the difficulty of the instruction. Here, for example, the second score is inversely proportional to the quality check score of the instruction set.

S23, determining weights of different dimension factors through a first function according to the first score and the second score;

In one or more embodiments, for each instruction data set, the sharpness factor and the score of the business relevance factor for the above instruction and the score S before and after LLM fine tuning are combined:

For example, I ₁ (D) indicates whether the instructions in instruction data set D have business relevance to the asset management or investment business itself, whether it is 1 or 0.I ₂ (D) represents whether the semantics of the instructions in instruction dataset D are clear, expressed as yes or no, 1 or 0.I ₃ (D) represents the score of the instruction dataset D before and after the fine tuning of the large language model, expressed numerically, the higher the score, the more efficient the instruction.

Using a first function, i.e. a linear function F, to obtain weights of different factors for instruction quality assessment, the first function F can be further assumed such that F (I (D)) can be used to approximate the model inference losses described aboveThe relationship between the inference loss L and these calculation indices can be expressed by the following formula:

in one or more embodiments, the value of the first function corresponds to, for example, an inference loss of the model for which the instruction set is directed.

S24, determining a second function according to the first function, and obtaining a quality check score of the instruction set through the second function; assuming that there is a second function, which is a multiple linear function proportional to the log loss, the first function is further rewritten as follows:

The parameters of the second function are solved by the least squares method such that the values of the first function minimize the inference losses corresponding to the model for which the instruction set is directed.

Specifically, the least square method includes the steps of:

Selecting an instruction data set D, calculating the values of its respective indices I _i (D), and model M's inferred losses over the evaluation data set Deval Is a value of (2).

These values are substituted into a second function where β ₀,β₁,…,β_n is an unknown parameter and ε is an error term.

Repeating the steps, and calculating different instruction data sets D to obtain a plurality of equations to form an equation set.

Solving the equation set by using a formula of a least square method to obtain an estimated value of the parameter beta ₀,β₁,…,β_n:

Where X is a matrix, each row of which is the value of each index of an instruction, Y is a vector, and each element of which is the value of the inference penalty of an instruction. Substituting these estimates into the second function, the value of the first function F is obtained as a quality assessment score for the instruction data set D.

The index employed by the present disclosure references the outcome of a large model as compared to an index using a machine model or a depth model. Furthermore, to investigate the variation in command quality, the present disclosure contrasts model performance after fine-tuning using model supervision. Compared with other methods which may need a longer training period and a lengthy evaluation flow, the scheme provided by the present disclosure has high efficiency, can quickly embody the quality level of the candidate instruction, and provides a more convenient tool.

In one or more embodiments, for example, different computational metrics may be used to measure the clarity and business relevance of the instruction data set in terms of the text, semantics, logic, emotion, style, etc. of the instruction, as well as the score before and after LLM fine-tuning. Here, for example, a linear function is not necessarily used, but a nonlinear function, such as a polynomial function, an exponential function, a logarithmic function, or the like, may be used, and it may be possible to better fit the inference loss of the model, and improve the accuracy of the evaluation. Here, the least square method is not necessarily used, and, for example, a maximum likelihood method, a gradient descent method, a genetic algorithm, or the like may be used to better optimize parameters of the function and improve flexibility of evaluation.

Here, for example, in evaluating the quality of an instruction in terms of asset management or investment business, some calculation metrics may be defined to measure the definition of the instruction and business relevance, as well as the score before and after LLM fine-tuning. The indexes can be considered from the aspects of the text, the semantics, the logic, the emotion, the style and the like of the instruction, and some indexes which can reflect the quality of the instruction are selected. For example:

I ₁ (D) instruction whether to have the asset management or the business correlation of the investment business itself, which is expressed by yes or no, and is 1 or 0.

I ₂ (D) whether the semantics of the instruction are clear, expressed as yes or no, 1 or 0.

I ₃ (D) whether the instruction meets the rules and common sense of the financial market, and whether the instruction is expressed as yes or not is 1 or not is 0.

I ₄ (D) whether the instruction contains sufficient details and parameters, expressed as yes or no, is 1 or no is 0.

Instructions with scores before and after the fine tuning of the large language model are represented by numbers, and the higher the score, the more effective the instruction is.

Then, a suitable functional form is selected for evaluating the quality of the instruction, inversely proportional to the model inference loss. For example, a functional form, such as a linear function, a nonlinear function, etc., that better fits the model inference losses may be selected based on the characteristics of the instruction data set. For example, instruction quality may be evaluated with the following nonlinear function:

where y is the quality label of the instruction, x _i is the fraction of the instruction on the ith index, and a and b are parameters of the nonlinear function. The meaning of the function is that the quality of the instruction is exponentially related to the value of each index of the instruction, and the smaller the value of each index of the instruction is, the higher the quality of the instruction is, and the smaller the reasoning loss of the model is.

Finally, a suitable parameter estimation method is selected to solve the parameters of the function. For example, a method that can better optimize parameters of the function, such as a least squares method, a maximum likelihood method, a gradient descent method, or the like, may be selected according to the form of the function and the distribution of data. For example, the parameters of the nonlinear function are solved by the gradient descent method:

Where J (a, b) is the cost function of the function, its definition is:

alpha is the learning rate, determines the speed of parameter update, m is the number of data points, and y _i and x _ij are the abscissa of the ith data point. The principle of the gradient descent method is that the value of the cost function is made smaller and smaller by continuously updating the value of the parameter until the minimum value is reached or convergence is achieved.

The method of the present disclosure has high human cognitive consistency and quality changes can be investigated by comparing the performance of different models. The present disclosure may also achieve the following technical effects:

1. Automated quality check scheme: the scheme provided by the disclosure can verify the instruction quality of the LLM in an automatic mode, and the innovation measure can remarkably reduce the requirement of relying on manual verification, so that a large amount of manpower and time resources are saved. This is particularly important for handling large instruction sets, as manual verification is not only time consuming and labor intensive, but also prone to inconsistencies in results due to artifacts.

2. Comprehensive general NLP and large model feedback: another innovation of the present disclosure is the incorporation of generic Natural Language Processing (NLP) technology and feedback from large models. This integrated approach allows the solution presented herein to outperform traditional approaches that rely solely on large model feedback in terms of efficiency. By combining the common NLP index with large model feedback, the scheme not only can evaluate the quality of the instruction more comprehensively, but also is more efficient in processing large amounts of data.

It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, these steps are not necessarily performed in order as indicated by the arrows. The steps are not strictly limited in order and may be performed in other orders, unless explicitly stated herein. Moreover, at least some of the steps in the flowcharts of the figures may include a plurality of sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, the order of their execution not necessarily being sequential, but may be performed in turn or alternately with other steps or at least a portion of the other steps or stages.

Those skilled in the art will appreciate that implementing all or part of the above-described methods in accordance with the embodiments may be accomplished by way of a computer program stored in a computer-readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. The storage medium may be a nonvolatile storage medium such as a magnetic disk, an optical disk, a Read Only Memory (ROM), or a Random Access Memory (RAM).

[ Instruction set automated quality check System ]

In order to achieve the technical solution in the embodiments of the present disclosure, an embodiment of the present disclosure provides an instruction set automation quality check system, for example, for implementing the instruction set automation quality check method, where the system may be specifically applied to various electronic terminal devices, as shown in fig. 3, including: a first score module 301, a second score module 302, a first function module 303, a second function module 304.

A first score module 301, configured to evaluate the instruction set according to a first dimension factor and obtain a first score; here, the first dimension factor comprises at least a sharpness factor and/or a traffic correlation factor, for example. The first score module 301 is, for example, used to implement the method of step S21, which is not described herein.

A second score module 302, configured to evaluate the instruction set Q according to a second dimension factor and obtain a second score; the second score module 302 is, for example, used to implement the method of step S22, which is not described herein. Here, for example, the second score is inversely proportional to the quality check score of the instruction set Q.

In one or more embodiments, the second score module 302 is further configured to: training the initial model according to a small amount of investment annotation data to obtain a pre-training model; determining a first loss fraction of the instruction set according to the pre-training model; determining a second loss fraction of the instruction set according to the initial model; the second score is determined based on the first and second loss scores.

A first function module 303, configured to determine weights of different dimension factors through a first function according to the first score and the second score; here, the first function module 303 is, for example, used to implement the method of step S23, which is not described herein.

A second function module 304, configured to determine a second function according to the first function, and obtain a quality check score of the instruction set through the second function; here, the second function module 304 is, for example, used to implement the method of step S24, which is not described herein. Here, the second function is proportional to, for example, the logarithmic loss.

It should be understood that while each block in the block diagrams of the figures may represent a module, a portion of the module contains one or more executable instructions for implementing the specified logical function(s), the modules are not necessarily sequentially executed in order. The modules and functional units in the embodiments of the apparatus in the present disclosure may be integrated into one processing module, or each unit may exist alone physically, or two or more modules or functional units may be integrated into one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented in the form of software functional modules and sold or used as a stand-alone product. The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like.

[ Instruction set automated quality check device ]

Referring now to fig. 4, a schematic diagram of an electronic device (e.g., a terminal device or server in fig. 1) 400 suitable for use in implementing embodiments of the present disclosure is shown. The terminal device in the embodiment of the present disclosure may be various terminal devices in the above-described system. The electronic device shown in fig. 4 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.

As shown in fig. 4, the electronic device 400 may include a processing means (e.g., a central processor, a graphics processor, etc.) 401 for controlling the overall operation of the electronic device. The processing means may comprise one or more processors to execute instructions to perform all or part of the steps of the methods described above. In addition, the processing device 401 may also include one or more modules for processing interactions with other devices.

The storage device 402 is used to store various types of data, and the storage device 402 may be a system, device or apparatus including various types of computer readable storage media, or a combination thereof, such as electronic, magnetic, optical, electromagnetic, infrared, or semiconductor, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

The sensor means 403 for sensing the prescribed measured information and converting it into a usable output signal according to a certain law may comprise one or more sensors. For example, it may include an acceleration sensor, a gyro sensor, a magnetic sensor, a pressure sensor, a temperature sensor, or the like for detecting changes in the on/off state, relative positioning, acceleration/deceleration, temperature, humidity, light, or the like of the electronic apparatus.

The processing means 401, the memory means 402 and the sensor means 403 are connected to each other by a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

The multimedia device 406 may include an input device such as a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc. for receiving input signals from a user, where various input devices may cooperate with various sensors of the sensor device 403 to perform gesture operation input, image recognition input, distance detection input, etc.; the multimedia device 406 may also include an output device such as a Liquid Crystal Display (LCD), speaker, vibrator, etc.

The power supply 407, which is used to provide power to various devices in the electronic apparatus, may include a power management system, one or more power supplies, and components to distribute power to other devices.

Communication means 408 may allow electronic device 400 to communicate wirelessly or by wire with other devices to exchange data.

Each of the above-described devices may also be connected to the I/O interface 405 to enable application of the electronic apparatus 400.

While an electronic device having various means is shown in the figures, it is to be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may be implemented or provided instead.

In particular, according to embodiments of the present disclosure, the processes described above with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a non-transitory computer readable medium, the computer program comprising program code for performing the method shown in the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via a communications device, or from a storage device. The above-described functions defined in the methods of the embodiments of the present disclosure are performed when the computer program is executed by a processing device.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It is noted that the computer readable medium described above in the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, fiber optic cables, RF (radio frequency), and the like, or any suitable combination of the foregoing.

The computer readable medium may be contained in the electronic device; or may exist alone without being incorporated into the electronic device.

Computer program code for carrying out operations of the present disclosure may be written in one or more programming languages, including, but not limited to, an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of remote computers, the remote computer may be connected to the user computer through any kind of network or may be connected to an external computer (e.g., connected through the internet using an internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.

The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), an Application Specific Standard Product (ASSP), a system on a chip (SOC), a Complex Programmable Logic Device (CPLD), and the like.

According to one or more embodiments of the present disclosure, there is provided an instruction set automatic quality check method based on investment annotation data, which adopts the following technical scheme, including:

Training the initial model according to the small investment annotation data to obtain a pre-training model;

determining a first loss fraction of the instruction set according to the pre-training model;

determining a second loss fraction of the instruction set from the initial model;

The second score is determined from the first and second loss scores.

Training the initial model for at least one period to obtain the pre-training model.

the value of the first function corresponds to an inference loss of the model for which the instruction set is directed.

According to one or more embodiments of the present disclosure, an automatic quality check method for an instruction set based on investment labeling data is provided, which adopts the following technical scheme, and further includes:

Solving parameters of the second function by a least squares method such that values of the first function minimize inference losses corresponding to a model for which the instruction set is directed.

The first dimension factor includes at least a sharpness factor and/or a business correlation factor.

In accordance with one or more embodiments of the present disclosure, there is provided an instruction set automated quality check system based on investment annotation data, comprising,

According to one or more embodiments of the present disclosure, there is provided an instruction set automated quality verification system based on investment annotation data, comprising, the second score module further configured to:

The second score is determined from the first and second loss scores.

According to one or more embodiments of the present disclosure, a computer device is provided, which adopts a technical solution including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a method as described in any one of the above.

According to one or more embodiments of the present disclosure, a computer readable storage medium is provided, in which a computer program is stored which, when executed by a processor, implements a method as described in any of the above.

The foregoing description is only of the preferred embodiments of the present disclosure and description of the principles of the technology being employed. It will be appreciated by persons skilled in the art that the scope of the disclosure referred to in this disclosure is not limited to the specific combinations of features described above, but also covers other embodiments which may be formed by any combination of features described above or equivalents thereof without departing from the spirit of the disclosure. Such as those described above, are mutually substituted with the technical features having similar functions disclosed in the present disclosure (but not limited thereto).

Moreover, although operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of the present disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are example forms of implementing the claims.

Claims

1. An instruction set automatic quality check method based on investment annotation data is characterized by comprising the following steps:

2. The instruction set automated quality check method of claim 1, wherein evaluating the instruction set according to a second dimension factor and obtaining a second score comprises:

Training the initial model according to a small amount of investment annotation data to obtain a pre-training model;

The second score is determined from the first and second loss scores.

3. The automated quality check method of instruction set of claim 2,

4. The automated quality check method of instruction set of claim 1,

5. The instruction set automated quality check method of claim 1, further comprising:

6. The automated quality check method of instruction set of claim 1,

7. An instruction set automated quality verification system based on investment annotation data, comprising:

8. The instruction set automated quality verification system of claim 7, wherein the second scoring module is further to:

The second score is determined from the first and second loss scores.

9. A computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the method of any of claims 1-6 when the computer program is executed.

10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the method according to any of claims 1-6.