WO2020244183A1 - 数据标注 - Google Patents

数据标注 Download PDF

Info

Publication number
WO2020244183A1
WO2020244183A1 PCT/CN2019/123406 CN2019123406W WO2020244183A1 WO 2020244183 A1 WO2020244183 A1 WO 2020244183A1 CN 2019123406 W CN2019123406 W CN 2019123406W WO 2020244183 A1 WO2020244183 A1 WO 2020244183A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
labeling
target
sample
untrusted
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/123406
Other languages
English (en)
French (fr)
Inventor
刘宇达
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Sankuai Online Technology Co Ltd
Original Assignee
Beijing Sankuai Online Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Sankuai Online Technology Co Ltd filed Critical Beijing Sankuai Online Technology Co Ltd
Publication of WO2020244183A1 publication Critical patent/WO2020244183A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • This application relates to the field of machine learning technology, in particular to data labeling in the process of machine learning.
  • Artificial intelligence technology usually involves machine learning. In the process of machine learning, there is a huge demand for training sample data, and a large amount of training sample data needs to be annotated. At present, it is usually necessary to manually label a large amount of training sample data, which makes the labeling work excessively dependent on people and the workload is huge. This consumes a lot of human resources, and the efficiency of labeling is low.
  • this application provides a method, device and electronic equipment for data labeling.
  • a method for data labeling including:
  • a data labeling device including:
  • the labeling module is used to label a plurality of data to be labelled through a pre-trained target labeling model to obtain a target set composed of labeling results corresponding to the data to be labelled;
  • a selection module configured to use a pre-trained target classifier to select untrusted data in the to-be-labeled data to verify the labeling result corresponding to the untrusted data;
  • the correction module is used to correct the annotation result corresponding to the untrusted data that has not passed the verification in the target set.
  • a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented .
  • an electronic device including a memory, a processor, and a computer program stored in the memory and capable of running on the processor.
  • the processor implements the first The method of any of the aspects.
  • Fig. 1 is a flowchart of a method for data labeling according to an exemplary embodiment of the present application
  • Fig. 2 is a flowchart of another data labeling method according to an exemplary embodiment of the present application.
  • Fig. 3 is a flowchart of another data labeling method according to an exemplary embodiment of the present application.
  • Fig. 4 is a block diagram of a data labeling device according to an exemplary embodiment of the present application.
  • Fig. 5 is a block diagram showing another data labeling device according to an exemplary embodiment of the present application.
  • Fig. 6 is a block diagram showing another data labeling device according to an exemplary embodiment of the present application.
  • Fig. 7 is a schematic structural diagram of an electronic device according to an exemplary embodiment of the present application.
  • first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other.
  • first information may also be referred to as second information, and similarly, the second information may also be referred to as first information.
  • word “if” as used herein can be interpreted as "when” or “when” or "in response to determination”.
  • Fig. 1 is a flow chart showing a method for data labeling according to an exemplary embodiment.
  • the method can be applied to a terminal device or a server.
  • the method includes the following steps:
  • step 101 a plurality of to-be-labeled data are labeled by a pre-trained target labeling model to obtain a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data.
  • a plurality of data to be labeled can be labeled by a pre-trained target labeling model to obtain a target set, which is composed of the labeling results corresponding to the data to be labeled.
  • the data to be labeled may be image-type data, sound-type data, or text-type data. It is understood that the data to be labeled may also be any other type of data. The specific type is not limited.
  • marking the data to be marked may be marking the marking target in the data to be marked, and setting a corresponding label (for example, category label, attribute label, ID (identifier) label, etc.) for the mark.
  • labeling image type data can be to mark the target object in the image type data (that is, the label target) with a label box, and set the label of the target object (for example, object category , Object attributes, object ID, etc.).
  • the target annotation model may be one model or multiple models with different structures.
  • the target annotation model may be a neural network type model.
  • the training sample data can be labelled manually, and then the target labeling model can be obtained by training using the manually labelled training sample data.
  • the target labeling model can also be continuously optimized and updated to make the target labeling model more accurate.
  • any image data to be annotated can be annotated in the following manner: first, each Each unit model is annotated in the image data; then, based on the result of each unit model being annotated in the image data, the NMS (non-maximum suppression) algorithm is used to determine one or the other of the image data Multiple labeling targets are determined, and the label corresponding to each labeling target is determined to complete the labeling of the image data.
  • the labeling target is the object to be labelled in the image data. Since multiple unit models with different structures are used for labeling, the accuracy of labeling can be improved.
  • a pre-trained target classifier is used to select untrusted data from the data to be labeled to verify the labeling result corresponding to the untrusted data.
  • a pre-trained target classifier may be used to select untrusted data from the above-mentioned data to be labeled to verify the labeling result corresponding to the untrusted data.
  • untrustworthy data can be data that is highly confusing, has insignificant features, is difficult to distinguish, is difficult to label, and is easily labeled incorrectly.
  • image type data untrustworthy data can be blurry images, or image data with large interfering objects in the image, or image data where the target object in the image is not obvious.
  • untrusted data may be sound data with large environmental noise, or sound data with too small target sound.
  • untrusted data may be text data with ambiguous semantic features.
  • the target classifier can be a pre-trained classifier. Any classifier that is known in the art and may appear in the future that can be applied to this application can be applied to this application.
  • the specific type is not limited.
  • manual classification and screening can be used to divide the training sample data into credible sample data and untrusted sample data.
  • the target classifier is trained using the training sample data filtered by manual classification.
  • the target classifier can also be continuously optimized and updated to make the target classifier more accurate.
  • the target classifier is used to select untrusted data that is easy to be labeled incorrectly from the above-mentioned data to be labeled, and then the labeling result corresponding to the untrusted data is verified to determine that the untrusted data corresponds to Is the labeling result correct?
  • the verification may be performed manually, or any other reasonable method may be used for verification, and it is understandable that this application does not limit the specific verification method.
  • the untrusted data if it is determined that the labeling result corresponding to the untrusted data is wrong, the untrusted data fails the verification. If it is determined that the labeling result corresponding to the untrusted data is correct, then the untrusted data is correct. Trusted data is verified.
  • step 103 the annotation result corresponding to the untrusted data that fails the verification in the target set is corrected.
  • the untrusted data if the non-trusted data passes the verification, no operation is performed. If the untrusted data fails the verification, the untrusted data needs to be correctly labeled again to obtain the correct labeling result. And use the correct labeling result corresponding to the untrusted data to replace the wrong labeling result corresponding to the untrusted data in the target set, so as to correct the untrusted data in the target set that has not passed the verification. Mark the result.
  • a labeling database may be established in advance, and the labeling database is used to store labeled data, and the data stored in the labeling database is used to perform machine learning training.
  • the corrected target set can be used as labeled data and stored in the labeled database.
  • the data labeling method provided by the above-mentioned embodiment of the present application labels a plurality of to-be-labeled data through a pre-trained target labeling model to obtain a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data, and use pre-trained target classification
  • the processor selects the untrusted data in the data to be labeled to verify the labeling result corresponding to the untrusted data, and correct the labeling result corresponding to the untrusted data in the target set that has not passed the verification.
  • this embodiment uses the pre-trained target labeling model to label the to-be-labeled data
  • the target classifier is used to screen out the to-be-labeled data that may be incorrectly labeled for sampling, and the incorrect labeling results are corrected.
  • the results of labeled data can be verified more specifically, and the accuracy of labeling can be improved.
  • FIG. 2 is a flowchart of another data labeling method according to an exemplary embodiment.
  • This embodiment describes the process of updating the target classifier.
  • This method can be applied to terminal devices and also Can be applied to the server. The method includes the following steps:
  • step 201 a plurality of to-be-labeled data is labeled by a pre-trained target labeling model, and a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data is obtained.
  • a pre-trained target classifier is used to select untrusted data in the data to be labeled, so as to verify the labeling result corresponding to the untrusted data.
  • step 203 the annotation result corresponding to the untrusted data that fails the verification in the target set is corrected.
  • step 204 the non-trusted data that has not passed the verification is determined as the first positive sample, and the non-trusted data that has passed the verification is determined as the first negative sample.
  • the untrustworthy data has strong confusing, inconspicuous features, difficult to distinguish, difficult to label, and easy to label errors. Therefore, if the untrusted data selected by the target classifier is incorrectly labeled, it means that the target classifier classifies the untrusted data more accurately. If the untrusted data filtered by the target classifier is labeled correctly, it means that the target classifier's classification of the untrusted data is not accurate enough. Therefore, the non-trusted data that has not passed the verification (that is, the non-trusted data that is marked incorrectly) is taken as the first positive sample, and the non-trusted data that has passed the verification (that is, the non-trusted data that is marked as correct) is determined as The first negative sample.
  • step 205 the target classifier is updated using the first positive sample and the first negative sample.
  • the first positive sample and the first negative sample may be used to retrain the target classifier to optimize the update of the target classifier, so that the target classifier can screen out more suitable untrustworthy data.
  • the data labeling method provided by the above-mentioned embodiment of the present application labels a plurality of to-be-labeled data through a pre-trained target labeling model to obtain a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data, and use the pre-trained target classification
  • the device selects the untrusted data in the data to be labeled to verify the labeling result corresponding to the untrusted data, and correct the labeling result corresponding to the untrusted data in the target set that has not passed the verification.
  • the untrusted data that has not passed the verification is determined as the first positive sample, and the untrusted data that has passed the verification is determined as the first negative sample, and the target classifier is updated using the first positive sample and the first negative sample. It not only saves a lot of human resources, improves the efficiency and accuracy of labeling, but also because this embodiment continuously optimizes and updates the target classifier during the labeling process, making the untrusted data screened by the target classifier more targeted .
  • FIG. 3 is a flowchart of another data labeling method according to an exemplary embodiment. This embodiment describes in detail the process of updating the target labeling model. This method can be applied to terminal devices. It can also be applied to the server. The method includes the following steps:
  • step 301 a plurality of to-be-labeled data is annotated through a pre-trained target labeling model, and a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data is obtained.
  • a pre-trained target classifier is used to select the untrusted data in the data to be labeled to verify the labeling result corresponding to the untrusted data.
  • step 303 the labeling result corresponding to the untrusted data that has not passed the verification in the target set is corrected.
  • step 304 the corrected target set is stored in a pre-established annotation database.
  • step 305 the target annotation model is updated using the annotation database.
  • a labeling database may be established in advance, and the labeling database is used to store labeled data, and the data stored in the labeling database is used to perform machine learning training.
  • the corrected target set can be used as labeled data and stored in the labeled database.
  • the data in the annotation database can be used as training samples to retrain the target annotation model to optimize and update the target annotation model, so that the target annotation model can more accurately annotate the to-be-labeled data.
  • the data labeling method provided by the above-mentioned embodiment of the present application labels a plurality of to-be-labeled data through a pre-trained target labeling model to obtain a target set composed of the labeling results corresponding to the above-mentioned to-be-labeled data, and use pre-trained target classification
  • the device selects the untrusted data in the data to be labeled to verify the labeling result corresponding to the untrusted data, correct the labeling result corresponding to the untrusted data in the target set that has not passed the verification, and store the corrected target set in A pre-established annotation database, and use the annotation database to update the target annotation model.
  • this embodiment uses the labeling database to optimize and update the target labeling model after each labeling is completed, the target labeling model is continuously improved, and the accuracy of labeling is further improved. degree.
  • the target annotation model may be trained in the following manner: iteratively perform an update operation on the annotation model until the stop condition is met, and use the iteratively updated annotation model as the target annotation model.
  • the training sample data may be manually annotated, and the manually annotated training sample data may be used to train the annotation model.
  • the method of this embodiment can be used to continue training the labeling model.
  • the update operation of the annotation model may be performed iteratively until the stopping condition is satisfied (for example, the objective function convergence is satisfied, or the number of iterations exceeds a preset number, etc.), and the annotation model after iterative update is used as the target annotation model.
  • the above update operation may include the following steps:
  • Step a Use the current target classifier to select untrusted samples in the sample data.
  • the currently trained target classifier can be used to select untrusted samples in the sample data.
  • the untrusted samples may have strong confusing, insignificant features, difficult to distinguish, difficult to label, and easy to label. Sample data that is marked incorrectly.
  • Step b Obtain the manual labeling result corresponding to the untrusted sample as the first sample result.
  • Step c Use the first sample result to update the current annotation model.
  • the untrusted samples are highly confusing and the features are not significant, it is more likely to make mistakes when using the labeling model to label the untrusted samples. Therefore, the untrusted samples can be labeled manually first, and the manual labeling results corresponding to the untrusted samples can be obtained as the first sample result.
  • the accuracy of the first sample result is relatively high, and the current labeling model can be continuously updated by using the first sample result to continuously improve the labeling model until the stopping condition is met, and the target labeling model is obtained. This helps to improve the accuracy of the target labeling model.
  • the above update operation may further include the following steps:
  • Step d Use the current target classifier to select credible samples in the sample data.
  • the current target classifier can also be used to select credible samples in the sample data, and the credible samples are opposite to the untrusted samples.
  • the credible sample may be sample data with prominent features, easy to distinguish, easy to label, and not easy to be incorrectly labeled.
  • Step e Use the current labeling model to label the credible samples to obtain the second sample result.
  • Step f Determine the verification result to be verified against the second sample result.
  • the current labeling model can be used to label the trusted samples to obtain the second sample result. It can be verified against the second sample result to determine whether the current labeling model is correct for labeling the trusted sample.
  • Step g Based on the verification result, the second sample result that fails the verification is determined as the second negative sample, and the second sample result that passes the verification is determined as the second positive sample.
  • the credible data filtered by the current target classifier is incorrectly labeled by the labeling model, it means that the target classifier is not accurate enough to classify the credible data. If the credible data filtered by the current target classifier is correctly labeled by the annotation model, it means that the current target classifier classifies the credible data more accurately. Therefore, the result of the second sample that fails the verification (that is, the credible data that is labeled incorrectly) can be determined as the second negative sample, and the result of the second sample that has passed the verification (that is, the credible data that is labeled correctly) can be determined as The second positive sample.
  • Step h Use the second positive sample and the second negative sample to update the current target classifier.
  • the second positive sample and the second negative sample can be used to optimize and update the current target classifier, so that the target classifier can more accurately screen trusted data and untrusted data.
  • this application also provides an embodiment of a data labeling device.
  • FIG. 4 is a block diagram of a data labeling device according to an exemplary embodiment of the present application.
  • the device may include: a labeling module 401, a selection module 402, and a correction module 403.
  • the labeling module 401 is used to label a plurality of data to be labelled through a pre-trained target labeling model to obtain a target set formed by labeling results corresponding to the data to be labelled.
  • the selection module 402 is used to select untrusted data in the data to be labeled using a pre-trained target classifier to verify the labeling result corresponding to the untrusted data.
  • the correction module 403 is used to correct the annotation result corresponding to the untrusted data in the target set that has not passed the verification.
  • FIG. 5 is a block diagram of another data labeling apparatus according to an exemplary embodiment of the present application. Based on the foregoing embodiment shown in FIG. 4, this embodiment may further include: The determination module 404 and the first update module 405.
  • the determining module 404 is configured to determine the untrusted data that has not passed the verification as the first positive sample, and determine the untrusted data that has passed the verification as the first negative sample.
  • the first update module 405 is configured to update the target classifier by using the first positive sample and the first negative sample.
  • the above-mentioned target annotation model may be trained in the following manner: iteratively perform an update operation on the annotation model until a stopping condition is met, and use the iteratively updated annotation model as the target annotation model.
  • the above update operation may include: using the current target classifier to select untrusted samples in the sample data, obtaining the manual labeling results corresponding to the untrusted samples, as the first sample result, and using the first sample result to update The current annotation model.
  • the above-mentioned update operation may further include: using the current target classifier to select credible samples in the sample data, using the current labeling model to label the credible samples to obtain the second sample result, and determining Based on the verification result of the verification of the second sample result, the second sample result that fails the verification is determined as the second negative sample, and the second sample result that has passed the verification is determined as the second positive sample.
  • the second positive sample and the second negative sample update the current target classifier.
  • the above-mentioned target labeling model may include multiple unit models with different structures, and the data to be labelled is image data to be labelled.
  • any image data to be labeled can be labeled as follows: each unit model is used to label the image data, and based on the result of each unit model being labeled in the image data, the non-maximum value suppression NMS algorithm is used Determine one or more labeled targets in the image data, and determine the label corresponding to each labeled target.
  • FIG. 6 is a block diagram of another data labeling apparatus according to an exemplary embodiment of the present application.
  • this embodiment may further include: Storage module 406 and second update module 407.
  • the storage module 406 is used to store the corrected target set in a pre-established annotation database.
  • the second update module 407 is used to update the target annotation model by using the annotation database.
  • the above-mentioned device may be pre-installed in the terminal device or server, or loaded into the terminal device or server through downloading or the like.
  • the corresponding modules in the above-mentioned device can cooperate with the modules in the terminal equipment or the server to realize the data labeling scheme.
  • the relevant part can refer to the part of the description of the method embodiment.
  • the device embodiments described above are merely illustrative.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network units.
  • Some or all of the modules can be selected according to actual needs to achieve the objectives of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative work.
  • the embodiment of the present application also provides a computer-readable storage medium, the storage medium stores a computer program, and the computer program can be used to execute the data labeling method provided by any one of the embodiments in FIG. 1 to FIG. 3.
  • the embodiment of the present application also proposes a schematic structural diagram of the electronic device according to an exemplary embodiment of the present application shown in FIG. 7. Please refer to FIG. 7.
  • the electronic device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include hardware required for other services.
  • the processor reads the corresponding computer program from the non-volatile memory to the memory and then runs it to form a data labeling device on the logical level.
  • this application does not exclude other implementations, such as logic devices or a combination of software and hardware, etc. That is to say, the execution body of the following processing flow is not limited to each logic unit, and can also be Hardware or logic device.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Investigating Or Analysing Biological Materials (AREA)
  • Document Processing Apparatus (AREA)

Abstract

一种数据标注的方法、装置及电子设备，所述方法包括：通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合（101）；利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果（102）；更正目标集合中未通过验证的非可信数据所对应的标注结果（103）。

Description

数据标注 技术领域
本申请涉及机器学习技术领域，特别涉及机器学习过程中的数据标注。
背景技术
随着人工智能技术的不断发展，人工智能技术在各个领域中已经得到了广泛地应用。人工智能技术通常涉及到机器学习，在机器学习的过程中，对训练样本数据的需求量巨大，并且需要对大量的训练样本数据进行标注。目前来说，通常需要人工对大量的训练样本数据进行标注，使得标注工作过分依赖于人，并且工作量巨大。从而耗费了大量的人力资源，并且标注的效率较低。
发明内容
为了解决上述技术问题之一，本申请提供一种数据标注的方法、装置及电子设备。
根据本申请实施例的第一方面，提供一种数据标注的方法，包括：
通过预先训练的目标标注模型对多个待标注数据进行标注，得到由所述待标注数据对应的标注结果构成的目标集合；
利用预先训练的目标分类器选取所述待标注数据中的非可信数据，以验证所述非可信数据对应的标注结果；
更正所述目标集合中未通过验证的非可信数据所对应的标注结果。
根据本申请实施例的第二方面，提供一种数据标注的装置，包括：
标注模块，用于通过预先训练的目标标注模型对多个待标注数据进行标注，得到由所述待标注数据对应的标注结果构成的目标集合；
选取模块，用于利用预先训练的目标分类器选取所述待标注数据中的非可信数据，以验证所述非可信数据对应的标注结果；
更正模块，用于更正所述目标集合中未通过验证的非可信数据所对应的标注结果。
根据本申请实施例的第三方面，提供一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述第一方面中任一项所述的方法。
根据本申请实施例的第四方面，提供一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述第一方面中任一项所述的方法。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本申请的实施例，并与说明书一起用于解释本申请的原理。
图1是本申请根据一示例性实施例示出的一种数据标注的方法的流程图；
图2是本申请根据一示例性实施例示出的另一种数据标注的方法的流程图；
图3是本申请根据一示例性实施例示出的另一种数据标注的方法的流程图；
图4是本申请根据一示例性实施例示出的一种数据标注的装置的框图；
图5是本申请根据一示例性实施例示出的另一种数据标注的装置的框图；
图6是本申请根据一示例性实施例示出的另一种数据标注的装置的框图；
图7是本申请根据一示例性实施例示出的一种电子设备的结构示意图。
具体实施方式
这里将详细地对示例性实施例进行说明，其示例表示在附图中。下面的描述涉及附图时，除非另有表示，不同附图中的相同数字表示相同或相似的要素。以下示例性实施例中所描述的实施方式并不代表与本申请相一致的所有实施方式。相反，它们仅是与如所附权利要求书中所详述的、本申请的一些方面相一致的装置和方法的例子。
在本申请使用的术语是仅仅出于描述特定实施例的目的，而非旨在限制本申请。在本申请和所附权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其他含义。还应当理解，本文中使用的术语“和/或”是指并包含一个或多个相关联的列出项目的任何或所有可能组合。
应当理解，尽管在本申请可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不 脱离本申请范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，如在此所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。
如图1所示，图1是根据一示例性实施例示出的一种数据标注的方法的流程图，该方法可以应用于终端设备中，也可以应用于服务器中。该方法包括以下步骤：
在步骤101中，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合。
在本实施例中，首先可以通过预先训练的目标标注模型对多个待标注数据进行标注，得到目标集合，该目标集合由上述待标注数据对应的标注结果构成。其中，待标注数据可以是图像类型的数据，也可以是声音类型的数据，还可以是文本类型的数据等，可以理解，待标注数据还可以是其它任意类型的数据，本申请对待标注数据的具体类型方面不限定。
在本实施例中，对待标注数据进行标注可以是将待标注数据中的标注目标进行标记，并针对标记设置相应的标签(例如，类别标签、属性标签、ID(identifier)标签等)。以图像类型的待标注数据为例，对图像类型的数据进行标注可以是将图像类型的数据中的目标物体(即标注目标)用标注框标记下来，并设置目标物体的标签(如，物体类别、物体属性、物体ID等)。
在本实施例中，目标标注模型可以是一个模型，也可以是多个结构不同的模型，可选地，如果待标注数据是图像类型的数据，则目标标注模型可以是神经网络类型的模型。在初始训练目标标注模型时，可以采用人工的方式标注训练样本数据，然后，利用人工标注的训练样本数据训练得到目标标注模型。在后续使用过程中，还可以不断对目标标注模型进行优化和更新，使目标标注模型标注的准确度更高。
在本实施例中，如果目标标注模型包括多个结构不同的单元模型，且待标注数据为待标注的图像数据，则可以通过如下方式标注任一待标注的图像数据：首先，可以分别采用每个单元模型在该图像数据中进行标注；然后，基于每个单元模型在该图像数据中进行标注的结果，利用NMS(non maximum suppression，非极大值抑制)算法确定该图像数据中的一个或多个标注目标，并确定每个标注目标对应的标签，从而完成对该图像数据的标注。其中，标注目标为该图像数据中需要标注的对象。由于采用多个结构不同的单元模型进行标注，可以提高标注的准确度。
在步骤102中，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果。
在本实施例中，可以利用预先训练的目标分类器从上述待标注数据中选取非可信数据，以验证非可信数据对应的标注结果。其中，非可信数据可以是具有较强迷惑性、特征不显著、难以分辨、难以标注且容易被标注错误的数据。以图像类型的数据为例，非可信数据可以是图像模糊，或者图像中具有较大干扰物体，或者图像中目标物体不明显的图像数据等。以声音类型的数据为例，非可信数据可以是环境噪声较大，或者目标声音太小的声音数据等。以文本类型的数据为例，非可信数据可以是语义特征模糊的文本数据等。
在本实施例中，目标分类器可以是预先训练好的分类器，本领域中已知的以及将来可能出现的任何能够应用于此的分类器都可以应用于本申请，本申请对目标分类器的具体类型方面不限定。在初始训练目标分类器时，可以采用人工分类筛选的方式，将训练样本数据分为可信样本数据和非可信样本数据。然后，利用人工分类筛选的训练样本数据训练得到目标分类器。在后续使用过程中，还可以不断对目标分类器进行优化和更新，使目标分类器分类的准确度更高。
在本实施例中，利用目标分类器从上述待标注数据中选取出容易被标注错误的非可信数据，然后，对非可信数据对应的标注结果进行验证，从而判断该非可信数据对应的标注结果是否正确。可以采用人工的方式进行验证，也可以采用其它任意合理的方式进行验证，可以理解，本申请对验证的具体方式方面不限定。针对任意非可信数据，如果确定该非可信数据对应的标注结果是错误的，则该非可信数据未通过验证，如果确定该非可信数据对应的标注结果是正确的，则该非可信数据通过验证。
在步骤103中，更正目标集合中未通过验证的非可信数据所对应的标注结果。
在本实施例中，对于选取出的非可信数据，如果该非可信数据通过验证，则不进行任何操作。如果该非可信数据未通过验证，则需要重新对该非可信数据进行正确的标注，从而得到正确的标注结果。并利用该非可信数据所对应的正确的标注结果，对目标集合中该非可信数据所对应的错误的标注结果进行替换，从而更正目标集合中未通过验证的非可信数据所对应的标注结果。
在本实施例中，可以预先建立标注数据库，标注数据库用于存放标注好的数据，利用标注数据库存放的数据，进行机器学习的训练。可以将经过更正的目标集合作为标注 好的数据，存放入标注数据库中。
本申请的上述实施例提供的数据标注的方法，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果，并更正目标集合中未通过验证的非可信数据所对应的标注结果。由于本实施例利用预先训练的目标标注模型对待标注数据进行标注后，又利用目标分类器筛选出有可能标注错误的待标注数据进行抽检，并对错误的标注结果进行更正。使得标注工作无需完全依赖于人，节省了大量的人力资源，提高了标注的效率。同时，能够更有针对性的验证被标注数据的结果，提高了标注的准确度。
如图2所示，图2是根据一示例性实施例示出的另一种数据标注的方法的流程图，该实施例描述了更新目标分类器的过程，该方法可以应用于终端设备中，也可以应用于服务器中。该方法包括以下步骤：
在步骤201中，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合。
在步骤202中，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果。
在步骤203中，更正目标集合中未通过验证的非可信数据所对应的标注结果。
在步骤204中，将未通过验证的非可信数据确定为第一正样本，并将通过验证的非可信数据确定为第一负样本。
在本实施例中，由于非可信数据具有较强迷惑性、特征不显著、难以分辨、难以标注且容易被标注错误。因此，如果目标分类器筛选出的非可信数据被标注错误，则说明目标分类器对该非可信数据的分类较为准确。如果目标分类器筛选出的非可信数据被标注正确，则说明目标分类器对该非可信数据的分类不够准确。所以，将未通过验证的非可信数据(即被标注错误的非可信数据)作为第一正样本，并将通过验证的非可信数据(即被标注正确的非可信数据)确定为第一负样本。
在步骤205中，利用第一正样本和第一负样本对目标分类器进行更新。
在本实施例中，可以利用第一正样本和第一负样本重新训练目标分类器，以对目标分类器进行优化更新，从而使目标分类器能够筛选出更合适的非可信数据。
需要说明的是，对于与图1实施例中相同的步骤，在上述图2实施例中不再进行赘述，相关内容可参见图1实施例。
本申请的上述实施例提供的数据标注的方法，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果，更正目标集合中未通过验证的非可信数据所对应的标注结果。将未通过验证的非可信数据确定为第一正样本，并将通过验证的非可信数据确定为第一负样本，利用第一正样本和第一负样本对目标分类器进行更新。不仅节省了大量的人力资源，提高了标注的效率和准确度，而且由于本实施例在标注过程中，不断对目标分类器进行优化更新，使得目标分类器筛选的非可信数据更有针对性。
如图3所示，图3是根据一示例性实施例示出的另一种数据标注的方法的流程图，该实施例详细描述了更新目标标注模型的过程，该方法可以应用于终端设备中，也可以应用于服务器中。该方法包括以下步骤：
在步骤301中，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合。
在步骤302中，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果。
在步骤303中，更正目标集合中未通过验证的非可信数据所对应的标注结果。
在步骤304中，将经过更正的目标集合存放入预先建立的标注数据库。
在步骤305中，利用该标注数据库对目标标注模型进行更新。
在本实施例中，可以预先建立标注数据库，标注数据库用于存放标注好的数据，利用标注数据库存放的数据，进行机器学习的训练。可以将经过更正的目标集合作为标注好的数据，存放入标注数据库中。
在本实施例中，可以利用标注数据库中的数据作为训练样本，重新训练目标标注模型，以对目标标注模型进行优化更新，从而使目标标注模型对待标注数据进行标注的结果更加准确。
需要说明的是，对于与图1和图2实施例中相同的步骤，在上述图3实施例中不再进行赘述，相关内容可参见图1和图2实施例。
本申请的上述实施例提供的数据标注的方法，通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合，利用预先训练的目标分类器选取上述待标注数据中的非可信数据，以验证非可信数据对应的标注结果，更正目标集合中未通过验证的非可信数据所对应的标注结果，将经过更正的目标集合存放入预先建立的标注数据库，并利用该标注数据库对目标标注模型进行更新。不仅节省了大量的人力资源，提高了标注的效率，而且由于本实施例在每次标注结束后，利用标注数据库对目标标注模型进行优化更新，使得目标标注模型不断完善，进一步提高了标注的准确度。
在一些可选实施方式中，目标标注模型可以通过如下方式训练：迭代执行对标注模型的更新操作，直至满足停止条件，将经过迭代更新后的标注模型作为目标标注模型。
在本实施例中，在初始训练标注模型时，可以采用人工的方式标注训练样本数据，并利用人工标注的训练样本数据训练标注模型。当训练的标注模型对待标注数据的标注结果达到准确度阈值后，可以采用本实施例的方式继续训练标注模型。具体来说，可以迭代执行对标注模型的更新操作，直至满足停止条件(例如，满足目标函数收敛，或者迭代次数超过预设次数等)，将经过迭代更新后的标注模型作为目标标注模型。其中，上述更新操作可以包括以下步骤：
步骤a：利用当前的目标分类器选取样本数据中的非可信样本。
在本实施例中，首先可以利用当前训练好的目标分类器选取样本数据中的非可信样本，该非可信样本可以是具有较强迷惑性、特征不显著、难以分辨、难以标注且容易被标注错误的样本数据。
步骤b：获取非可信样本对应的人工标注结果，作为第一样本结果。
步骤c：利用该第一样本结果更新当前的标注模型。
在本实施例中，由于非可信样本具有较强迷惑性，并且特征不显著，因此，利用标注模型对非可信样本进行标注更容易出现错误。所以，可以首先采用人工的方式对非可信样本进行标注，从而得到非可信样本对应的人工标注结果，作为第一样本结果。该第一样本结果的准确率较高，可以利用该第一样本结果不断更新当前的标注模型，使标注模型不断完善，直至满足停止条件，得到目标标注模型。从而有助于提高目标标注模型标注的准确度。
在另一些可选实施方式中，上述更新操作还可以包括以下步骤：
步骤d：利用当前的目标分类器选取样本数据中的可信样本。
在本实施例中，还可以利用当前的目标分类器选取样本数据中的可信样本，该可信样本与非可信样本相反。该可信样本可以是特征显著、容易分辨、容易标注且不易被标注错误的样本数据。
步骤e：利用当前的标注模型对可信样本进行标注，得到第二样本结果。
步骤f：确定针对第二样本结果进行验证的验证结果。
在本实施例中，由于可信样本特征显著且容易分辨，因此，利用标注模型对可信样本进行标注不容易出现错误。可以利用当前的标注模型对可信样本进行标注，得到第二样本结果。可以针对第二样本结果进行验证，以确定当前的标注模型对可信样本进行标注的结果是否正确。
步骤g：基于该验证结果，将未通过验证的第二样本结果确定为第二负样本，并将通过验证的第二样本结果确定为第二正样本。
在本实施例中，如果当前的目标分类器筛选出的可信数据被标注模型标注错误，则说明目标分类器对该可信数据的分类不够准确。如果当前的目标分类器筛选出的可信数据被标注模型标注正确，则说明当前的目标分类器对该可信数据的分类比较准确。所以，可以将未通过验证的第二样本结果(即被标注错误的可信数据)确定为第二负样本，并将通过验证的第二样本结果(即被标注正确的可信数据)确定为第二正样本。
步骤h：利用第二正样本和第二负样本对当前的目标分类器进行更新。
在本实施例中，可以利用第二正样本和第二负样本对当前的目标分类器进行优化更新，从而使目标分类器对可信数据和非可信数据的筛选更加准确。
应当注意，尽管在上述实施例中，以特定顺序描述了本申请方法的操作，但是，这并非要求或者暗示必须按照该特定顺序来执行这些操作，或是必须执行全部所示的操作才能实现期望的结果。相反，流程图中描绘的步骤可以改变执行顺序。附加地或备选地，可以省略某些步骤，将多个步骤合并为一个步骤执行，和/或将一个步骤分解为多个步骤执行。
与前述数据标注的方法实施例相对应，本申请还提供了数据标注的装置的实施例。
如图4所示，图4是本申请根据一示例性实施例示出的一种数据标注的装置框图，该装置可以包括：标注模块401，选取模块402和更正模块403。
其中，标注模块401，用于通过预先训练的目标标注模型对多个待标注数据进行标注，得到由上述待标注数据对应的标注结果构成的目标集合。
选取模块402，用于利用预先训练的目标分类器选取待标注数据中的非可信数据，以验证非可信数据对应的标注结果。
更正模块403，用于更正目标集合中未通过验证的非可信数据所对应的标注结果。
如图5所示，图5是本申请根据一示例性实施例示出的另一种数据标注的装置框图，该实施例在前述图4所示实施例的基础上，该装置还可以进一步包括：确定模块404和第一更新模块405。
其中，确定模块404，用于将未通过验证的非可信数据确定为第一正样本，并将通过验证的非可信数据确定为第一负样本。
第一更新模块405，用于利用第一正样本和第一负样本对目标分类器进行更新。
在一些可选实施方式中，上述目标标注模型可以通过如下方式训练：迭代执行对标注模型的更新操作，直至满足停止条件，将经过迭代更新后的标注模型作为所述目标标注模型。其中，上述更新操作可以包括：利用当前的目标分类器选取样本数据中的非可信样本，获取非可信样本对应的人工标注结果，作为第一样本结果，并利用第一样本结果更新当前的标注模型。
在另一些可选实施方式中，上述更新操作还可以包括：利用当前的目标分类器选取样本数据中的可信样本，利用当前的标注模型对可信样本进行标注，得到第二样本结果，确定针对第二样本结果进行验证的验证结果，基于该验证结果，将未通过验证的第二样本结果确定为第二负样本，并将通过验证的第二样本结果确定为第二正样本，并利用第二正样本和第二负样本对当前的目标分类器进行更新。
在另一些可选实施方式中，上述目标标注模型可以包括多个结构不同的单元模型，待标注数据为待标注的图像数据。
可以通过如下方式标注任一待标注的图像数据：分别采用每个单元模型在该图像数据中进行标注，基于每个单元模型在该图像数据中进行标注的结果，利用非极大值抑制NMS算法确定该图像数据中的一个或多个标注目标，并确定每个标注目标对应的标签。
如图6所示，图6是本申请根据一示例性实施例示出的另一种数据标注的装置框图，该实施例在前述图4所示实施例的基础上，该装置还可以进一步包括：存放模块406和 第二更新模块407。
其中，存放模块406，用于将经过更正的目标集合存放入预先建立的标注数据库中。
第二更新模块407，用于利用标注数据库对目标标注模型进行更新。
应当理解，上述装置可以预先设置在终端设备或服务器中，也可以通过下载等方式而加载到终端设备或服务器中。上述装置中的相应模块可以与终端设备或服务器中的模块相互配合以实现数据标注的方案。
对于装置实施例而言，由于其基本对应于方法实施例，所以相关之处参见方法实施例的部分说明即可。以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本申请方案的目的。本领域普通技术人员在不付出创造性劳动的情况下，即可以理解并实施。
本申请实施例还提供了一种计算机可读存储介质，该存储介质存储有计算机程序，计算机程序可用于执行上述图1至图3任一实施例提供的数据标注的方法。
对应于上述的数据标注的方法，本申请实施例还提出了图7所示的根据本申请的一示例性实施例的电子设备的示意结构图。请参考图7，在硬件层面，该电子设备包括处理器、内部总线、网络接口、内存以及非易失性存储器，当然还可能包括其他业务所需要的硬件。处理器从非易失性存储器中读取对应的计算机程序到内存中然后运行，在逻辑层面上形成数据标注的装置。当然，除了软件实现方式之外，本申请并不排除其他实现方式，比如逻辑器件抑或软硬件结合的方式等等，也就是说以下处理流程的执行主体并不限定于各个逻辑单元，也可以是硬件或逻辑器件。
本领域技术人员在考虑说明书及实践这里公开的发明后，将容易想到本申请的其它实施方案。本申请旨在涵盖本申请的任何变型、用途或者适应性变化，这些变型、用途或者适应性变化遵循本申请的一般性原理并包括本申请未公开的本技术领域中的公知常识或惯用技术手段。说明书和实施例仅被视为示例性的，本申请的真正范围和精神由下面的权利要求指出。
应当理解的是，本申请并不局限于上面已经描述并在附图中示出的精确结构，并且可以在不脱离其范围进行各种修改和改变。本申请的范围仅由所附的权利要求来限制。

Claims (10)

  1. 一种数据标注的方法，包括：
    通过预先训练的目标标注模型对多个待标注数据进行标注，得到由所述待标注数据对应的标注结果构成的目标集合；
    利用预先训练的目标分类器选取所述待标注数据中的非可信数据，以验证所述非可信数据对应的标注结果；
    更正所述目标集合中未通过验证的非可信数据所对应的标注结果。
  2. 根据权利要求1所述的方法，还包括：
    将未通过验证的非可信数据确定为第一正样本，并将通过验证的非可信数据确定为第一负样本；
    利用所述第一正样本和所述第一负样本对所述目标分类器进行更新。
  3. 根据权利要求1所述的方法，所述目标标注模型通过如下方式训练：
    迭代执行对标注模型的更新操作，直至满足停止条件；
    将经过迭代更新后的标注模型作为所述目标标注模型；
    其中，所述更新操作包括：
    利用当前的目标分类器选取样本数据中的非可信样本；
    获取所述非可信样本对应的人工标注结果，作为第一样本结果；
    利用所述第一样本结果更新当前的标注模型。
  4. 根据权利要求3所述的方法，所述更新操作还包括：
    利用当前的目标分类器选取所述样本数据中的可信样本；
    利用当前的标注模型对所述可信样本进行标注，得到第二样本结果；
    确定针对所述第二样本结果进行验证的验证结果；
    基于所述验证结果，将未通过验证的第二样本结果确定为第二负样本，并将通过验证的第二样本结果确定为第二正样本；
    利用所述第二正样本和所述第二负样本对当前的目标分类器进行更新。
  5. 根据权利要求1所述的方法，所述目标标注模型包括多个结构不同的单元模型；所述待标注数据为待标注的图像数据；
    通过如下方式标注任一待标注的图像数据：
    分别采用每个所述单元模型在该图像数据中进行标注；
    基于每个所述单元模型在该图像数据中进行标注的结果，利用非极大值抑制NMS算法确定该图像数据中的一个或多个标注目标；并
    确定每个所述标注目标对应的标签。
  6. 根据权利要求1所述的方法，还包括：
    将经过更正的目标集合存放入预先建立的标注数据库中；
    利用所述标注数据库对所述目标标注模型进行更新。
  7. 一种数据标注的装置，包括：
    标注模块，用于通过预先训练的目标标注模型对多个待标注数据进行标注，得到由所述待标注数据对应的标注结果构成的目标集合；
    选取模块，用于利用预先训练的目标分类器选取所述待标注数据中的非可信数据，以验证所述非可信数据对应的标注结果；
    更正模块，用于更正所述目标集合中未通过验证的非可信数据所对应的标注结果。
  8. 根据权利要求7所述的装置，还包括：
    确定模块，用于将未通过验证的非可信数据确定为第一正样本，并将通过验证的非可信数据确定为第一负样本；
    第一更新模块，用于利用所述第一正样本和所述第一负样本对所述目标分类器进行更新。
  9. 一种计算机可读存储介质，所述存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述权利要求1-6中任一项所述的方法。
  10. 一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现上述权利要求1-6中任一项所述的方法。
PCT/CN2019/123406 2019-06-05 2019-12-05 数据标注 Ceased WO2020244183A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910487643.6A CN110288007B (zh) 2019-06-05 2019-06-05 数据标注的方法、装置及电子设备
CN201910487643.6 2019-06-05

Publications (1)

Publication Number Publication Date
WO2020244183A1 true WO2020244183A1 (zh) 2020-12-10

Family

ID=68003424

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/123406 Ceased WO2020244183A1 (zh) 2019-06-05 2019-12-05 数据标注

Country Status (2)

Country Link
CN (1) CN110288007B (zh)
WO (1) WO2020244183A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112861962A (zh) * 2021-02-03 2021-05-28 北京百度网讯科技有限公司 样本处理方法、装置、电子设备和存储介质
CN112884060A (zh) * 2021-03-09 2021-06-01 联仁健康医疗大数据科技股份有限公司 图像标注方法、装置、电子设备及存储介质
CN117909333A (zh) * 2024-02-02 2024-04-19 深圳天朴科技有限公司 基于大数据结合人工智能实现数据的筛选方法及系统

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110288007B (zh) * 2019-06-05 2021-02-02 北京三快在线科技有限公司 数据标注的方法、装置及电子设备
CN110797101B (zh) * 2019-10-28 2023-11-03 腾讯医疗健康(深圳)有限公司 医学数据处理方法、装置、可读存储介质和计算机设备
CN113469205B (zh) * 2020-03-31 2023-01-17 阿里巴巴集团控股有限公司 数据处理方法及系统、网络模型及其训练方法、电子设备
CN113591885A (zh) * 2020-04-30 2021-11-02 阿里巴巴集团控股有限公司 目标检测模型训练方法、设备及计算机存储介质
CN111897991B (zh) * 2020-06-19 2022-08-26 济南信通达电气科技有限公司 一种图像标注方法及设备
CN112163424B (zh) * 2020-09-17 2024-07-19 中国建设银行股份有限公司 数据的标注方法、装置、设备和介质
CN112163132B (zh) * 2020-09-21 2024-05-10 中国建设银行股份有限公司 一种数据标注方法、装置、存储介质及电子设备
CN115409076A (zh) * 2021-05-26 2022-11-29 中国移动通信集团浙江有限公司 数据标注方法、装置、设备及存储介质
CN115329979A (zh) * 2022-07-14 2022-11-11 阿波罗智联(北京)科技有限公司 标注数据处理方法、装置、设备以及存储介质
CN115797930A (zh) * 2022-11-09 2023-03-14 深圳杉谷科技有限公司 图像数据自动标注方法、设备、存储介质及装置
CN116701879B (zh) * 2023-07-10 2025-09-30 清华大学 一种用于对样本进行加权的可视分析系统和方法

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103324937A (zh) * 2012-03-21 2013-09-25 日电（中国）有限公司 标注目标的方法和装置
US20180204111A1 (en) * 2013-02-28 2018-07-19 Z Advanced Computing, Inc. System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform
CN108875768A (zh) * 2018-01-23 2018-11-23 北京迈格威科技有限公司 数据标注方法、装置和系统及存储介质
CN109242013A (zh) * 2018-08-28 2019-01-18 北京九狐时代智能科技有限公司 一种数据标注方法、装置、电子设备及存储介质
CN109241997A (zh) * 2018-08-03 2019-01-18 硕橙（厦门）科技有限公司 一种生成训练集的方法及装置
CN109635838A (zh) * 2018-11-12 2019-04-16 平安科技（深圳）有限公司 人脸样本图片标注方法、装置、计算机设备及存储介质
CN110288007A (zh) * 2019-06-05 2019-09-27 北京三快在线科技有限公司 数据标注的方法、装置及电子设备

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050027664A1 (en) * 2003-07-31 2005-02-03 Johnson David E. Interactive machine learning system for automated annotation of information in text
US20100023319A1 (en) * 2008-07-28 2010-01-28 International Business Machines Corporation Model-driven feedback for annotation
CN101980210A (zh) * 2010-11-12 2011-02-23 百度在线网络技术（北京）有限公司 一种标的词分类分级方法及系统
CN102541838B (zh) * 2010-12-24 2015-03-11 日电（中国）有限公司 用于优化情感分类器的方法和设备
US8855430B1 (en) * 2012-05-30 2014-10-07 Google Inc. Refining image annotations
CN104281569B (zh) * 2013-07-01 2017-08-01 富士通株式会社 构建装置和方法、分类装置和方法以及电子设备
CN105224947B (zh) * 2014-06-06 2018-11-13 株式会社理光 分类器训练方法和系统
CN105117429B (zh) * 2015-08-05 2018-11-23 广东工业大学 基于主动学习和多标签多示例学习的场景图像标注方法
CN108197668A (zh) * 2018-01-31 2018-06-22 达闼科技（北京）有限公司 模型数据集的建立方法及云系统
CN109543713B (zh) * 2018-10-16 2021-03-26 北京奇艺世纪科技有限公司 训练集的修正方法及装置
CN109446961B (zh) * 2018-10-19 2020-10-30 北京达佳互联信息技术有限公司 姿势检测方法、装置、设备及存储介质
CN109460795A (zh) * 2018-12-17 2019-03-12 北京三快在线科技有限公司 分类器训练方法、装置、电子设备及计算机可读介质
CN109784391B (zh) * 2019-01-04 2021-01-05 杭州比智科技有限公司 基于多模型的样本标注方法及装置

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103324937A (zh) * 2012-03-21 2013-09-25 日电（中国）有限公司 标注目标的方法和装置
US20180204111A1 (en) * 2013-02-28 2018-07-19 Z Advanced Computing, Inc. System and Method for Extremely Efficient Image and Pattern Recognition and Artificial Intelligence Platform
CN108875768A (zh) * 2018-01-23 2018-11-23 北京迈格威科技有限公司 数据标注方法、装置和系统及存储介质
CN109241997A (zh) * 2018-08-03 2019-01-18 硕橙（厦门）科技有限公司 一种生成训练集的方法及装置
CN109242013A (zh) * 2018-08-28 2019-01-18 北京九狐时代智能科技有限公司 一种数据标注方法、装置、电子设备及存储介质
CN109635838A (zh) * 2018-11-12 2019-04-16 平安科技（深圳）有限公司 人脸样本图片标注方法、装置、计算机设备及存储介质
CN110288007A (zh) * 2019-06-05 2019-09-27 北京三快在线科技有限公司 数据标注的方法、装置及电子设备

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112861962A (zh) * 2021-02-03 2021-05-28 北京百度网讯科技有限公司 样本处理方法、装置、电子设备和存储介质
CN112861962B (zh) * 2021-02-03 2024-04-09 北京百度网讯科技有限公司 样本处理方法、装置、电子设备和存储介质
CN112884060A (zh) * 2021-03-09 2021-06-01 联仁健康医疗大数据科技股份有限公司 图像标注方法、装置、电子设备及存储介质
CN112884060B (zh) * 2021-03-09 2024-04-26 联仁健康医疗大数据科技股份有限公司 图像标注方法、装置、电子设备及存储介质
CN117909333A (zh) * 2024-02-02 2024-04-19 深圳天朴科技有限公司 基于大数据结合人工智能实现数据的筛选方法及系统

Also Published As

Publication number Publication date
CN110288007A (zh) 2019-09-27
CN110288007B (zh) 2021-02-02

Similar Documents

Publication Publication Date Title
WO2020244183A1 (zh) 数据标注
US10489722B2 (en) Semiautomatic machine learning model improvement and benchmarking
US10803398B2 (en) Apparatus and method for information processing
WO2021218024A1 (zh) 命名实体识别模型的训练方法、装置、计算机设备
US10832036B2 (en) Meta-learning for facial recognition
CN110880019A (zh) 通过无监督域适应训练目标域分类模型的方法
GB2589495A (en) Closed loop automatic dataset creation systems and methods
CN108121795A (zh) 用户行为预测方法及装置
CN108076154A (zh) 应用信息推荐方法、装置及存储介质和服务器
CN115810135A (zh) 样本分析的方法、电子设备、存储介质和程序产品
CN111639766B (zh) 样本数据的生成方法以及装置
CN112862093A (zh) 一种图神经网络训练方法及装置
CN112115993B (zh) 一种基于元学习的零样本和小样本证件照异常检测方法
CN109616101B (zh) 声学模型训练方法、装置、计算机设备和可读存储介质
CN109272003A (zh) 一种消除深度学习模型中未知错误的方法与装置
CN106485261A (zh) 一种图像识别的方法和装置
WO2020125404A1 (zh) 构建神经网络的方法、装置和计算机可读介质
CN111709966A (zh) 眼底图像分割模型训练方法及设备
CN111444346A (zh) 一种用于文本分类的词向量对抗样本生成方法及装置
WO2019095899A1 (zh) 素材标注方法以及装置、终端和计算机可读存储介质
CN115205727A (zh) 一种基于无监督学习的实验智能评分方法和系统
CN112364990B (zh) 通过元学习实现语法改错少样本领域适应的方法及系统
CN113220883B (zh) 一种文本分类方法、装置及存储介质
CN109254899A (zh) 一种服务器监控方法和装置
CN113869516A (zh) 知识图谱嵌入模型训练方法、装置、电子设备及介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19931810

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19931810

Country of ref document: EP

Kind code of ref document: A1