WO2022224378A1 - Système de collecte de données et programme associé - Google Patents

Système de collecte de données et programme associé Download PDF

Info

Publication number
WO2022224378A1
WO2022224378A1 PCT/JP2021/016183 JP2021016183W WO2022224378A1 WO 2022224378 A1 WO2022224378 A1 WO 2022224378A1 JP 2021016183 W JP2021016183 W JP 2021016183W WO 2022224378 A1 WO2022224378 A1 WO 2022224378A1
Authority
WO
WIPO (PCT)
Prior art keywords
work
data
worker
workers
task
Prior art date
Application number
PCT/JP2021/016183
Other languages
English (en)
Japanese (ja)
Inventor
良 高品
翔吾 藤井
俊策 遠藤
Original Assignee
株式会社Apto
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 株式会社Apto filed Critical 株式会社Apto
Priority to PCT/JP2021/016183 priority Critical patent/WO2022224378A1/fr
Priority to JP2023515951A priority patent/JPWO2022224378A1/ja
Publication of WO2022224378A1 publication Critical patent/WO2022224378A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Definitions

  • the present invention relates to a data collection system and its program for collecting annotation data for artificial intelligence.
  • Annotation here refers to adding a tag (metadata) to the original data to give it some meaning. It refers to giving the correct answer according to the answer. For example, in AI development for distinguishing whether an image is a dog or a cat, an image of a dog is tagged with "dog", and an image of a cat is tagged with "cat".
  • Non-Patent Document 1 after installing an annotation application on the work terminal operated by the worker, the worker performs the work according to the instructions on the screen, and the work result is transmitted. platform is disclosed.
  • the drawings in the literature describe semantic segmentation of images (painting), image bounding boxes (rectangular selection), and image classification requiring input of the meaning of images as work required of the operator.
  • items such as 300 points for image-related work, 520 points for voice-related work, 450 points for text-related work, and 1270 points in total are described. From this, it can be seen that the worker earns points by performing work according to the type.
  • Patent Document 1 discloses an information processing device in which Oracle labels unlabeled data.
  • the term "oracle” is used to mean a person who knows the answer of annotation, and it is stated that any machine or program may be used.
  • a classification probability of unlabeled data into a classification class and a classification uncertainty are calculated by a classifier trained based on learning data and labels.
  • a predetermined oracle classifier trained based on the training data and oracle information calculates the classification probability and classification confidence of the unlabeled data to the oracle.
  • a predetermined number of unlabeled data are selected in descending order of the sum of uncertainty and certainty.
  • a request for labeling is made to a predetermined number of oracles in descending order of the classification probability of the selected unlabeled data.
  • the correct label is determined by taking a majority vote, and the correctness of the label is determined in addition to the oracle information in the training data. Recorded.
  • learning of the oracle classifier is performed using only learning data with correct labels.
  • Non-Patent Document 1 the quality of annotation data (learning data) collected from a large number of workers varies due to the superiority or inferiority of individual workers who perform annotation. In order to ensure this quality, it is preferable to reward good workers for their efforts and to encourage unsatisfactory workers to make further efforts with respect to giving points that serve as incentives for workers. For that purpose, it is necessary to evaluate workers in some way. At that time, if the correct answer for the work item (task) to be requested to the worker is prepared in advance, the correct answer and the work result can be compared. However, in reality, the data collecting side does not grasp such a correct answer, so it is necessary to set some evaluation criteria in order to evaluate workers.
  • Patent Document 1 discloses that when a plurality of different labels (tags) are assigned to annotation data, the correct label is selected by a majority vote to resolve label discrepancies, and only learning data with correct labels It enables learning using Therefore, Patent Literature 1 does not evaluate the worker who made the annotation, and therefore does not describe evaluation criteria therefor, nor does it suggest such.
  • the purpose of the present invention is to evaluate the worker who made the annotation in the collection of annotation data for artificial intelligence.
  • the first invention provides a data collection system that is network-connected to a plurality of work terminals where individual workers work, and that collects annotation data for artificial intelligence.
  • This data collection system has a data transmission section, a data reception section, an annotation data storage section, a correct answer determination section, and a work evaluation section.
  • the data transmission unit transmits, as a task, original data to be worked on and work instructions to be performed by the worker with respect to the original data to the work terminal.
  • the data receiving unit receives, from the work terminal, annotation data in which the result of work performed by the worker in response to the work instruction is added as a tag to the original data regarding a certain task.
  • the annotation data storage unit stores annotation data received by the data reception unit.
  • the correct answer determination unit reads annotation data collected from a plurality of workers regarding the same task from the annotation data storage unit, and determines the correct answer for this task based on a majority vote of the work results in the read annotation data.
  • the work evaluation unit evaluates the workers who participated in the majority decision based on the correct answer determined by the majority decision.
  • the work evaluation unit evaluates the workers involved in the majority decision based on the result of matching the correct answer determined by the majority decision with the work results of the workers involved in the majority decision.
  • a point granting unit may be provided that grants points to the worker as an incentive for the worker, taking into consideration the rating indicating the degree of evaluation of the worker.
  • the work evaluation unit reflects the evaluation results of the workers involved in the majority decision in the ratings of the workers.
  • the work evaluation section raises the rating of the worker who performed the work corresponding to the correct answer among the plurality of workers involved in the majority decision, and lowers the rating of the other workers.
  • the work evaluation unit may reduce the rating of a worker who is judged to have too short a task work time required to work on one task by comparison with a predetermined threshold value.
  • the work evaluation unit generates a distribution of the total work time of a task set, which is a collection of a plurality of tasks, for all workers involved in the request from the client, and The rating of a worker judged to have a too short overall work time may be lowered.
  • the work evaluation unit may lower the rating of a worker who selects an option other than the obvious correct answer in a selection-type task including options managed by the system as the obvious correct answer.
  • the work evaluation unit may reduce the rating of a worker who selects an obviously incorrect option in a selection-type task including an option managed by the system as a clearly incorrect answer.
  • a second invention provides a data collection program that is executed on a computer network-connected to a plurality of work terminals where individual workers work, and that collects annotation data for artificial intelligence.
  • This data collection program causes the computer to perform the following steps.
  • original data to be worked on and work instructions to be given by the worker to this original data are generated for a given task.
  • the annotation data received from the work terminal for a certain task and to which the result of the work performed by the worker according to the work instruction is added as a tag to the original data is stored in the storage device.
  • annotation data collected from a plurality of workers regarding the same task are read out from the storage device, and the correct answer for this task is determined by a majority vote of the work results in the read annotation data.
  • the correct answers determined by the majority vote are used as evaluation criteria to evaluate the workers involved in the majority vote.
  • the workers involved in the majority vote are evaluated based on the result of matching the correct answer determined by the majority vote with the work results of the workers involved in the majority vote. preferably.
  • a fifth step may be provided in which points are given to the worker as an incentive for the worker, taking into consideration the rating indicating the degree of evaluation of the worker.
  • a sixth step of lowering the rating of a worker who is judged to have too short working time for one task by comparison with a predetermined threshold value for all workers involved in the request from the requester, the distribution of the total work time of the task set, which is a collection of multiple tasks, is generated, and compared with a predetermined threshold, the total work time
  • a seventh step may be provided in which the rating of the worker whose time is judged to be too short is lowered.
  • an eighth step of lowering the rating of a worker who selects an option other than the obvious correct answer may be provided.
  • a ninth step of lowering the rating of the operator who selected the clearly incorrect option may be provided.
  • the correct answer for the same task is determined by a majority vote of the work results collected from multiple workers.
  • the correct answer determined by majority vote has objective rationality, and can be used as an evaluation criterion for evaluating the workers involved in this majority vote. As a result, even if the annotation data collecting side does not know the correct answer for the task, it is possible to appropriately evaluate the worker who made the annotation.
  • FIG. 1 is an overall view of the annotation system according to this embodiment.
  • This annotation system 1 is mainly composed of a data collection system 2 and a plurality of external terminals 3 and 4 network-connected thereto.
  • the data collection system 2 receives a request from a client and collects annotation data by instructing a plurality of workers to perform annotation work.
  • the request terminal 3 is operated by a requester who requests collection of annotation data.
  • the work terminal 4 is operated by a worker who actually performs annotation work, and has dedicated software (annotation application) installed.
  • the requester requests the data collection system 2 to collect annotation data (procedure 1).
  • the data collection system 2 that has received the collection request transmits an annotation work request to a plurality of workers regarding this requested item (procedure 2).
  • the worker who receives this request operates his own work terminal 4 to make an annotation, and transmits the work result to the data collection system 2 (procedure 3).
  • data transmission/reception in procedures 2 and 3 is repeated until the collection of annotation data for the preset target number is completed.
  • the data collection system 2 compiles the annotation data, adds comments on the collection results as appropriate, and transmits them to the requester as the collection results in response to the collection request in step 1 (step 4).
  • the annotation data is compiled by extracting annotation data belonging to a specific project from the annotation data storage unit 2i, which will be described later.
  • the annotation data is sent in response to a request from the client, and the client can download the annotation data from the management screen at any time.
  • the data collection system 2 gives points to the worker who made the annotation (procedure 5). The points are incentives for the workers, and the workers can exchange them for gifts or the like by collecting the points.
  • the data collection system 2 evaluates the worker who made the annotation and updates the rating for each worker (procedure 6).
  • the rating means the degree of evaluation (superiority or inferiority) of the worker, and the points given vary according to this rating. Specifically, the higher the rating, the more points are given, and the lower the rating, the less points are given. As a result, excellent workers are rewarded for their efforts, and unsatisfactory workers are encouraged to make further efforts.
  • the updated rating will be applied to the next point award (the rating before the update will be applied to this point award).
  • Annotation targets include images, texts, and voices, and the tasks (work contents) are divided into the work type in which the worker actually performs the work, and the worker's selection from among multiple options. It is roughly divided into types. Specifically, for the image system, there are tasks to select items related to the image from options, tasks to input the meaning of the image as text, tasks to fill in part of the image, and part of the image to be a rectangle or polygon. There are tasks surrounded by .
  • the writing system includes a task of classifying sentences (may be of a selective type), a task of inputting text translations and summaries of sentences, and a task of evaluating sentences by points (may be of a selective type). )and so on.
  • the speech system there is a task of transcribing a specific speech into text.
  • FIG. 2 is a block diagram of the data collection system 2.
  • This data collection system 2 includes a data transmission section 2a, a data reception section 2b, a data management section 2c, a point provision section 2d, a correct answer determination section 2e, a work evaluation section 2f, and storage devices 2g to 2k. Mainly composed. Necessary information is stored in the project storage unit 2g and the original data storage unit 2h before the annotation work is performed.
  • the project storage unit 2g stores the specific contents of each project requested by the client.
  • the “project” means, for example, “I want 10,000 pieces of annotation data in which an image of a cat is painted over” or “I want 10,000 pieces of annotation data in which an image of a dog is enclosed in a rectangle.” It is the requested project and its contents.
  • the requester can arbitrarily set the contents of the project through the display screen on the request terminal 3 .
  • FIG. 3 is a diagram showing an example of a request screen displayed on the request terminal 3.
  • This request screen has input items such as "project name”, “project type”, “number of tasks per set”, “number of times the same task is repeated”, “budget”, and “implementation deadline”. is displayed.
  • project name a label for identifying the requested item is entered.
  • project type either “private”, which allows only specific workers to work, or "public”, which allows anyone to work, is selected.
  • the "number of tasks per set” specifies the number of tasks to be treated as a set. For example, if “1" is specified, the task set will be completed if the worker completes the task once.
  • a large number of image data, text data, and voice data are stored in the original data storage unit 2h as original data to be annotated.
  • the original data is stored in association with the project, and there are as many pieces of original data as the target number of items to be processed in one project.
  • Another method is to outsource the collection of original data itself to the data collection system 2 .
  • the data collection system 2 that undertakes the collection requests a large number of workers to provide specific original data (for example, images of cats) according to the requester's designation, and receives the original data from them. By doing so, the target number of original data is collected.
  • the data management unit 2c In response to a work request from the work terminal 4, the data management unit 2c generates work data necessary for the worker to perform the annotation work.
  • This work data includes, for at least one task, original data to be worked on and work instructions to be given by the worker to this original data.
  • the work data includes: Contains original data and work instructions for multiple cases. The number of pieces of work data to be generated is determined by the contents of the project (the number of pieces of work per set) stored in the project storage unit 2g. read out.
  • the data transmission unit 2a collectively transmits the work data generated by the data management unit 2c to the work terminal 4.
  • a work screen for a certain task is displayed on the work terminal 4 as illustrated in FIG.
  • an image of a cat is displayed as a work target, and a work instruction to paint over the area of the cat is displayed.
  • the worker fills in the area of the cat according to this work instruction, and then taps the work confirmation button A.
  • annotation data is generated to which the result of the work performed by the worker in response to the work instruction (filled cat area) is added as a tag for the original data (cat image).
  • FIG. 5 is an explanatory diagram showing the management structure of annotation data in the annotation data storage unit 2i.
  • “task set ID”, “worker”, “task ID”, “work result”, "original file name” and “task work time” are associated with each other.
  • “task set ID” is a unique number for identifying a task set, and is numbered for each task set.
  • “Worker” is the worker who performed the work of this task set.
  • “Task ID” is a unique number for identifying a task, and is numbered for each task.
  • a task ID is associated with a project, and which task belongs to which project is managed.
  • the “work result” is the work result annotated by the worker.
  • "Original file name” is the file name of the original data that is the object of annotation.
  • "Task working time” is the time required for working on one task. This work time is counted by the annotation application installed in the work terminal 4 and transmitted to the data collection system 2 together with the annotation data.
  • worker "A” performs 10 tasks for task set ID "001"
  • the resulting "work result”, "original file name” and "task work time” are the task managed by each.
  • the point granting unit 2d grants points to the worker who made the annotation according to a predetermined rule as an incentive for the worker. This point is given uniformly to all workers who have made annotations, regardless of whether the work result is correct or not. Also, when giving points, the current rating of the worker is considered, and the rating is reflected in the acquired points. The current rating for each worker is stored in the rating storage unit 2j, and the point giving unit 2d acquires the rating of the corresponding worker from the rating storage unit 2j.
  • the rating of a certain worker may be reflected in the point grant rate, and the points granted to the worker may be calculated by multiplying the standard points by the point grant rate.
  • a point correction value (including plus/minus) may be used instead of the point award rate.
  • the given points are calculated by adding or subtracting the point correction value to the standard points.
  • a rating distribution may be generated for all workers involved in a certain project, and points to be given may be determined based on the positions of the workers in this distribution.
  • the point giving unit 2d updates the point storage unit 2k based on the points given this time. In the point storage unit 2k, the cumulative points acquired by the workers so far are stored for each worker and for each work type (image system, text system, audio system), and the current points are added to the cumulative points. .
  • the correct answer determination unit 2e reads annotation data collected from a plurality of workers regarding the same task from the annotation data storage unit 2i, and determines the results of the work in the annotation data by majority, in other words, the statistical mode. , to determine the correct answer for this task. For example, in the example of FIG. 5, three workers A to C duplicated work for task ID "1", thereby producing three work results "AAA-result", “BBB-result", and "BBB-result”. "CCC-result" is obtained. In this case, the correct answer for the task ID "1" is determined by the majority of the three work results.
  • the correct answer is identified by comparing how many times each option has been selected.
  • the correct answer is derived by evaluating whether or not a plurality of work results are similar using an object detection evaluation method.
  • IoU Intersection over Union
  • AP Average Precision
  • mAP mean Average Precision
  • the work evaluation unit 2f evaluates the workers involved in the majority decision based on the correct answer determined by the majority decision.
  • the evaluation of the worker is performed by collating the correct answer by majority decision with the work result of the worker, and the evaluation result is reflected in the rating of the worker. For example, among the three workers A to C who participated in the majority decision, the workers A and B who performed the work corresponding to the correct answer will be rated higher, and the other workers, that is, the work corresponding to the incorrect answer, will be rated higher. Lower the rating for worker C who did it.
  • the ratings stored in the rating storage unit 2j are updated for each of the workers A to C.
  • the correct answer determination unit 2e if the correct answer cannot be uniquely identified by majority vote, for example, if there are multiple options with the maximum votes in the selection type task, the current rating is uniformly given to all workers involved in this majority vote. Appropriate action is taken, such as maintaining, uniformly lowering the current rating, or uniformly increasing the current rating.
  • the correct answer by majority vote is used exclusively for evaluating the workers who made the annotations, and is not used for the purpose of selecting the annotation data collected from the workers.
  • the collected annotation data is provided as-is to the requester as a deliverable (work result) for the work request.
  • the work evaluation unit 2f may update the rating from the following viewpoints, in addition to the viewpoint of whether or not the above work results are correct.
  • the work evaluation unit 2f lowers the rating of the worker whose task work time is too short by pressing the work confirmation button A repeatedly.
  • ten task working times T1 to T10 for worker A are individually compared with a predetermined threshold value, and if even one is less than the threshold value, It judges that the work time is too short, and lowers the rating of worker A.
  • the rating may be increased for a worker whose task working time is determined to be appropriate by comparison with the threshold value.
  • the work evaluation unit 2f lowers the rating of the worker whose overall work time for the task set is statistically too short.
  • the total working time of the task set is specified as the sum of the task working times T1-T10 shown in FIG. Specifically, as shown in FIG. 6, first, the distribution of the total work time is generated for all workers involved in one project. Next, if the worker is significantly out of the maximum work time by comparison with a predetermined threshold value, it is determined that the total work time is too short, and the rating of the worker is lowered. Note that the rating may be increased for a worker determined to be within the maximum working hours by comparison with the threshold value.
  • the work evaluation unit 2f lowers the worker's rating when the worker selects an option other than the obvious correct answer in a selection-type task that includes an obvious correct answer.
  • Clearly correct answers are managed in the data collection system 2, unlike normal options, and it is judged that the worker who selects anything other than the obvious correct answer is not serious about the work.
  • An example of an obvious correct answer is "Yes” as an option to "Is there a cat in the image?" for an image of a cat.
  • the work evaluation unit 2f lowers the rating of the worker when the worker selects a clearly incorrect option in a selection-type task including a clearly incorrect answer.
  • Clearly incorrect answers are managed in the data collection system 2, unlike normal options, and it is judged that the worker who selects a clearly incorrect answer is not doing the work seriously.
  • An example of an obvious incorrect answer is "No" as an option to "Is there a cat in the image?" for an image of a cat.
  • the correct answer for the same task is determined by a majority vote of the work results collected from a plurality of workers.
  • the number of times a given task is to be performed in duplicate is set in advance for each requested item (project) as the "number of times the same task is to be performed in duplicate".
  • the correct answer determined by majority vote has objective rationality, and therefore can be used as an evaluation criterion for evaluating the workers involved in this majority vote. Thereby, even if the data collection system 2 does not grasp the correct answer of the task, it becomes possible to appropriately evaluate the worker who made the annotation.
  • the worker who has made the annotation is given points as an incentive for the worker after considering the rating set for each worker.
  • the rating is updated for each worker according to the result of matching the correct answer determined by the majority with the work results of the workers involved in the correct answer, and is applied in the next and subsequent point assignments.
  • the behavior of the worker is checked based on the working time of the task, the total working time of the task set, or the clear correct/incorrect answers managed by the data collection system 2 side.
  • the present invention can also be regarded as a data collection program that causes a computer constituting the data collection system 2 to execute the above-described processing, and a data collection method using a computer.
  • annotation system data collection system 2a data transmission unit 2b data reception unit 2c data management unit 2d point provision unit 2e correct answer determination unit 2f work evaluation unit 2g project storage unit 2h original data storage unit 2i annotation data storage unit 2j rating storage unit 2k Point storage unit 3 request terminal 4 work terminal

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Artificial Intelligence (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

[Problème] Évaluer des travailleurs qui ont fourni des annotations lors de la collecte de données d'annotation pour une intelligence artificielle. [Solution] Une unité de transmission de données 2a transmet, sous la forme d'une certaine tâche, des données originales et une instruction de travail à exécuter par un travailleur à un terminal de travail. Une unité de réception de données 2b reçoit, depuis le terminal de travail, des données d'annotation dans lesquelles le résultat du travail effectué par le travailleur en relation avec la tâche est ajouté aux données d'origine sous forme d'étiquette. Les données d'annotation reçues sont stockées dans une unité de stockage de données d'annotation 2i. Une unité de détermination de réponse correcte 2e lit, à partir de l'unité de stockage de données d'annotation 2i, des données d'annotation collectées pour la même tâche parmi une pluralité de travailleurs, et détermine une réponse correcte pour cette tâche sur la base du résultat de travail le plus courant dans les données d'annotation lues. Une unité d'évaluation de travail 2f évalue les travailleurs qui ont fourni les données d'annotation, en utilisant, comme critère d'évaluation, la réponse correcte déterminée sur la base du résultat de travail le plus courant.
PCT/JP2021/016183 2021-04-21 2021-04-21 Système de collecte de données et programme associé WO2022224378A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2021/016183 WO2022224378A1 (fr) 2021-04-21 2021-04-21 Système de collecte de données et programme associé
JP2023515951A JPWO2022224378A1 (fr) 2021-04-21 2021-04-21

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/016183 WO2022224378A1 (fr) 2021-04-21 2021-04-21 Système de collecte de données et programme associé

Publications (1)

Publication Number Publication Date
WO2022224378A1 true WO2022224378A1 (fr) 2022-10-27

Family

ID=83722133

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/016183 WO2022224378A1 (fr) 2021-04-21 2021-04-21 Système de collecte de données et programme associé

Country Status (2)

Country Link
JP (1) JPWO2022224378A1 (fr)
WO (1) WO2022224378A1 (fr)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018106662A (ja) * 2016-12-22 2018-07-05 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
WO2019003485A1 (fr) * 2017-06-30 2019-01-03 株式会社Ａｂｅｊａ Système informatique et procédé d'apprentissage automatique ou d'inférence

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018106662A (ja) * 2016-12-22 2018-07-05 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
WO2019003485A1 (fr) * 2017-06-30 2019-01-03 株式会社Ａｂｅｊａ Système informatique et procédé d'apprentissage automatique ou d'inférence

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
BURMANIA ALEC; PARTHASARATHY SRINIVAS; BUSSO CARLOS: "Increasing the Reliability of Crowdsourcing Evaluations Using Online Quality Assessment", IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, IEEE, USA, vol. 7, no. 4, 1 October 2016 (2016-10-01), USA , pages 374 - 388, XP011635114, ISSN: 1949-3045, DOI: 10.1109/TAFFC.2015.2493525 *
CAO HOUWEI; COOPER DAVID G.; KEUTMANN MICHAEL K.; GUR RUBEN C.; NENKOVA ANI; VERMA RAGINI: "CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset", IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, IEEE, USA, vol. 5, no. 4, 1 October 2014 (2014-10-01), USA , pages 377 - 390, XP011565283, ISSN: 1949-3045, DOI: 10.1109/TAFFC.2014.2336244 *

Also Published As

Publication number Publication date
JPWO2022224378A1 (fr) 2022-10-27

Similar Documents

Publication Publication Date Title
Vis et al. Health technology assessment frameworks for eHealth: A systematic review
AU2020200909A1 (en) Evaluation control
CN104885116B (zh) 法庭用系统，法庭用方法，以及法庭用程序
CN112346567A (zh) 基于ai的虚拟交互模型生成方法、装置及计算机设备
CN109670727A (zh) 一种基于众包的分词标注质量评估系统及评估方法
CN111694937B (zh) 基于人工智能的面试方法、装置、计算机设备及存储介质
CN110688478B (zh) 一种答案排序方法、装置及存储介质
CN114187988A (zh) 数据处理的方法及装置、系统、存储介质
CN109636345A (zh) 一种业务办理工作流的智能管理方法及系统
CN117745048B (zh) 一种人工智能辅助的创意流程管理方法
CN111444341A (zh) 用户画像构建方法、装置、设备及可读存储介质
CN112418779A (zh) 一种基于自然语言理解的在线自助面试方法
CN114330267A (zh) 一种基于语义学关联的结构化报告模板的设计方法
CN112734142B (zh) 基于深度学习的资源学习路径规划方法及装置
WO2022224378A1 (fr) Système de collecte de données et programme associé
CN114023462A (zh) 一种基于图形化表示的计算机化临床指南构建方法及装置
JP7488974B1 (ja) 情報処理システム、情報処理方法及びプログラム
CN113918609A (zh) 试卷创建方法、装置、计算机设备和存储介质
KR20230092678A (ko) 비대면 온라인 교육 플랫폼 제공 방법 및 시스템
CN113469291A (zh) 数据处理方法、装置、电子设备及存储介质
KR101802377B1 (ko) 유형별 퀴즈 기반의 지식 수집용 클라우드 소싱 서비스 시스템
JP7518313B1 (ja) 検索支援システム、検索支援方法及びプログラム
CN118132741B (zh) 一种基于Agent的自主学习方法及系统
JP5815911B1 (ja) 文書分析システム、文書分析システムの制御方法、および、文書分析システムの制御プログラム
US20240370831A1 (en) System and method of automating the recruitment process between employment candidates and recruiting entities

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21937872

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023515951

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED DD/MM/YYYY15/02/2024

122 Ep: pct application non-entry in european phase

Ref document number: 21937872

Country of ref document: EP

Kind code of ref document: A1