GB2500303A - Estimating reliability of predicted classifications made by a machine learning algorithm - Google Patents

Estimating reliability of predicted classifications made by a machine learning algorithm Download PDF

Info

Publication number
GB2500303A
GB2500303A GB1301447.7A GB201301447A GB2500303A GB 2500303 A GB2500303 A GB 2500303A GB 201301447 A GB201301447 A GB 201301447A GB 2500303 A GB2500303 A GB 2500303A
Authority
GB
United Kingdom
Prior art keywords
reliability
training
classifier
label
prediction
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB1301447.7A
Other languages
English (en)
Other versions
GB201301447D0 (en
Inventor
Ruty Rinott
Noam Slonim
Ehud Aharoni
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of GB201301447D0 publication Critical patent/GB201301447D0/en
Publication of GB2500303A publication Critical patent/GB2500303A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/35—Clustering; Classification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
    • G06F18/2193—Validation; Performance evaluation; Active pattern learning techniques based on specific statistical tests
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/048—Fuzzy inferencing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/10—Office automation; Time management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Business, Economics & Management (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Public Health (AREA)
  • Biomedical Technology (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • General Business, Economics & Management (AREA)
  • Tourism & Hospitality (AREA)
  • Quality & Reliability (AREA)
  • Operations Research (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Pathology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Epidemiology (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Automation & Control Theory (AREA)
  • Fuzzy Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
GB1301447.7A 2012-02-19 2013-01-28 Estimating reliability of predicted classifications made by a machine learning algorithm Withdrawn GB2500303A (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US13/400,070 US9087303B2 (en) 2012-02-19 2012-02-19 Classification reliability prediction

Publications (2)

Publication Number Publication Date
GB201301447D0 GB201301447D0 (en) 2013-03-13
GB2500303A true GB2500303A (en) 2013-09-18

Family

ID=47890864

Family Applications (1)

Application Number Title Priority Date Filing Date
GB1301447.7A Withdrawn GB2500303A (en) 2012-02-19 2013-01-28 Estimating reliability of predicted classifications made by a machine learning algorithm

Country Status (4)

Country Link
US (2) US9087303B2 (de)
CN (1) CN103258239B (de)
DE (1) DE102013202457A1 (de)
GB (1) GB2500303A (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
RU2672394C1 (ru) * 2017-07-26 2018-11-14 Общество С Ограниченной Ответственностью "Яндекс" Способы и системы для оценки обучающих объектов посредством алгоритма машинного обучения

Families Citing this family (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10915557B2 (en) * 2013-01-31 2021-02-09 Walmart Apollo, Llc Product classification data transfer and management
US20150142717A1 (en) * 2013-11-19 2015-05-21 Microsoft Corporation Providing reasons for classification predictions and suggestions
US10043112B2 (en) * 2014-03-07 2018-08-07 Qualcomm Incorporated Photo management
US10325220B2 (en) * 2014-11-17 2019-06-18 Oath Inc. System and method for large-scale multi-label learning using incomplete label assignments
EP3136270B1 (de) * 2015-08-26 2021-02-24 Viavi Solutions Inc. Identifizierung von rohmaterial mit spektroskopie
JP6946081B2 (ja) * 2016-12-22 2021-10-06 キヤノン株式会社 情報処理装置、情報処理方法、プログラム
WO2018130442A1 (en) 2017-01-11 2018-07-19 Koninklijke Philips N.V. Method and system for automated inclusion or exclusion criteria detection
CN107330522B (zh) * 2017-07-04 2021-06-08 北京百度网讯科技有限公司 用于更新深度学习模型的方法、装置及系统
CN108052987B (zh) * 2017-12-29 2020-11-13 苏州体素信息科技有限公司 图像分类输出结果的检测方法
US11455493B2 (en) 2018-05-16 2022-09-27 International Business Machines Corporation Explanations for artificial intelligence based recommendations
US10990452B2 (en) * 2018-07-24 2021-04-27 Vmware, Inc. Reliability determination of workload migration activities
CN110858326B (zh) * 2018-08-15 2024-06-07 第四范式（北京）技术有限公司 模型训练及获取附加特征数据的方法、装置、设备及介质
CN110059743B (zh) * 2019-04-15 2021-10-29 北京致远慧图科技有限公司 确定预测的可靠性度量的方法、设备和存储介质
US11568169B2 (en) * 2019-04-28 2023-01-31 International Business Machines Corporation Identifying data drifts that have an adverse effect on predictors
US10891546B2 (en) 2019-04-29 2021-01-12 Google Llc Network anomaly detection
US11710068B2 (en) * 2019-11-24 2023-07-25 International Business Machines Corporation Labeling a dataset
US11941496B2 (en) * 2020-03-19 2024-03-26 International Business Machines Corporation Providing predictions based on a prediction accuracy model using machine learning
CN111797895B (zh) * 2020-05-30 2024-04-26 华为技术有限公司 一种分类器的训练方法、数据处理方法、系统以及设备
CN112836772A (zh) * 2021-04-02 2021-05-25 四川大学华西医院 基于LightGBM集成多个BERT模型的随机对照试验识别方法
DE102022209898A1 (de) * 2022-09-20 2024-03-21 Siemens Mobility GmbH Sichere steuerung von technisch-physikalischen systemen
CN116757534B (zh) * 2023-06-15 2024-03-15 中国标准化研究院 一种基于神经训练网络的智能冰箱可靠性分析方法

Family Cites Families (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5822741A (en) * 1996-02-05 1998-10-13 Lockheed Martin Corporation Neural network/conceptual clustering fraud detection architecture
JP2002092305A (ja) * 2000-09-13 2002-03-29 Hitachi Ltd スコア算出方法及びスコア提供方法
US7107254B1 (en) * 2001-05-07 2006-09-12 Microsoft Corporation Probablistic models and methods for combining multiple content classifiers
US20030126606A1 (en) * 2001-12-27 2003-07-03 Koninklijke Philips Esectronics N.V. Hierarchical decision fusion of recommender scores
US7512496B2 (en) 2002-09-25 2009-03-31 Soheil Shams Apparatus, method, and computer program product for determining confidence measures and combined confidence measures for assessing the quality of microarrays
US7644049B2 (en) 2004-11-19 2010-01-05 Intel Corporation Decision forest based classifier for determining predictive importance in real-time data analysis
US7577709B1 (en) 2005-02-17 2009-08-18 Aol Llc Reliability measure for a classifier
JP4735973B2 (ja) * 2006-03-27 2011-07-27 学校法人明治大学 電力価格ゾーン予測方法、及び電力価格ゾーン予測プログラム
US7890438B2 (en) * 2007-12-12 2011-02-15 Xerox Corporation Stacked generalization learning for document annotation
IL188726A (en) 2008-01-10 2013-05-30 Deutsche Telekom Ag A stacking scheme for tasks was classified
US20110106734A1 (en) 2009-04-24 2011-05-05 Terrance Boult System and appartus for failure prediction and fusion in classification and recognition
CN102110365B (zh) * 2009-12-28 2013-11-06 日电（中国）有限公司 基于时空关系的路况预测方法和系统
US8924313B2 (en) * 2010-06-03 2014-12-30 Xerox Corporation Multi-label classification using a learned combination of base classifiers

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
None *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
RU2672394C1 (ru) * 2017-07-26 2018-11-14 Общество С Ограниченной Ответственностью "Яндекс" Способы и системы для оценки обучающих объектов посредством алгоритма машинного обучения
US11416765B2 (en) 2017-07-26 2022-08-16 Yandex Europe Ag Methods and systems for evaluating training objects by a machine learning algorithm

Also Published As

Publication number Publication date
CN103258239B (zh) 2016-12-28
US9342789B2 (en) 2016-05-17
US9087303B2 (en) 2015-07-21
CN103258239A (zh) 2013-08-21
GB201301447D0 (en) 2013-03-13
US20130218813A1 (en) 2013-08-22
DE102013202457A1 (de) 2013-08-22
US20150262070A1 (en) 2015-09-17

Similar Documents

Publication Publication Date Title
US9342789B2 (en) Classification reliability prediction
US11475358B2 (en) Annotation pipeline for machine learning algorithm training and optimization
JP6839342B2 (ja) 情報処理装置、情報処理方法およびプログラム
US9257052B2 (en) Evaluating candidate answers to questions in a target knowledge domain
US10971255B2 (en) Multimodal learning framework for analysis of clinical trials
US20180260735A1 (en) Training a hidden markov model
Raturi et al. An analytical approach for health data analysis and finding the correlations of attributes using decision tree and W-logistic modal process
US20140214732A1 (en) Inconsistency detection between structured and non-structured data
US20200394211A1 (en) Multi-term query subsumption for document classification
US11734143B2 (en) Performance measurement of predictors
Ramlakhan et al. Building artificial intelligence and machine learning models: a primer for emergency physicians
US20160232213A1 (en) Information Processing System, Information Processing Method, and Recording Medium with Program Stored Thereon
US9720759B2 (en) Server, model applicability/non-applicability determining method and non-transitory computer readable medium
US9189764B2 (en) Usage of quantitative information gain to support decisions in sequential clinical risk assessment examinations
Jafarzadeh et al. Bayesian modeling and inference for diagnostic accuracy and probability of disease based on multiple diagnostic biomarkers with and without a perfect reference standard
US20220391724A1 (en) Unsupervised Anomaly Detection With Self-Trained Classification
Chen et al. Tracking drift severity in data streams
Farhad et al. Keep your distance: determining sampling and distance thresholds in machine learning monitoring
CN112183714B (zh) 基于人工神经网络的自动化数据切片
CN114496116A (zh) 挥发性有机化合物检测和分类
KR101868744B1 (ko) 임상 진료 지침 제공 방법 및 이를 실행하는 프로그램이 기록된 기록 매체
CN115349129A (zh) 生成具有不确定性区间的性能预测
Klonecki et al. Cost-constrained Group Feature Selection Using Information Theory
Wang et al. Biomarker evaluation under imperfect nested case‐control design
Kaur et al. Improved Framework for Bug Severity Classification using N-gram Features with Convolution Neural Network

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)