WO2021055847A1 - Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine - Google Patents

Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine Download PDF

Info

Publication number
WO2021055847A1
WO2021055847A1 PCT/US2020/051627 US2020051627W WO2021055847A1 WO 2021055847 A1 WO2021055847 A1 WO 2021055847A1 US 2020051627 W US2020051627 W US 2020051627W WO 2021055847 A1 WO2021055847 A1 WO 2021055847A1
Authority
WO
WIPO (PCT)
Prior art keywords
outlier
activity
model
data
machine learning
Prior art date
Application number
PCT/US2020/051627
Other languages
English (en)
Inventor
Jones RICHARD B.
Original Assignee
Hartford Steam Boiler Inspection And Insurance Company
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hartford Steam Boiler Inspection And Insurance Company filed Critical Hartford Steam Boiler Inspection And Insurance Company
Priority to CN202080072815.4A priority Critical patent/CN114556382A/zh
Priority to CA3154671A priority patent/CA3154671A1/fr
Priority to GB2204238.6A priority patent/GB2603358B/en
Priority to JP2022517214A priority patent/JP7399269B2/ja
Priority to BR112022005003A priority patent/BR112022005003A2/pt
Priority to EP20785887.9A priority patent/EP4022532A1/fr
Priority to KR1020227012917A priority patent/KR20220066924A/ko
Publication of WO2021055847A1 publication Critical patent/WO2021055847A1/fr
Priority to JP2023205241A priority patent/JP2024026276A/ja

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/01—Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

  • the present disclosure generally relates to improved computer-based systems, computing components and computing objects configured to implement bias reduction in machine learning models.
  • a machine learning model may include one or more computers or processing devices to form predictions or determinations based on patterns and inferences learned from sample/training data. Bias in sample/training data selection can propagate into machine learning model predictions and determinations.
  • Embodiments of the present disclosure include methods for dynamic outlier bias reduced machine learning models.
  • the methods include receiving, by at least one processor, a training data set of target variables representing at least one activity -related attribute for at least one user activity; receiving, by the at least one processor, at least one bias criteria used to determine one or more outliers; determining, by the at least one processor, a set of model parameters for a machine learning model including: (1) applying, by the at least one processor, the machine learning model having a set of initial model parameters to the training data set to determine a set of model predicted values; (2) generating, by the at least one processor, an error set of data element errors by comparing the set of model predicted values to corresponding actual values of the training data set; (3) generating, by the at least one processor, a data selection vector to identify non-outlier target variables based at least in part on the error set of data element errors and the at least one bias criteria; (4) utilizing, by the at least one processor, the data selection vector on the training data set to generate a non-
  • Embodiments of the present disclosure include systems for dynamic outlier bias reduced machine learning models.
  • the systems include at least one processor in communication with a non-transitory computer-readable storage medium having software instructions stored thereon, where the software instructions, when executed, cause the at least one processor to perform steps to: receive a training data set of target variables representing at least one activity-related attribute for at least one user activity; receive at least one bias criteria used to determine one or more outliers; determine a set of model parameters for a machine learning model including: (1) apply the machine learning model having a set of initial model parameters to the training data set to determine a set of model predicted values; (2) generate an error set of data element errors by comparing the set of model predicted values to corresponding actual values of the training data set; (3) generate a data selection vector to identify non-outlier target variables based at least in part on the error set of data element errors and the at least one bias criteria; (4) utilize the data selection vector on the training data set to generate a non-outlier data set; (5) determine a set of updated model parameters
  • the systems and methods of embodiments of the present disclosure further including: applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set ; training, by the at least one processor, using the outlier training data set, at least one outlier-specific model parameter of at least one outlier-specific machine learning model to predict outlier data values; and utilizing, by the at least one processor, the outlier-specific machine learning model to predict outlier activity-related data values for the set of outlier activity -related data.
  • the systems and methods of embodiments of the present disclosure further including: training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the generalized machine learning model to predict outlier activity-related data values for the set of outlier activity-related data; and utilizing, by the at least one processor, the generalized machine learning model to predict the activity-related data values.
  • the systems and methods of embodiments of the present disclosure further including: applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set; training, by the at least one processor, using the outlier training data set, an outlier-specific model parameters of an outlier-specific machine learning model to predict outlier data values; training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the outlier-specific machine learning model to predict outlier activity-related data values for the set of outlier activity-related data; and utilizing, by the at least one processor, the outlier-specific machine learning model to predict the activity -related data values.
  • the systems and methods of embodiments of the present disclosure further including: training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the generalized machine learning model to predict the activity-related data values for the set of activity-related data; utilizing, by the at least one processor, the outlier classifier machine learning model to identify outlier activity-related data values of the activity-related data values; and removing, by the at least one processor, the outlier activity -related data values.
  • the training data set includes the at least one activity-related attribute of concrete compressive strength as a function of concrete composition and concrete curing exposure.
  • training data set includes the at least one activity-related attribute of energy use data as a function of household environmental conditions and lighting conditions.
  • the systems and methods of embodiments of the present disclosure further including: receiving, by the at least one processor, an application programming interface (API) request to generate a prediction with at least one data element; and instantiating, by the at least one processor, at least one cloud computing resource to schedule execution of the machine learning model; utilizing, by the at least one processor according to the schedule for execution, the machine learning model to predict at least one activity-related data element value for the at least one data element; and returning, by the at least one processor, the at least one activity- related data element value to a computing device associated with the API request.
  • API application programming interface
  • the training data set includes the at least one activity-related attribute of three-dimensional patient imagery of a medical dataset; and where the machine learning model is configured to predict the activity-related data values including two or more physically-based rendering parameters based on the medical dataset.
  • the training data set includes the at least one activity-related attribute of simulated control results for electronic machine commands; and where the machine learning model is configured to predict the activity -related data values including control commands for the electronic machine.
  • the systems and methods of embodiments of the present disclosure further including: splitting, by the at least one processor, the set of activity-related data into a plurality of subsets of activity -related data; determining, by the at least one processor, an ensemble model for each subset of activity-related data of the plurality of subsets of activity-related data; where the machine learning model includes an ensemble of models; where each ensemble model includes a random combination of models from the ensemble of models; utilizing, by the at least one processor, each ensemble model separately to predict ensemble-specific activity-related data values; determining, by the at least one processor, an error for each ensemble model based on the ensemble-specific activity-related data values and known values; and selecting, by the at least one processor, a highest performing ensemble model based on a lowest error.
  • FIGS. 1-14B show one or more schematic flow diagrams, certain computer-based architectures, and/or screenshots of various specialized graphical user interfaces which are illustrative of some exemplary aspects of at least some embodiments of the present disclosure.
  • the term “real-time” is directed to an event/action that can occur instantaneously or almost instantaneously in time when another event/action has occurred.
  • the “real-time processing,” “real-time computation,” and “real-time execution” all pertain to the performance of a computation during the actual time that the related physical process (e.g., a user interacting with an application on a mobile device) occurs, in order that results of the computation can be used in guiding the physical process.
  • events and/or actions in accordance with the present disclosure can be in real-time and/or based on a predetermined periodicity of at least one of: nanosecond, several nanoseconds, millisecond, several milliseconds, second, several seconds, minute, several minutes, hourly, several hours, daily, several days, weekly, monthly, etc.
  • exemplary inventive, specially programmed computing systems with associated devices are configured to operate in the distributed network environment, communicating with one another over one or more suitable data communication networks (e.g., the Internet, satellite, etc.) and utilizing one or more suitable data communication protocols/modes such as, without limitation, IPX/SPX, X.25, AX.25, AppleTalk(TM), TCP/IP (e.g., HTTP), near-field wireless communication (NFC), RFID, Narrow Band Internet of Things (NBIOT), 3G, 4G, 5G, GSM, GPRS, WiFi, WiMax, CDMA, satellite, ZigBee, and other suitable communication modes.
  • the NFC can represent a short- range wireless communications technology in which NFC-enabled devices are “swiped,” “bumped,” “tap” or otherwise moved in close proximity to communicate.
  • a machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device).
  • a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others.
  • computer engine and “engine” identify at least one software component and/or a combination of at least one software component and at least one hardware component which are designed/programmed/configured to manage/control other software and/or hardware components (such as the libraries, software development kits (SDKs), objects, etc.).
  • SDKs software development kits
  • Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, and so forth), integrated circuits, application specific integrated circuits (ASIC), programmable logic devices (PLD), digital signal processors (DSP), field programmable gate array (FPGA), logic gates, registers, semiconductor device, chips, microchips, chip sets, and so forth.
  • the one or more processors may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors; x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU).
  • the one or more processors may be dual-core processor(s), dual-core mobile processor(s), and so forth.
  • Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces (API), instruction sets, computing code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. Determining whether an embodiment is implemented using hardware elements and/or software elements may vary in accordance with any number of factors, such as desired computational rate, power levels, heat tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds and other design or performance constraints.
  • One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein.
  • Such representations known as “IP cores” may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the logic or processor.
  • IP cores may be stored on a tangible, machine readable medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that make the logic or processor.
  • various embodiments described herein may, of course, be implemented using any appropriate hardware and/or computing software languages (e.g., C++, Objective-C, Swift, Java, JavaScript, Python, Perl, QT, etc ).
  • one or more of exemplary inventive computer-based devices of the present disclosure may include or be incorporated, partially or entirely into at least one personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile internet device (MID), messaging device, data communication device, and so forth.
  • PC personal computer
  • laptop computer ultra-laptop computer
  • tablet touch pad
  • portable computer handheld computer
  • palmtop computer personal digital assistant
  • PDA personal digital assistant
  • cellular telephone combination cellular telephone/PDA
  • television smart device (e.g., smart phone, smart tablet or smart television), mobile internet device (MID), messaging device, data communication device, and so forth.
  • smart device e.g., smart phone, smart tablet or smart television
  • MID mobile internet device
  • server should be understood to refer to a service point which provides processing, database, and communication facilities.
  • server can refer to a single, physical processor with associated communications and data storage and database facilities, or it can refer to a networked or clustered complex of processors and associated network and storage devices, as well as operating software and one or more database systems and application software that support the services provided by the server. Cloud servers are examples.
  • one or more of exemplary inventive computer-based systems of the present disclosure may obtain, manipulate, transfer, store, transform, generate, and/or output any digital object and/or data unit (e.g., from inside and/or outside of a particular application) that can be in any suitable form such as, without limitation, a file, a contact, a task, an email, a tweet, a map, an entire application (e.g., a calculator), etc.
  • any digital object and/or data unit e.g., from inside and/or outside of a particular application
  • any suitable form such as, without limitation, a file, a contact, a task, an email, a tweet, a map, an entire application (e.g., a calculator), etc.
  • one or more of exemplary inventive computer-based systems of the present disclosure may be implemented across one or more of various computer platforms such as, but not limited to: (1) AmigaOS, AmigaOS 4, (2) FreeBSD, NetBSD, OpenBSD, (3) Linux, (4) Microsoft Windows, (5) OpenVMS, (6) OS X (Mac OS), (7) OS/2, (8) Solaris, (9) Tru64 UNIX, (10) VM, (11) Android, (12) Bada, (13) BlackBerry OS, (14) Firefox OS, (15) iOS, (16) Embedded Linux, (17) Palm OS, (18) Symbian, (19) Tizen, (20) WebOS, (21) Windows Mobile, (22) Windows Phone, (23) Adobe AIR, (24) Adobe Flash, (25) Adobe Shockwave, (26) Binary Runtime Environment for Wireless (BREW), (27) Cocoa (API), (28) Cocoa Touch, (29) Java Platforms, (30) JavaFX, (31) JavaFX Mobile, (32) Microsoft XNA, (33) Mono, (34) Mozilla Prism,
  • exemplary inventive computer-based systems, and/or exemplary inventive computer-based devices of the present disclosure may be configured to utilize hardwired circuitry that may be used in place of or in combination with software instructions to implement features consistent with principles of the disclosure.
  • implementations consistent with principles of the disclosure are not limited to any specific combination of hardware circuitry and software.
  • various embodiments may be embodied in many different ways as a software component such as, without limitation, a stand-alone software package, a combination of software packages, or it may be a software package incorporated as a “tool” in a larger software product.
  • exemplary software specifically programmed in accordance with one or more principles of the present disclosure may be downloadable from a network, for example, a website, as a stand-alone product or as an add-in package for installation in an existing software application.
  • exemplary software specifically programmed in accordance with one or more principles of the present disclosure may also be available as a client-server software application, or as a web-enabled software application.
  • exemplary software specifically programmed in accordance with one or more principles of the present disclosure may also be embodied as a software package installed on a hardware device.
  • exemplary inventive computer-based systems/platforms, exemplary inventive computer-based devices, and/or exemplary inventive computer-based components of the present disclosure may be configured to handle numerous concurrent users that may be, but is not limited to, at least 100 (e.g., but not limited to, 100-999), at least 1,000 (e.g., but not limited to, 1,000-9,999 ), at least 10,000 (e.g., but not limited to, 10,000-99,999 ), at least 100,000 (e.g., but not limited to, 100,000-999,999), at least 1,000,000 (e.g., but not limited to, 1,000,000-9,999,999), at least 10,000,000 (e.g., but not limited to, 10, 000, GOO- 99, 999, 999), at least 100,000,000 (e.g., but not limited to, 100,000,000-999,999,999), at least 1,000,000,000 (e.g., but not limited to, 1,000,000,000-10,000,000,000).
  • exemplary inventive computer-based systems and/or exemplary inventive computer-based devices of the present disclosure may be configured to output to distinct, specifically programmed graphical user interface implementations of the present disclosure (e.g., a desktop, a web app., etc.).
  • a final output may be displayed on a displaying screen which may be, without limitation, a screen of a computer, a screen of a mobile device, or the like.
  • the display may be a holographic display.
  • the display may be a transparent surface that may receive a visual projection.
  • Such projections may convey various forms of information, images, and/or objects.
  • such projections may be a visual overlay for a mobile augmented reality (MAR) application.
  • MAR mobile augmented reality
  • cloud As used herein, terms “cloud,” “Internet cloud,” “cloud computing,” “cloud architecture,” and similar terms correspond to at least one of the following: (1) a large number of computers connected through a real-time communication network (e.g., Internet); (2) providing the ability to run a program or application on many connected computers (e.g., physical machines, virtual machines (VMs)) at the same time; (3) network-based services, which appear to be provided by real server hardware, and are in fact served up by virtual hardware (e.g., virtual servers), simulated by software running on one or more real machines (e.g., allowing to be moved around and scaled up (or down) on the fly without affecting the end user).
  • a real-time communication network e.g., Internet
  • VMs virtual machines
  • the exemplary inventive computer-based systems and/or the exemplary inventive computer-based devices of the present disclosure may be configured to securely store and/or transmit data by utilizing one or more of encryption techniques (e.g., private/public key pair, Triple Data Encryption Standard (3DES), block cipher algorithms (e.g., IDEA, RC2, RC5, CAST and Skipjack), cryptographic hash algorithms (e.g., MD5, RIPEMD- 160, RTRO, SHA-1, SHA-2, Tiger (TTH), WHIRLPOOL, RNGs).
  • encryption techniques e.g., private/public key pair, Triple Data Encryption Standard (3DES), block cipher algorithms (e.g., IDEA, RC2, RC5, CAST and Skipjack), cryptographic hash algorithms (e.g., MD5, RIPEMD- 160, RTRO, SHA-1, SHA-2, Tiger (TTH), WHIRLPOOL, RNGs).
  • the term “user” shall have a meaning of at least one user.
  • the terms “user”, “subscriber” “consumer” or “customer” should be understood to refer to a user of an application or applications as described herein and/or a consumer of data supplied by a data provider.
  • the terms “user” or “subscriber” can refer to a person who receives data provided by the data or service provider over the Internet in a browser session, or can refer to an automated software application which receives the data and stores or processes the data.
  • FIG. 1 depicts a block diagram of an exemplary computer-based system 100 for bias reduction in machine learning in accordance with one or more embodiments of the present disclosure.
  • the exemplary inventive computing devices and/or the exemplary inventive computing components of the exemplary computer-based system 100 may be configured to manage a large number of members and/or concurrent transactions, as detailed herein.
  • the exemplary computer-based system/platform 100 may be based on a scalable computer and/or network architecture that incorporates varies strategies for assessing the data, caching, searching, and/or database connection pooling, including dynamic outlier bias reduction (DOBR) as described in embodiments herein.
  • DOBR dynamic outlier bias reduction
  • An example of the scalable architecture is an architecture that is capable of operating multiple servers.
  • members 102-104 (e.g., clients) of the exemplary computer-based system 100 may include virtually any computing device capable of receiving and sending a message over a network (e.g., cloud network), such as network 105, to and from another computing device, such as servers 106 and 107, each other, and the like.
  • a network e.g., cloud network
  • the member devices 102-104 may be personal computers, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, and the like.
  • one or more member devices within member devices 102-104 may include computing devices that typically connect using a wireless communications medium such as cell phones, smart phones, pagers, walkie talkies, radio frequency (RF) devices, infrared (IR) devices, CBs, integrated devices combining one or more of the preceding devices, or virtually any mobile computing device, and the like.
  • RF radio frequency
  • IR infrared
  • one or more member devices within member devices 102-104 may be devices that are capable of connecting using a wired or wireless communication medium such as a PDA, POCKET PC, wearable computer, a laptop, tablet, desktop computer, a netbook, a video game device, a pager, a smart phone, an ultra-mobile personal computer (UMPC), and/or any other device that is equipped to communicate over a wired and/or wireless communication medium (e.g., NFC, RFID, NBIOT, 3G, 4G, 5G, GSM, GPRS, WiFi, WiMax, CDMA, satellite, ZigBee, etc ).
  • a wired or wireless communication medium such as a PDA, POCKET PC, wearable computer, a laptop, tablet, desktop computer, a netbook, a video game device, a pager, a smart phone, an ultra-mobile personal computer (UMPC), and/or any other device that is equipped to communicate over a wired and/or wireless communication medium (e.g., NFC, RFID
  • one or more member devices within member devices 102-104 may include may run one or more applications, such as Internet browsers, mobile applications, voice calls, video games, videoconferencing, and email, among others. In some embodiments, one or more member devices within member devices 102-104 may be configured to receive and to send web pages, and the like.
  • applications such as Internet browsers, mobile applications, voice calls, video games, videoconferencing, and email, among others.
  • one or more member devices within member devices 102-104 may be configured to receive and to send web pages, and the like.
  • an exemplary specifically programmed browser application of the present disclosure may be configured to receive and display graphics, text, multimedia, and the like, employing virtually any web based language, including, but not limited to Standard Generalized Markup Language (SMGL), such as HyperText Markup Language (HTML), a wireless application protocol (WAP), a Handheld Device Markup Language (HDML), such as Wireless Markup Language (WML), WMLScript, XML, JavaScript, and the like.
  • SMGL Standard Generalized Markup Language
  • HTML HyperText Markup Language
  • WAP wireless application protocol
  • HDML Handheld Device Markup Language
  • WMLScript Wireless Markup Language
  • a member device within member devices 102- 104 may be specifically programmed by either Java, .Net, QT, C, C++ and/or other suitable programming language.
  • one or more member devices within member devices 102-104 may be specifically programmed include or execute an application to perform a variety of possible tasks, such as, without limitation, messaging functionality, browsing, searching, playing, streaming or displaying various forms of content, including locally stored or uploaded messages, images and/or video, and/or games.
  • the exemplary network 105 may provide network access, data transport and/or other services to any computing device coupled to it.
  • the exemplary network 105 may include and implement at least one specialized network architecture that may be based at least in part on one or more standards set by, for example, without limitation, Global System for Mobile communication (GSM) Association, the Internet Engineering Task Force (IETF), and the Worldwide Interoperability for Microwave Access (WiMAX) forum.
  • GSM Global System for Mobile communication
  • IETF Internet Engineering Task Force
  • WiMAX Worldwide Interoperability for Microwave Access
  • the exemplary network 105 may implement one or more of a GSM architecture, a General Packet Radio Service (GPRS) architecture, a Universal Mobile Telecommunications System (UMTS) architecture, and an evolution of UMTS referred to as Long Term Evolution (LTE).
  • GSM Global System for Mobile communication
  • IETF Internet Engineering Task Force
  • WiMAX Worldwide Interoperability for Microwave Access
  • the exemplary network 105 may implement one or more of a
  • the exemplary network 105 may include and implement, as an alternative or in conjunction with one or more of the above, a WiMAX architecture defined by the WiMAX forum. In some embodiments and, optionally, in combination of any embodiment described above or below, the exemplary network 105 may also include, for instance, at least one of a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an enterprise IP network, or any combination thereof.
  • LAN local area network
  • WAN wide area network
  • VLAN virtual LAN
  • VPN layer 3 virtual private network
  • enterprise IP network or any combination thereof.
  • At least one computer network communication over the exemplary network 105 may be transmitted based at least in part on one of more communication modes such as but not limited to: NFC, RFID, Narrow Band Internet of Things (NBIOT), ZigBee, 3G, 4G, 5G, GSM, GPRS, WiFi, WiMax, CDMA, satellite and any combination thereof.
  • the exemplary network 105 may also include mass storage, such as network attached storage (NAS), a storage area network (SAN), a content delivery network (CDN) or other forms of computer or machine- readable media.
  • NAS network attached storage
  • SAN storage area network
  • CDN content delivery network
  • the exemplary server 106 or the exemplary server 107 may be a web server (or a series of servers) running a network operating system, examples of which may include but are not limited to Microsoft Windows Server, Novell NetWare, or Linux.
  • the exemplary server 106 or the exemplary server 107 may be used for and/or provide cloud and/or network computing.
  • the exemplary server 106 or the exemplary server 107 may have connections to external systems like email, SMS messaging, text messaging, ad content providers, etc. Any of the features of the exemplary server 106 may be also implemented in the exemplary server 107 and vice versa.
  • one or more of the exemplary servers 106 and 107 may be specifically programmed to perform, in non-limiting example, as authentication servers, search servers, email servers, social networking services servers, SMS servers, IM servers, MMS servers, exchange servers, photo-sharing services servers, advertisement providing servers, fmancial/banking-related services servers, travel services servers, or any similarly suitable service-base servers for users of the member computing devices 101-104.
  • one or more exemplary computing member devices 102-104, the exemplary server 106, and/or the exemplary server 107 may include a specifically programmed software module that may be configured to send, process, and receive information using a scripting language, a remote procedure call, an email, a tweet, Short Message Service (SMS), Multimedia Message Service (MMS), instant messaging (IM), internet relay chat (IRC), mIRC, Jabber, an application programming interface, Simple Object Access Protocol (SOAP) methods, Common Object Request Broker Architecture (CORBA), HTTP (Hypertext Transfer Protocol), REST (Representational State Transfer), or any combination thereof.
  • SMS Short Message Service
  • MMS Multimedia Message Service
  • IM instant messaging
  • IRC internet relay chat
  • mIRC Jabber
  • SOAP Simple Object Access Protocol
  • CORBA Common Object Request Broker Architecture
  • HTTP Hypertext Transfer Protocol
  • REST Real-S Transfer Protocol
  • FIG. 2 depicts a block diagram of another exemplary computer-based system/platform 200 in accordance with one or more embodiments of the present disclosure.
  • the member computing devices 202a, 202b through 202n shown each at least includes a computer-readable medium, such as a random-access memory (RAM) 208 coupled to a processor 210 or FLASH memory.
  • the processor 210 may execute computer-executable program instructions stored in memory 208.
  • the processor 210 may include a microprocessor, an ASIC, and/or a state machine.
  • the processor 210 may include, or may be in communication with, media, for example computer-readable media, which stores instructions that, when executed by the processor 210, may cause the processor 210 to perform one or more steps described herein.
  • examples of computer-readable media may include, but are not limited to, an electronic, optical, magnetic, or other storage or transmission device capable of providing a processor, such as the processor 210 of client 202a, with computer-readable instructions.
  • suitable media may include, but are not limited to, a floppy disk, CD-ROM, DVD, magnetic disk, memory chip, ROM, RAM, an ASIC, a configured processor, all optical media, all magnetic tape or other magnetic media, or any other medium from which a computer processor can read instructions.
  • various other forms of computer-readable media may transmit or carry instructions to a computer, including a router, private or public network, or other transmission device or channel, both wired and wireless.
  • the instructions may comprise code from any computer-programming language, including, for example, C, C++, Visual Basic, Java, Python, Perl, JavaScript, and etc.
  • member computing devices 202a through 202n may also comprise a number of external or internal devices such as a mouse, a CD-ROM, DVD, a physical or virtual keyboard, a display, or other input or output devices.
  • examples of member computing devices 202a through 202n e.g., clients
  • member computing devices 202a through 202n may be specifically programmed with one or more application programs in accordance with one or more principles/methodologies detailed herein.
  • member computing devices 202a through 202n may operate on any operating system capable of supporting a browser or browser-enabled application, such as MicrosoftTM, WindowsTM, and/or Linux.
  • member computing devices 202a through 202n shown may include, for example, personal computers executing a browser application program such as Microsoft Corporation's Internet ExplorerTM, Apple Computer, Inc.'s SafariTM, Mozilla Firefox, and/or Opera.
  • users, 212a through 212n may communicate over the exemplary network 206 with each other and/or with other systems and/or devices coupled to the network 206.
  • exemplary server devices 204 and 213 may be also coupled to the network 206.
  • one or more member computing devices 202a through 202n may be mobile clients.
  • At least one database of exemplary databases 207 and 215 may be any type of database, including a database managed by a database management system (DBMS).
  • DBMS database management system
  • an exemplary DBMS-managed database may be specifically programmed as an engine that controls organization, storage, management, and/or retrieval of data in the respective database.
  • the exemplary DBMS-managed database may be specifically programmed to provide the ability to query, backup and replicate, enforce rules, provide security, compute, perform change and access logging, and/or automate optimization.
  • the exemplary DBMS-managed database may be chosen from Oracle database, IBM DB2, Adaptive Server Enterprise, FileMaker, Microsoft Access, Microsoft SQL Server, MySQL, PostgreSQL, and a NoSQL implementation.
  • the exemplary DBMS-managed database may be specifically programmed to define each respective schema of each database in the exemplary DBMS, according to a particular database model of the present disclosure which may include a hierarchical model, network model, relational model, object model, or some other suitable organization that may result in one or more applicable data structures that may include fields, records, files, and/or objects.
  • the exemplary DBMS-managed database may be specifically programmed to include metadata about the data that is stored.
  • the exemplary inventive computer-based systems/platforms, the exemplary inventive computer-based devices, and/or the exemplary inventive computer-based components of the present disclosure may be specifically configured to operate in a cloud computing/architecture such as, but not limiting to: infrastructure a service (IaaS), platform as a service (PaaS), and/or software as a service (SaaS).
  • Figures 3 and 4 illustrate schematics of exemplary implementations of the cloud computing/architecture(s) in which the exemplary inventive computer-based systems/platforms, the exemplary inventive computer-based devices, and/or the exemplary inventive computer-based components of the present disclosure may be specifically configured to operate.
  • Dynamic Outlier Bias Reduction may be used to improve the accuracy and understanding of generalized linear models specifically for benchmarking studies.
  • DOBR Dynamic Outlier Bias Reduction
  • it is a method that may be applied to a wide variety of analysis models where there are one or more independent variables and one dependent variable.
  • the present disclosure, and embodiments therein, are illustrative of the inventive application of DOBR to improving the accuracy of machine learning model predictions.
  • DOBR is not a predictive model. Instead, in embodiments, it is an add-on method to predictive or interpretive models that can improve the accuracy of model predictions.
  • DOBR identified outliers are based on the difference between the data supplied target variable and the model computed value. As outliers are identified, via a pre-determined selection criterion, the outlier dependent data records and model produced dependent variables are removed from the analysis. Further analysis may continue with these records permanently removed. However, in other embodiments of the exemplary inventive system and methods, at each model iteration, the outlier identification process includes the entire dataset so that all records undergo outlier scrutiny using the last iteration’s predictive model as defined by its calculation parameters.
  • FIG. 5 illustrates a block diagram of an exemplary inventive bias reduction system in machine learning in accordance with one or more embodiments of the present disclosure.
  • a bias reduction system 300 may include component for dynamic outlier bias reduction (DOBR) in datasets under analysis by, e.g., machine learning engines.
  • DOBR provides an iterative process to remove outlier records subject to a pre-defmed criterion.
  • This condition is the user-defined error acceptance value expressed as a percentage. It refers to how much error the user is willing to accept in the model based potentially on their insights and other analysis results that will be described later in this discussion. A value of 100% signifies that all of the error is accepted and no records will be removed in the DOBR process. If 0% is chosen, then all of the records are removed. Generally, error acceptance values in the range of 80 to 95% have been observed for industrial applications.
  • a user may interact with the bias reduction system 300 to administer the error acceptance value via a user input device 308 and view results via a display device 312, among other user interaction behaviors using the display device 312 and user input device 308.
  • the bias reduction system 300 may analyze a dataset 311 received into a database 310 or other storage in communication with the bias reduction system 300.
  • the bias reduction system 300 may receive the dataset 311 via the database 310 or other storage device and make predictions using one or more machine learning models with dynamic outlier bias reduction for improved accuracy and efficiency.
  • the bias reduction system 300 includes a combination of hardware and software components, including, e.g., storage and memory devices, cache, buffers, a bus, input/output (I/O) interfaces, processors, controllers, networking and communications devices, an operating system, a kernel, device drivers, among other components.
  • a processor 307 is in communication with multiple other components to implement functions of the other components.
  • each component has time scheduled on the processor 307 for execution of component functions, however in some embodiments, each component is scheduled to one or more processors in a processing system of the processor 307. In other embodiments, each component has its own processor included therewith.
  • components of the bias reduction system 300 may include, e.g., a DOBR engine 301 in communication with a model index 302 and model library 303, a regressor parameter library 305, a classifier parameter library 304 and a DOBR filter 306, among other possible components.
  • Each component may include a combination of hardware and software to implement component functions, such as, e.g., memory and storage devices, processing devices, communications devices, input/output (I/O) interfaces, controllers, networking and communications devices, an operating system, a kernel, device drivers, a set of instructions, among other components.
  • the DOBR engine 301 includes a model engine for instantiating and executing machine learning models.
  • the DOBR engine 301 may access models for instantiation in a model library 303 through the use of a model index 302.
  • the model library 303 may include a library of machine learning models that may be selectively accessed and instantiated for use by an engine such as the DOBR engine 301.
  • the model library 303 may include machine learning models such as, e.g., a support vector machine (SVM), a Linear Regressor, a Lasso model, Decision Tree regressors, Decision Tree classifiers, Random Forest regressors, Random Forest classifiers, K Neighbors regressors, K Neighbors classifiers, Gradient Boosting regressors, Gradient Boosting classifiers, among other possible classifiers and regressors.
  • SVM support vector machine
  • the model library 303 may import models according to the following example pseudo-code 1 : Pseudo-Code 1 [0059]
  • the DOBR engine 301 may employ a model index 302 that indexes each model to a model identifier to use as a function by the DOBR engine 301.
  • models including, e.g., Linear Regression, XGBoost Regression, Support Vector Regression, Lasso, K Neighbors Regression, Bagging Regression, Gradient Boosting Regression, Random Forest Regression, Decision Tree Regression, among other regression models and classification models, may be indexed by a number identifier and labeled with a name.
  • pseudo-code 2 depicts an example of a model index code for use by the model index 302.
  • the software instructions are stored within a memory of the respective model library 303 or model index 302 and buffered in a cache for provision to the processor 307.
  • the DOBR engine 301 may utilize the model index 302 by accessing or calling the index via communications and/or I/O devices, the use the index to call models as functions from the model library 303 via communications and/or
  • the bias reduction system 300 may record model parameters in, e.g., memory or storage, such as, e.g., hard drives, solid state drives, random access memory (RAM), flash storage, among other storage and memory devices.
  • regressor parameters may be logged and adjusted in a regressor parameter library 305.
  • the regressor parameter library 305 may include storage and communication hardware configured with sufficient memory and bandwidth to store, adjust and communicate a multitude of parameters for multiple regressors, e.g., in real time.
  • respective parameters may be initialized and updated in the regressor parameter library 305.
  • a user via the user input device 308, may establish an initial set of parameters.
  • the initial set of parameters may be predetermined or randomly generated.
  • the DOBR engine 301 may correlate a model from as identified in the model index 302 to a set of parameters in the regressor parameter library 305.
  • the DOBR engine 301 may call a set of parameters according to, e.g., an identification (ID) number associated with a given regression model.
  • ID identification
  • the regressor parameter library 305 may identify parameters for each regression model similar to pseudo-code 3 below:
  • classifier parameters may be logged and adjusted in a classifier parameter library 304.
  • the classifier parameter library 304 may include storage and communication hardware configured with sufficient memory and bandwidth to store, adjust and communicate a multitude of parameters for multiple regressors, e.g., in real time. For example, for each classification machine learning model instantiated by the DOBR engine 301, respective parameters may be initialized and updated in the regressor parameter library 305.
  • a user via the user input device 308, may establish an initial set of parameters. However, in some embodiments, the initial set of parameters may be predetermined.
  • the DOBR engine 301 may correlate a model from as identified in the model index 302 to a set of parameters in the regressor parameter library 305.
  • the DOBR engine 301 may call a set of parameters according to, e.g., an identification (ID) number associated with a given regression model.
  • the regressor parameter library 305 may identify parameters for each regression model similar to pseudo-code 4 below:
  • the DOBR engine 301 may load one or more instantiated and initialized models, e.g., into a cache or buffer of the DOBR engine 301.
  • the dataset 311 may then be loaded from the database 310 into, e.g., a same or different cache or buffer or other storage device of the DOBR engine 301.
  • the processor 307 or a processor in the DOBR engine 301 may then execute each model to transform the dataset 311 into, e.g., a respective prediction of activity-related data values that characterize the results or parameters of an activity based on certain input attributes related to the activity. For example, appliance energy usage in home and/or commercial environments, concrete compressive strength in a variety of applications and formulations, object or image recognition, speech recognition, or other machine learning applications.
  • the DOBR engine 301 may be modelling appliance energy usage based on a dataset 311 of historical energy usage, time of year, time of day, location, among other factors.
  • the DOBR engine 301 may called a set of regressors from the model library 303 via the model index 302 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then called a parameter file or log associated with regressors for appliance energy usage estimation in the regressor parameter library 305 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then utilize a processor 307 to predict a future energy consumption based on the models and model parameters, time and date, location, or other factor and combinations thereof.
  • the DOBR engine 301 may be modelling concrete compressive strength based on a dataset 311 of concrete materials, time of year, time of day, location, humidity, curing time, age, among other factors.
  • the DOBR engine 301 may called a set of regressors from the model library 303 via the model index 302 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then called a parameter file or log associated with regressors for concrete compressive strength estimation in the regressor parameter library 305 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then utilize a processor 307 to predict a future concrete compressive strength based on the models and model parameters for a particular concrete formulation, time and date, location, or other factor and combinations thereof.
  • the DOBR engine 301 may be performing speech recognition based on a dataset 311 of utterances and ground-truth transcriptions, among other factors.
  • the DOBR engine 301 may called a set of classifiers from the model library 303 via the model index 302 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then called a parameter file or log associated with classifiers for speech recognition in the classifier parameter library 304 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then utilize a processor 307 to predict a transcription of recorded speech data based on the models and model parameters for a set of one or more utterances.
  • the DOBR engine 301 may be automatically predicting rendering settings for medical imagery based on a dataset 311 of settings for multiple rendering parameters across imaging and/or visualizations, among other factors, as described in U.S. Patent No. 10,339,695.
  • the DOBR engine 301 may called a set of classifiers from the model library 303 via the model index 302 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then called a parameter file or log associated with classifiers for rendering settings in the classifier parameter library 304 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then utilize a processor 307 to predict a rendering settings data based on the models and model parameters for a set of one or more medical datasets.
  • the DOBR engine 301 may be performing robotic control of machinery based on a dataset 311 of machine control command results and simulated results of machine control commands, among other factors, as described in U.S. Patent No. 10,317,854.
  • the DOBR engine 301 may called a set of regression models from the model library 303 via the model index 302 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then called a parameter file or log associated with regression model for robotic control in the regressor parameter library 305 connected to a bus of the DOBR engine 301.
  • the DOBR engine 301 may then utilize a processor 307 to predict a a success or failure of a particular control commands based on the models and model parameters for a set of control commands, environmental information, sensor data and/or simulations of the commands..
  • the bias reduction system 300 may implement the machine learning models in a cloud environment, e.g., as a cloud service for remote users.
  • a cloud service may be designed to support large numbers of users and a wide variety of algorithms and problem sizes, including those described above, as well as other potential models, datasets and parameter tunings specific to a user’s use case, as described in U.S. Patent No. 10,452,992.
  • a number of programmatic interfaces may be defined by the service in which the bias reduction system 300 is implemented, which guide non-expert users to start using machine learning best practices relatively quickly, without the users having to expend a lot of time and effort on tuning models, or on learning advanced statistics or artificial intelligence techniques.
  • the interfaces may, for example, allow non-experts to rely on default settings or parameters for various aspects of the procedures used for building, training and using machine learning models, where the defaults are derived from the one or more sets of parameters in the classifier parameter library 304 and/or regressor parameter library 305 for similar models to the individual user.
  • the default settings or parameters may be used as a starting point to customize a user’s machine learning model using training with the user’s datasets via the DOBR engine 301 and optimizer 306.
  • users may customize the parameters or settings they wish to use for various types of machine learning tasks, such as input record handling, feature processing, model building, execution and evaluation.
  • the cloud-service bias reduction system 300 may have extendable built-in capabilities of the service, e.g., by registering customized functions with the service.
  • the modules may in some cases be shared with other users of the service, while in other cases the use of the customized modules may be restricted to their implementers/owners.
  • the bias reduction system 300 may include models in the model library 303 that enable an ensemble approach to machine learning model training and implementation, as described in U.S. Patent No. 9,646,262. Such an approach may be useful for applications to data analytics using electronic datasets of electronic activity data.
  • the database 310 may include one or more structured or unstructured data sources.
  • An unsupervised learning module in certain embodiments, is configured to assemble an unstructured data set into an organized data set using a plurality of unsupervised learning techniques, e.g., in an ensemble of models from the model library 303.
  • the unsupervised learning module is configured to assemble an unstructured data set into multiple versions of an organized data set
  • a supervised learning module in certain embodiments, is configured to generate one or more machine learning ensembles based on each version of multiple versions of an organized data set and to determine which machine learning ensemble exhibits a highest predictive performance according to, e.g., model error after training each model in each ensemble using the DOBR engine 301 and optimizer 306.
  • outliers in the dataset 311 may reduce the accuracy of the implemented models, thus increasing the number of training iterations to achieve an accurate set of parameters for a given model in a given application.
  • the DOBR engine 301 may include a DOBR filter 301b to dynamically test data point errors in the dataset to determine outliers.
  • outliers may be removed to provide a more accurate or representative dataset 311.
  • the DOBR filter 301b may provide an iterative mechanism for removing outlier data points subject to a pre-defmed criterion, e.g., the user-defined error acceptance value described above and provided, e.g., by a user via the user input device 308.
  • the user-defined error acceptance value expressed as a percentage where, e.g., a value of 100% signifies that all of the error is accepted and no data points will be removed by the filter 301b, while a value of, e.g., 0% results in all of the data points being removed.
  • the filter 301b may be configured with an error acceptance value in the range of between, e.g., about 80% and about 95%.
  • the filter 301b may be configured to perform functions as depicted in pseudo-code 6 below: Pseudo-Code 6
  • the DOBR filter 301b works in conjunction with an optimizer 306, which is configured to determine error and optimize parameters for each model in the regressor parameter library 305 and the classifier parameter library 304.
  • the optimizer 306 may determine model and communicate the error to the filter 301b of the DOBR engine 301.
  • the optimizer 306 may include, e.g., storage and/or memory devices and communication devices with sufficient memory capacity and bandwidth to receive the dataset 311 and model predictions and determine, e.g., outliers, convergence, error, absolute value error, among other error measures.
  • the optimizer 306 may be configured to perform functions as depicted in pseudo-code 7 below:
  • the bias reduction system 300 may then return to a user, via, e.g., the display 312, machine learning model predictions, an outlier analysis, a convergence of predictions, among other data produced by the DOBR engine 301 in a more accurate and efficient manner due to the reduction in outliers that would otherwise bias predictions.
  • FIG. 6 illustrates a flowchart of an exemplary inventive methodology in accordance with one or more embodiments of the present disclosure.
  • DOBR such as the DOBR engine 301 and filter 301b described above, provides an iterative process to remove outlier records subject to a pre-defmed criterion.
  • This condition is the user-defined error acceptance value expressed as a percentage. It refers to how much error the user is willing to accept in the model based potentially on their insights and other analysis results that will be described later in this discussion. A value of 100% signifies that all of the error is accepted and no records will be removed in the DOBR process. If 0% is chosen, then all of the records are removed.
  • error acceptance values in the range of 80 to 95% have been observed for industrial applications.
  • DOBR DOBR
  • embodiments of the DOBR methodology can also determine the percentage of the dataset that represents model outliers. This pre-analysis step can assist in setting the proper error acceptance value or if outliers are present at all.
  • C is a function of the error acceptance criterion
  • f() is a comparative function
  • y is a value of a data record
  • y pred is a predicted value
  • y tar is a target value.
  • P x is a percentile function
  • i is an index of a record entry
  • m is the number of record entries.
  • DOBR can be implemented to reduce bias in training the model M 408.
  • the solution model M 408 is implemented by a model engine, including, e.g., a processing device and a memory and/or storage device.
  • the exemplary methodology computes model coefficients, M(c ) 402 and model estimates ⁇ y pred ⁇ 410 for all records applying the solution model, M 408, to the complete input dataset (x, y tar ] 404 according to, e.g., equation 3 below:
  • a total error function 418 computes initial model total error e 0 according to, e.g., equation 4 below:
  • an error function 412 computes model errors according to, e.g., equation 5 below:
  • the error function 412 computes new a data record selection vector ⁇ I ⁇ according to, e.g., equation 6 below:
  • a data record selector 414 computes the non-outlier data records to be included in model computation by selecting only records where the record selection vector is equal to 1, according to, e.g., equation 7 below:
  • the model 408 with the latest coefficients 402 computes new predicted values 420 and model coefficients 402 from the DOBR selected data records 416 according to, e.g., equation 8 below:
  • the model 408 using the new model coefficients computes new prediction values 420 for the complete dataset.
  • This step reproduces computing the predicted values 420 for the DOBR selected records in the formal steps, but in practice the new model can be applied to just the DOBR removed records according to, e.g., equation 9 below:
  • the total error function 418 computes model total error according to, e.g., equation 10 below:
  • a convergence test 424 tests model convergence according to, e.g., equation 11 below:
  • b is a convergence criteria 422, such as, e.g., 0.5%.
  • the convergence test 424 may terminate the iterative process if, for example, the percent error is less than, e.g., 0.5%. Otherwise, the process may return to the initial dataset 404. Each of the steps above may then be performed and the convergence criteria 422 retested. The process is repeated until the convergence test 424 is below the convergence criteria 424.
  • FIG. 7 is a graph illustrating an example of relationship between model error and an error acceptance criterion of another exemplary computer-based machine learning model with reduced bias in accordance with one or more embodiments of the present disclosure.
  • the change in slope at this point suggests the model’s variability is not changing with respect to removing data records, or in other words, no outliers are present at these levels of error acceptance.
  • This visual test can help set the appropriate error acceptance level and also determine if DOBR is needed at all. If the slope of the line in FIG. 7 does not change, then the model accounts satisfactorily for the observed variability in the data. There are no model outliers and DOBR does not need to be applied.
  • the and model error versus value can be used as a metric for identifying a best performing model or ensemble of models for a particular scenario. Because different datasets may vary in degree of linearity, the exact oc value for the data and for the model may change the performance of the model.
  • the model error as a function of error acceptance level can be used to determine the degree to which a given model can account for variability in data by having a model error that indicates more or less tolerance for data variability in order to form accurate predictions.
  • precision and accuracy in model predictions may be tuned by selecting a model and/or model parameters that exhibit, e.g., low model error for a high error acceptance value to select for a model that is more tolerant to outlier data.
  • model selection may be automated by employing, e.g., rule- based programming and/or machine learning models to identify the best performing model for a dataset according to a balance of model error and error acceptance criteria.
  • a model may be automatically selected that optimally accounts for outliers in the dataset.
  • model error may be compared across models for one or more error acceptance values, with the model having the lowest model error being automatically selected to generate predictions.
  • the DOBR machine learning techniques according to aspects of the present disclosure provide more effective model training, as well as improved visibility into data and model behaviors for individual dataset.
  • machine learning models can be more effectively and efficiently trialed for various types of data.
  • the model performance can then be more efficiently assessed to determine that an optimal model for the application and for the type of data.
  • artificial intelligence applications may be improved with models selected and trained using DOBR for the type of intelligence being produced.
  • business intelligence and data analytics, as well as other applications such as physical behavior prediction, content recommendation, resource use predictions, natural language processing, and other machine learning applications, may be improved by using DOBR to both tune model parameters and select models based on outlier characteristics and model error in response to outliers.
  • FIG. 8 is a graph illustrating an example of relationship between model error and an error acceptance criterion of another exemplary computer-based machine learning model with reduced bias in accordance with one or more embodiments of the present disclosure.
  • DOBR concrete compression strength dataset 504 downloaded from the University of Califomia-Irvine’s machine learning data repository. This data set contains 1030 observations, records, or instances with 8 independent variables. The first seven describe the concrete composition with age given in days: cement amount, superplasticizer, blast furnace slag, coarse aggregate, fly ash, fine aggregate, water, and age.
  • the output variable is concrete compressive strength measured in megapascals (MPa). For comparison, 1 MPa 145 psi.
  • a linear regression model is constructed according to, e.g., equation 12 below:
  • a i are coefficient computed by a linear regression model
  • x i are observations of the 8 variables
  • i is the variable index
  • FIG. 9 and FIG. 10 displays the results of these calculations using the complete dataset 512 (FIG. 9) and the DOBR version (FIG. 10) with the outliers identified and removed from the calculation.
  • the outlier values 516 marked in red crosses, are computed from the non-outlier model. Both of these plots show the actual versus predicted target values with the diagonal line 510 and 514, respectively for FIG. 9 and FIG. 10, depicting equality.
  • the complete dataset calculation (FIG. 9) shows how outliers can bias results.
  • the DOBR modified plot (FIG. 10) shows the bias removed with the diagonal line 514 bisecting the non-outlier values 518 and also apparent groups of outlier data points 516 that may warrant further study.
  • FIG. 9 is a graph illustrating an example of relationship between compressive strength and predicted compressive strength of a baseline computer-based machine learning model without reduced bias in accordance with one or more embodiments of the present disclosure.
  • FIG. 10 is a graph illustrating an example of relationship between compressive strength and predicted compressive strength of another exemplary computer-based machine learning model with reduced bias in accordance with one or more embodiments of the present disclosure.
  • the identification of outliers and the patterns they sometime form in the above type of plots have been useful for additional benefits of the DOBR method in industrial applications.
  • the outliers can form patterns or groups that simply are not observed by other methods. This information is created by simply using DOBR with the analyst supplied model. No additional information or assumptions are required. In practice, the DOBR defined outlier set can supply useful information to improve, provide insights, or validate the underlying model.
  • FIG. 11 is a block diagram of another exemplary computer-based system for machine learning predictions with DOBR in accordance with one or more embodiments of the present disclosure.
  • a machine learning procedure starts with a dataset, X, consisting of n independent variables and in records in length and an array (m x 1) of target variables, ⁇ .
  • the dataset [X, ? ⁇ is divided into two randomly selected subsets of pre-determined size: one to train the model and the other to test its predictive accuracy, as per, e.g., equation 13 below:
  • a 70%/30% split of is used for training (n records) and testing (j records) (e.g., 70 % of the records are training and 30 % are testing), however any suitable split may be employed, such as, e.g., 50%/50%, 60%/40%, 80%/20%, 90%/10%, 95%/5%, or other suitable train/test split.
  • a machine learning model, L trained using is tested by computing a set of predicted target variables, expressed as in, e.g., equation 14 below:
  • the model accuracy is then measured as the norm, which may have, e.g., the following form:
  • outliers in model predictions ⁇ y pred ⁇ .
  • ⁇ y pred ⁇ are due to the inability of the model L. to transform the specific input values to prediction values near the known target variable.
  • the input data for these records contain effects of factors and/or phenomena that the model cannot map to reality as given by the target variables. Keeping these records in the dataset can bias the results since the model coefficients are computed under the assumption that all data records are equally valid.
  • the DOBR process described above works for a given dataset where the analyst desires the best model that fits the data by removing outliers that adversely biases the results. It increases the predictive accuracy of the model by restricting the model solution to a subset of the initial dataset that has the outliers removed.
  • a DOBR assisted solution has two output results: a) A set of x values, model parameters, and model solutions for which the model describes the data, and b) A set of x values, model parameters, and model solutions for which the model does not describes the data.
  • DOBR in addition to computing a more accurate model for the restricted dataset, in embodiments, DOBR also provides an outlier dataset that can be further studied relative to the given model to understand the cause or causes of high model prediction error.
  • the predictive model is computed from the training data and that model alone is used in the testing phase. Since, by design the testing phase may not use the target values to determine outliers, the DOBR methodology described above with reference to FIG. 6 may not apply. However, there is an exemplary aspect of the DOBR methodology may not have been utilized above: the outlier - non-outlier classification potential as suggested by DOBR’s output results mentioned previously.
  • the dataset may be divided into two randomly selected parts: one for training and one for testing. In the training phase both the independent and target variables are kept, but in testing the target variables are hidden andthe independent variables are used to predict the target variable. The known target variable values are only used to measure the model’s predictive error.
  • DOBR can be implemented to reduce bias in training the machine learning model L 608.
  • the machine learning model L 608 is implemented by a model engine, including, e.g., a processing device and a memory and/or storage device.
  • the exemplary methodology model estimates, ⁇ y train ⁇ 606 for all records applying the machine learning model L 608, to the complete input dataset 604 according to, e.g., equation 16 below:
  • the total error function 618 compute initial model total error e 0 according to, e.g., equation 17 below:
  • error function 612 computes model errors according to, e.g., equation 18 below:
  • E is a predicted record error
  • k denotes an iteration
  • the error function 612 computes new data record selection vector according to, e.g., equation 19 below:
  • a data record selector 614 computes the non-outlier data records to be included in model computation by selecting only records where the record selection vector is equal to 1, according to, e.g., equation 20 below:
  • the machine learning module 608 with the latest coefficients 602 compute new predicted values 620 for the complete training set 604 using the DOBR selected data records according to, e.g., equation 21 below:
  • the total error function 618 compute model total error according to, e.g., equation 22 below:
  • a convergence test 624 test model convergence according to, e.g., equation 23 below:
  • b is a convergence criteria 622, such as, e.g., 0.5%.
  • the convergence test 624 may terminate the iterative process if, for example, the percent error is less than, e.g., 0.5%. Otherwise, the process may return to the training dataset 604.
  • the DOBR iteration procedure measures how well the model can predict itself rather than measuring its accuracy relative to the test dataset.
  • the objective here is to test the capability of the model to predict the target variable and records with large deviations are systematically removed to improve the model’s ability to focus on the larger majority of the data where the data predictions are relatively good. This process must be done on the same dataset. It doesn’t make any sense to remove records from the training set if outliers are identified in the testing set.
  • This process is fundamental to the DOBR method in that the records that are removed in a previous iteration are re-entered after a new model (new model parameters) have been computed. This process requires that the same dataset be used.
  • this iteration procedure is performed after the learning model is defined.
  • the user selects the machine learning algorithm and then determines the specific hyperparameters that “tune” or configure the model.
  • These parameters can be chosen using standard techniques such as cross-validation or simply by plotting testing error as a function of the specific, user-supplied parameter ranges.
  • the particular values used may optimize prediction accuracy versus computing time while ensuring the model is neither under- or over-fitted. There are several robust tools to assist in this process but the user experience and intuition are also valuable advantages in selecting the best model hyperparameters.
  • the Error Acceptance vs Model Error plot is computed from this step by applying a sequence of error acceptance values and tabulating or plotting the results. These plots identify the fraction of the dataset that are outliers in the sense that their error contribution is marginally larger than the error contribution of data records that fit the model. Also in practice, these plots can show more than one type of variation not explained by the model. The slope can vary as it convergences to the slope of the model. These variations can assist in researching the nature of additional data-coded behavior that is unexplained by the model. The records that occupy the different slope intervals can be identified and their further investigation can provide insights that may help in constructing an even more robust model.
  • Model 1 where is a reference model that is used as a basis to measure accuracy improvements
  • the errors associated with Model 1 and Model 2 are, e.g., respectively.
  • the base model suggests it might be a better predictor for non-outlier records.
  • the test dataset is uncensored, containing both non outliers and outliers. Therefore, is it uncertain if applying a non-outlier, customized model to uncensored test data will produce a better predictive model compared to However, in many cases, may be observed to be either statistically equal to or greater than [0150]
  • the DOBR model computed from the selected (non outlier) records always produces a lower model error since the identified outlier records are omitted. In the limiting case of no outliers the DOBR model error equals the total model error since the datasets are the same.
  • the objective may be to develop a model using a subset of available data (training) and then measure its predictive accuracy on another subset (testing).
  • the DOBR methodology removes model outliers each iteration before computing model parameters.
  • machine learning model development this can be done in the training phase, but by definition, the target values in testing can only be used to measure the model’s predictive accuracy without advanced knowledge of outliers.
  • This observation means that the standard DOBR methodology may be generalized utilizing more of the DOBR model information computed in the training phase.
  • FIG. 11 is a block diagram of another exemplary computer-based system for machine learning with reduced bias in accordance with one or more embodiments of the present disclosure.
  • the DOBR selected training dataset values for the non-outliers ( x,y) in upon training, as described above, the following information is produced: the DOBR selected training dataset values for the non-outliers ( x,y) in , the DOBR training data selection vector for the non-outliers ⁇ l in ⁇ , the DOBR selected training dataset values for the outliers ( x,y) out , and the DOBR training data selection vector for the outliers
  • DOBR classifies the training data into two mutually exclusive subsets.
  • selection vectors that provide a binary: (non-outlier or outlier) classification value for each record in the training dataset, e.g., according to equation 24 below:
  • the complete set of the training data attributes, x train , and the DOBR produced classifications, ⁇ / train ⁇ are used to construct/train a classifier machine learning model, C, e.g., stored in the model library 303.
  • This model is applied to the testing dataset, x test to classify the testing data records as outliers or non-outliers based on the training dataset DOBR-established knowledge.
  • the classifier machine learning model C is implemented according to equation 25 below:
  • ⁇ I c ⁇ produces two testing predictive datasets; and respectively.
  • the above information creates several possible “full dataset” predictive models for analysis of the test dataset.
  • the three that have shown the most predictive improvements for the entire dataset are:
  • the machine learning model L 608 is defined by the non-outlier data, and applied to the DOBR test classified data to predict the non-outlier test values. The same procedure is done for the outlier data.
  • the objective of this combination is to use the most precise predictive model with its corresponding dataset. In other words, this model tests the overall predictive accuracy of the non-outlier and outlier models applied separately on their respective datasets that were defined with the DOBR classification.
  • the machine learning model L 608 is defined by the training data, and is also applied to the DOBR test classified data
  • This model uses the broad knowledge of L(x,y) train to predict the target values of the DOBR defined outlier and non-outlier x values. The purpose of this model is test the predictive accuracy of the full training model applied separately to the DOBR classified non-outlier and outlier datasets.
  • the third model ⁇ y 3 ⁇ is a hybrid that joins the predictive characteristics of the previous two approaches. This model tests the predictive benefit, if any, of joining the model 608 trained on the total training with L(x, y) out , the specific model trained on the DOBR classified outliers in the training set applied to their respective, classified datasets. There are additional hybrid models that may be explored in further research. [0160] In each of these three models and other embodiments, the complete test dataset is predicted utilizing both the DOBR classified non-outlier and outlier records. The ability of the DOBR method to improve machine learning model overall predictive accuracy is being tested with these models.
  • DOBR farnesoid-outliers
  • DOBR defined outliers are records that contain variation not adequately described in the current variables (or features) given utilized machine learning model.
  • the analyst has three options.
  • the first option is to apply the base model, and do not apply DOBR. This is data-driven strategy when the Risk Acceptance vs. Model Error curve is close to a linear relationship.
  • the second is to apply one or more of the models: and combine e.g. average the results.
  • the third is to develop predictions for just the non-outlier records and further research the outlier data to develop a modeling strategy for this specialized new dataset - e.g. change the machine learning model or add variables to account for unexplained variation, etc.
  • the DOBR method was not initially designed to improve the prediction of the complete dataset. By design, the method converges to the best set of outliers based on the provided model and dataset. The remaining data and model calculations provide improved accuracy but there is no guidance on how to make predictions for the outliers. The implicit decision is to apply a different model to the outlier dataset that reflects the unique data variations that are not present in the non-outlier model.
  • two models are defined to test non-outlier prediction accuracy - removing the outliers from the analysis. The first choice for selecting the non-outlier dataset applies the DOBR classification vector, ⁇ I c ⁇ , to the reference model, ⁇ y ref ⁇ , according, e.g., model 6 below:
  • the reference model utilizes the complete training data-defmed model to make predictions from the dataset, x test .
  • the classification vector is then applied to remove predicted outliers based on the DOBR method’s knowledge obtained from the training dataset.
  • This model applies DOBR to the most general or broad domain model.
  • the second model applies DOBR in the most narrow or “precise” manner by using the DOBR model created from the training stage from non-outlier training data, to only the records selected by the classification model, ⁇ I c ⁇ , according, e.g., model 7 below:
  • ⁇ y ref ⁇ as a comparison basis for models ⁇ y 1 ⁇ , ⁇ y 2 ⁇ , and ⁇ y 3 ⁇ (Models 3, 4 and 5, respectively).
  • the comparison basis is ⁇ I c ⁇ * y test .
  • error can be determined according to, e.g., equations 26, 27 and 28, below:
  • the measure of DOBR’s predictive accuracy is gauged by how much, (if any), are less than
  • the measure of improvement is the decrease in error relative to the outlier adjusted base error The adjustment are described below with regards to the example results.
  • the accuracy of the previously defined five models may be tested with seven machine learning regression models: Linear Regression, k Nearest Neighbor, LASSO, Support Vector, Decision Tree, Bagging, and Random Forest. These machine learning regression models are examples of a broad spectrum of model constructs. Additional or alternative models are also contemplated, such as neural networks, clustering, ensemble models, among others and combinations thereof.
  • Linear regression is a method that gives analysts insights regarding the process where the coefficients (or the model parameters), can have a technical/process-related meaning.
  • a model of the process, represented by an equation, must be supplied by the analyst and the coefficients are determined by minimizing the error between the predicted and the data supplied target values.
  • LASSO an abbreviation for ‘least absolute shrinkage and selection operator,’ is a regression-related methodology where an addition term is added to the objective function. This term is the sum of the absolute values of the regression coefficients and it is minimized subject to a user supplied parameter. The purpose of this additional term is to add a penalty for increasing the value of variable (or feature) coefficients. The minimization only retains the dominant coefficients and can help in reducing the hard to interpret effects of variable (or feature) covariance or collinearity.
  • Decision Tree Regression can mimic human thinking and are intuitive and easy to interpret.
  • the model chooses a decision tree construct that logically shows how the x values produce the target variable.
  • the specific parameters like maximum depth and minimum samples per leaf are set by the analyst in the training/test machine learning exercise.
  • Random Forest Regression builds on the Decision Tree method. Just like forests are made with trees, a random forest regression model is made with groups of decision trees. The analyst defines the forest structure by supplying the of estimators (number of trees in the forest), some parameters similar to decision trees maximum depth of the trees, leaf characteristics, and technical parameters related to how the model error is computed and applied.
  • k-NN refers to k nearest neighbors methods where the predicted value is computed from the k nearest neighbors in the x (or feature) domain. Selecting the metric to measure distance and the specific number of nearest neighbors to use are the major parameters set by the analyst in tuning a model for predictions on a given dataset. It is a straightforward method that can work well for regression and classification predictions.
  • Support Vector Regression is a versatile, machine learning method that has several variations. Regression means fitting a model to data and the optimization is usually a minimization of error between predicted and the target variables. With support vector regression, the error criterion is generalized to say that if the error is less than some value ‘e’, then we say that’s ‘close is good enough’ and only errors greater than ‘s’, are measured and optimized. In addition to this attribute, the method allows the data to be transformed into nonlinear domains with standard or in some cases, user-defined transformation functions or kernels. The multi-dimensional data structure is used where the objective is to compute robust predictions - not to model the data in the tradition spirit of regression.
  • Bagging Regression computes prediction estimates from drawing random subsets with replacement. Each random sample computes a decision tree (by default) prediction of the target variable.
  • the final ensemble prediction value can be computed several ways - the average value is one example.
  • the primary machine learning variables are the number of estimators in each ensemble, the number of variables (or features) and samples to draw to train each estimator, and selection/replacement guidelines. The method can reduce the variance compared other methods like decision tree regression.
  • the classifier model is an illustrative example since it is applied to the DOBR non-outlier/outlier classifications and the training set x values to define non-outlier and outliers in the test dataset.
  • FIG. 12 is a graph illustrating an example of relationship between model error and an error acceptance criterion of some exemplary computer-based machine learning models with reduced bias for predicting concrete strength in accordance with one or more embodiments of the present disclosure.
  • the first example uses the same dataset as described above with reference to concrete compression strength, where DOBR is applied to a complete dataset.
  • this dataset contains concrete compression strength as a function of its composition and exposure as defined by 8 quantitative input variables.
  • the dataset has 1,030 records or instances and can be found at the University of California, Irvine machine learning repository archive.
  • the machine learning training exercise divides this dataset into a 70%:30% split with model tuning performed on the training dataset (70%) and the prediction results measured with the testing (30%) dataset.
  • Default model parameters are not shown since they do not add information to the results.
  • the tuning process is an exercise in selecting parameters that minimized training and testing dataset errors using the mean squared error as the indicator. More sophisticated algorithms could be applied but the straightforward approach was used simply to ensure the results did not over- or under- fit either dataset error.
  • a determination the percentage of data, if any, where the error is excessively large is performed.
  • the machine learning models are applied for a sequence of error acceptance fractions record the corresponding model errors. This is done solely for the training data set since the test data set is used only to measure the machine learning model’s prediction accuracy.
  • error acceptance refers to the amount of total model error the user is willing to acceptance and also indicates the fraction of data that the model adequately describes.
  • the error acceptance percentage sequence ranges from 100% to 60% in increments of 2.
  • FIG. 13 is a graph illustrating an example of relationship between model error and an error acceptance criterion of some exemplary computer-based machine learning models with reduced bias for predicting energy use in accordance with one or more embodiments of the present disclosure.
  • the second example contains appliance energy use data along with household environmental and lighting conditions with a sampling every 10 minutes for 41 ⁇ 2 months. It is comprised of 29 attributes: 28 input variables and 1 output (target variable) and 19,735 records: The dataset and documentation can be found at the University of California, Irvine machine learning repository archive.
  • a determination the percentage of data, if any, where the error is excessively large is performed.
  • the machine learning models are applied for a sequence of error acceptance fractions record the corresponding model errors. This is done solely for the training data set since the test data set is used only to measure the machine learning model’s prediction accuracy.
  • error acceptance refers to the amount of total model error the user is willing to acceptance and also indicates the fraction of data that the model adequately describes.
  • error acceptance percentage sequence ranges from 100% to 60% in increments of 2.
  • FIG. 12 and FIG. 13 show, in part, the machine learning models’ capability to adapt to high variation data.
  • the linear behavior for several models applied to the Concrete Data show they can almost completely adequately describe the entire training dataset.
  • the nonlinearity of the results for the energy dataset suggest that there is a significant percentage of data records where models produce inaccurate predictions or outliers.
  • the straight line defined by the low error acceptance percentages may extrapolated to determine the error acceptance value where the fraction of outliers begins, as per an embodiment of the present invention. This process could be automated but in practice, it may be performed by hand to ensure that the selected error acceptance value reflects the analyst’s judgment.
  • the extrapolation exercise and error acceptance percentage selection is a relatively simple process but it has very important implications. It indicates how good the proposed model fits the data.
  • the error acceptance value complement is the dataset percentage that are outliers, i.e., the percentage of records where the model fails to make relatively accurate predictions. This is important information in choosing the machine learning (or any model) for a given dataset and practical application.
  • Table 3 represents the error acceptance values chosen for each mode for the two example datasets.
  • the predictive accuracy of just the DOBR selected values are compared to the reference model.
  • This is the basic utility of DOBR since the method by itself does not provide any specific information about increasing the predictive accuracy for the complete dataset. Therefore, the DOBR analysis presents the analyst with a potential tradeoff: to have better predictive power for part of the dataset but with no information provided for outlier records.
  • the question addressed in this section is how much, if any, are the DOBR selected results more accurate compared to the corresponding reference model test data predictions.
  • the reference error is computed for the complete dataset.
  • the adjusted reference error values for comparison with the non-outlier datasets is computed by multiplying the complete reference error by the error acceptance value. For example, if the reference error is 10.0 and the error acceptance value is 80%, then the adjusted reference error is 10 x 80% or 8.0.
  • the interpretation utilizes the definition of “error acceptance.” If the non-outlier data is computed on 80% of the data for example, then 80% of the total error should still be remaining in the non- outlier data. This is the error acceptance definition.
  • the results show the mean ⁇ a 95% confidence interval from 100 random trial selections of training and test data subsets.
  • the Support Vector results were computed from fewer iterations (5 or 10) for manage computing time issues.
  • Table 4 shows that there is little, if any predictive improvement using the DOBR selected records. This result is not surprising and in fact, is expected based on the shape of the error acceptance vs. model error curve shown in Fig. 12.
  • Table 5 indicates there is considerable improvement in the DOBR selected predictions from the reference model values for both the Bagging and Random Forest classifiers, see FIG. 14A and FIG. 14B, respectively, below.
  • the DOBR model shows the most improvement suggesting the removal of outliers before model learning, coupled with the DOBR classification provides better results than just using the DOBR classification to the full (non-DOBR) model.
  • the difference in improvement results between models shows that model choice is important. Even though this decision is made by the analyst, it is interesting to compare the prediction accuracy by model. Model run time and several other factors are also important and this research is not designed or intended to suggest the viability of one model over another.
  • Table 6 shows the predictive accuracy results for train/test samplings of the concrete compression strength dataset with the Bagging Classifier.
  • the Random Forest Classifier was not applied to this dataset.
  • the table displays the root mean square error (see, Equation 15) at a 95% confidence level between the test data and each of the models for 100 random selections of the training and test datasets.
  • Bagging reference model has the largest reference error values but its DOBR augmented model results are generally in the same statistical ranges as the other models. Also for practical reasons the Support Vector model was run for only 10 iterations. This explains the increase in uncertainty across its model results.
  • FIG. 14A and FIG. 14B illustrate plots of non-outlier and outlier distributions in classifier models according to an exemplary embodiment of an inventive exemplary computer based system with a DOBR classifier in accordance with one or more embodiments of the present disclosure.
  • the error distributions for the two model representations can be analyzed: ⁇ y 4 ⁇ , the Random Forest classifier (FIG. 14A) and ⁇ y 5 ⁇ the Bagging classifier (FIG. 14B) of the outlier and non-outlier datasets.
  • the non-outlier errors should be smaller than the outlier errors by design but the exemplary inventive DOBR model and the classification process are constructed from the training data exclusively so the testing dataset can contain information not previously seen. Consequently, the model and classification calculations may not be precise and the extent of classification errors can be visualized in these plots. This work is performed for the Linear and Bagging Regression models as these two approaches have the largest and smallest improvement benefits, respectively, presented in Tables 5.
  • FIG. 15 illustrates plots of model error as a function of error acceptance values for an example use case of an exemplary embodiment of an inventive exemplary computer based system with a DOBR-trained machine learning model for predicting non-production time in well drilling in accordance with one or more embodiments of the present disclosure.
  • NPT non-productive time
  • the classification function that transfers the DOBR computed outlier information to the test dataset may be chosen as a random forest model with the number of estimators equal to, e.g., 5. This tuning activity is also accomplished in the training part of the analysis.
  • the metric for parameter selection is to compute the percentage of correctly classified elements of the training set and compare it to the model’s error acceptance value.
  • Data_xin_scrub and D ata_yin_scrub are the DOBR computed inliers from the training set
  • Data_xout_scrub and Data_yout_scrub are the DOBR computed outliers from the training set
  • DOBR_Model_testin and DOBR_Model_testout are the test dataset inliers and outliers, respectively, computed from the DOBR classification model
  • BaseModeljyin, BaseModeljyout are non-DOBR computed model results classified into inliers and outliers using the DOBR classification model, and / assigns BaseModel values to
  • image rendering and visualization may leverage machine learning models to automatically predict and implement rendering parameters based on, e.g., medical data, as described in U.S. Patent No. 10,339,695.
  • DOBR may be employed to train and implement machine learning models for content-based rendering.
  • a medical dataset representing a three- dimensional region of a patient may be employed as input data.
  • outliers from a training medical dataset may be removed such that the machine learning model may be trained on non-outlier data according to the DOBR techniques described above.
  • the machine-learned model is trained with deep learning of the non-outlier data from the training medical dataset to extract features from the medical dataset and to output values for two or more physically-based rendering parameters based on input of the medical dataset.
  • the physically-based rendering parameters are controls for consistent data handling, lighting design, viewing design, material propriety, or internal renderer property.
  • a physically-based renderer renders a photorealistic image of the three-dimensional region of the patient using the output values resulting from the applying.
  • a machine learning model may be trained with the DOBR techniques described above to generate a control command for a machine to output the control command, as described in U.S. Patent No. 10,317,854.
  • a simulator may perform a simulation of a work operation of the machine based on the control command.
  • the simulator may generate a complete data set for training the machine learning model by simulating physical actions of the machine based on the control command.
  • Such a dataset may be processed using the DOBR iterations to ensure any outlier simulations are removed when training the model parameters including the work operation data, control command data and machine data used as input for each simulation.
  • the exemplary DOBR machine learning model may be based at least in part on is Monte Carlo method of computational algorithms (e.g., the Solovay-Strassen type algorithms, the Baillie-PSW type algorithms, the Miller-Rabin type algorithms, and/or Schreier-Sims type algorithms) that may consider the historical quality data for the desired non-outlier data.
  • Monte Carlo method of computational algorithms e.g., the Solovay-Strassen type algorithms, the Baillie-PSW type algorithms, the Miller-Rabin type algorithms, and/or Schreier-Sims type algorithms
  • the exemplary DOBR machine learning model may be continuously trained by, for example without limitation, applying at least one machine learning technique (such as, but not limited to, decision trees, boosting, support-vector machines, neural networks, nearest neighbor algorithms, Naive Bayes, bagging, random forests, etc.) to the collected and/or compiled sensor data (e.g., various type of visual data about environmental and/or cargo’s physical/visual appearance).
  • an exemplary neutral network technique may be one of, without limitation, feedforward neural network, radial basis function network, recurrent neural network, convolutional network (e.g., U-net) or other suitable network.
  • an exemplary implementation of Neural Network may be executed as follows: i) Define Neural Network architecture/model, ii) Transfer the input data to the exemplary neural network model, iii) Train the exemplary model incrementally, iv) determine the accuracy for a specific number of timesteps, v) apply the exemplary trained model to process the newly -received input data, vi) optionally and in parallel, continue to train the exemplary trained model with a predetermined periodicity.
  • the exemplary trained neural network model may specify a neural network by at least a neural network topology, a series of activation functions, and connection weights.
  • the topology of a neural network may include a configuration of nodes of the neural network and connections between such nodes.
  • the exemplary trained neural network model may also be specified to include other parameters, including but not limited to, bias values/functions and/or aggregation functions.
  • an activation function of a node may be a step function, sine function, continuous or piecewise linear function, sigmoid function, hyperbolic tangent function, or other type of mathematical function that represents a threshold at which the node is activated.
  • the exemplary aggregation function may be a mathematical function that combines (e.g., sum, product, etc.) input signals to the node.
  • an output of the exemplary aggregation function may be used as input to the exemplary activation function.
  • the bias may be a constant value or function that may be used by the aggregation function and/or the activation function to make the node more or less likely to be activated.
  • an exemplary connection data for each connection in the exemplary neural network may include at least one of a node pair or a connection weight.
  • the exemplary connection data for that connection may include the node pair ⁇ N 1 , N2>.
  • the connection weight may be a numerical quantity that influences if and/or how the output of N1 is modified before being input at N2.
  • a node may have a connection to itself (e.g., the connection data may include the node pair ⁇ N1, N1>).
  • the exemplary trained neural network model may also include a species identifier (ID) and fitness data.
  • ID a species identifier
  • each species ID may indicate which of a plurality of species (e.g., cargo’s loss categories) the model is classified in.
  • the fitness data may indicate how well the exemplary trained neural network model models the input sensory data set.
  • the fitness data may include a fitness value that is determined based on evaluating the fitness function with respect to the model.
  • the exemplary fitness function may be an objective function that is based on a frequency and/or magnitude of errors produced by testing the exemplary trained neural network model on the input sensory data set.
  • testing the exemplary trained neural network model may include inputting each of the ten values of A from the input sensor data set, comparing the predicted values of B to the corresponding actual values of B from the input sensor data set, and determining if and/or by how much the two predicted and actual values of B differ.
  • the exemplary fitness function may be based on factors unrelated to error frequency or error rate, such as number of input nodes, node layers, hidden layers, connections, computational complexity, etc. [0234] In some embodiments and, optionally, in combination of any embodiment described above or below, the present disclosure may utilize several aspects of at least one of:
  • a method comprising: receiving, by at least one processor, a training data set of target variables representing at least one activity-related attribute for at least one user activity; receiving, by the at least one processor, at least one bias criteria used to determine one or more outliers; determining, by the at least one processor, a set of model parameters for a machine learning model comprising:
  • a system comprising: at least one processor in communication with a non-transitory computer-readable storage medium having software instructions stored thereon, wherein the software instructions, when executed, cause the at least one processor to perform steps to: receive a training data set of target variables representing at least one activity- related attribute for at least one user activity; receive at least one bias criteria used to determine one or more outliers; determine a set of model parameters for a machine learning model comprising:
  • Clause 3 The systems and methods of clauses 1 and/or 2, further comprising: applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set; training, by the at least one processor, using the outlier training data set, at least one outlier-specific model parameter of at least one outlier-specific machine learning model to predict outlier data values; and utilizing, by the at least one processor, the outlier-specific machine learning model to predict outlier activity-related data values for the set of outlier activity -related data.
  • Clause 4 The systems and methods of clauses 1 and/or 2, further comprising: training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the generalized machine learning model to predict outlier activity-related data values for the set of outlier activity -related data; and utilizing, by the at least one processor, the generalized machine learning model to predict the activity -related data values.
  • Clause 5 The systems and methods of clauses 1 and/or 2, further comprising: applying, by the at least one processor, the data selection vector to the training data set to determine an outlier training data set; training, by the at least one processor, using the outlier training data set, an outlier- specific model parameters of an outlier-specific machine learning model to predict outlier data values; training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the outlier-specific machine learning model to predict outlier activity-related data values for the set of outlier activity -related data; and utilizing, by the at least one processor, the outlier-specific machine learning model to predict the activity -related data values.
  • Clause 6 The systems and methods of clauses 1 and/or 2, further comprising: training, by the at least one processor, using the training data set, generalized model parameters of a generalized machine learning model to predict data values; utilizing, by the at least one processor, the generalized machine learning model to predict the activity -related data values for the set of activity-related data; utilizing, by the at least one processor, the outlier classifier machine learning model to identify outlier activity-related data values of the activity -related data values; and removing, by the at least one processor, the outlier activity-related data values.
  • Clause 7 The systems and methods of clauses 1 and/or 2, wherein the training data set comprises the at least one activity-related attribute of concrete compressive strength as a function of concrete composition and concrete curing exposure.
  • Clause 8 The systems and methods of clauses 1 and/or 2, wherein the training data set comprises the at least one activity-related attribute of energy use data as a function of household environmental conditions and lighting conditions.
  • Clause 9 The systems and methods of clauses 1 and/or 2, further comprising: receiving, by the at least one processor, an application programming interface (API) request to generate a prediction with at least one data element; and instantiating, by the at least one processor, at least one cloud computing resource to schedule execution of the machine learning model; utilizing, by the at least one processor according to the schedule for execution, the machine learning model to predict at least one activity-related data element value for the at least one data element; and returning, by the at least one processor, the at least one activity-related data element value to a computing device associated with the API request.
  • API application programming interface
  • Clause 10 The systems and methods of clauses 1 and/or 2, wherein the training data set comprises the at least one activity-related attribute of three-dimensional patient imagery of a medical dataset; and wherein the machine learning model is configured to predict the activity-related data values comprising two or more physically-based rendering parameters based on the medical dataset.
  • Clause 11 The systems and methods of clauses 1 and/or 2, wherein the training data set comprises the at least one activity-related attribute of simulated control results for electronic machine commands; and wherein the machine learning model is configured to predict the activity-related data values comprising control commands for the electronic machine.
  • Clause 12 The systems and methods of clauses 1 and/or 2, further comprising: splitting, by the at least one processor, the set of activity-related data into a plurality of subsets of activity-related data; determining, by the at least one processor, an ensemble model for each subset of activity-related data of the plurality of subsets of activity -related data; wherein the machine learning model comprises an ensemble of models; wherein each ensemble model comprises a random combination of models from the ensemble of models; utilizing, by the at least one processor, each ensemble model separately to predict ensemble-specific activity -related data values; determining, by the at least one processor, an error for each ensemble model based on the ensemble-specific activity-related data values and known values; and selecting, by the at least one processor, a highest performing ensemble model based on a lowest error.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computing Systems (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Algebra (AREA)
  • Operations Research (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Multimedia (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Feedback Control In General (AREA)

Abstract

La présente invention porte sur des systèmes et des procédés comprenant des processeurs servant à recevoir des données de formation concernant une activité d'utilisateur ; à recevoir des critères de biais ; à déterminer un ensemble de paramètres de modèle concernant un modèle d'apprentissage machine, consistant : (1) à appliquer le modèle d'apprentissage machine aux données d'apprentissage, (2) à générer des erreurs de prédiction de modèle, (3) à générer un vecteur de sélection de données de manière à identifier des variables cibles non aberrantes à partir des erreurs de prédiction de modèle, (4) à utiliser le vecteur de sélection de données de manière à générer un ensemble de données non aberrantes, (5) à déterminer des paramètres de modèle mis à jour à partir de l'ensemble de données non aberrantes, et (6) à répéter les étapes (1) à (5) jusqu'à ce qu'un critère de fin d'exécution de censuration soit satisfait ; à former des paramètres de modèle de classificateur concernant un modèle d'apprentissage machine de classificateur de valeurs aberrantes ; à appliquer le modèle d'apprentissage machine de classificateur de valeurs aberrantes à des données liées à l'activité de manière à déterminer des données non aberrantes liées à une activité ; et à appliquer le modèle d'apprentissage machine aux données non aberrantes liées à l'activité de manière à prédire des attributs futurs liés à l'activité concernant l'activité de l'utilisateur.
PCT/US2020/051627 2019-09-18 2020-09-18 Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine WO2021055847A1 (fr)

Priority Applications (8)

Application Number Priority Date Filing Date Title
CN202080072815.4A CN114556382A (zh) 2019-09-18 2020-09-18 被配置为在机器学习模型中实现动态离群值偏倚减少的基于计算机的系统、计算组件和计算对象
CA3154671A CA3154671A1 (fr) 2019-09-18 2020-09-18 Systemes informatiques, composants informatiques et objets informatiques configures pour mettre en ƒuvre une reduction dynamique de biais d'observations aberrantes dans des modele s d'apprentissage machine
GB2204238.6A GB2603358B (en) 2019-09-18 2020-09-18 Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
JP2022517214A JP7399269B2 (ja) 2019-09-18 2020-09-18 機械学習モデルにおいて動的外れ値偏り低減を実装するように構成されるコンピュータベースシステム、コンピュータコンポーネント及びコンピュータオブジェクト
BR112022005003A BR112022005003A2 (pt) 2019-09-18 2020-09-18 Sistemas baseados em computador, componentes de computação e objetos de computação configurados para implementar redução dinâmica de viés de valor atípico em modelos de aprendizado de máquina
EP20785887.9A EP4022532A1 (fr) 2019-09-18 2020-09-18 Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en oeuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine
KR1020227012917A KR20220066924A (ko) 2019-09-18 2020-09-18 머신 러닝 모델에서 동적 이상치 바이어스 감소를 구현하도록 구성된 컴퓨터 기반 시스템, 컴퓨팅 구성요소 및 컴퓨팅 객체
JP2023205241A JP2024026276A (ja) 2019-09-18 2023-12-05 機械学習モデルにおいて動的外れ値偏り低減を実装するように構成されるコンピュータベースシステム、コンピュータコンポーネント及びコンピュータオブジェクト

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962902074P 2019-09-18 2019-09-18
US62/902,074 2019-09-18

Publications (1)

Publication Number Publication Date
WO2021055847A1 true WO2021055847A1 (fr) 2021-03-25

Family

ID=72717923

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/US2020/051627 WO2021055847A1 (fr) 2019-09-18 2020-09-18 Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine
PCT/US2021/022861 WO2022060411A1 (fr) 2019-09-18 2021-03-18 Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/US2021/022861 WO2022060411A1 (fr) 2019-09-18 2021-03-18 Systèmes informatiques, composants informatiques et objets informatiques configurés pour mettre en œuvre une réduction dynamique de biais d'observations aberrantes dans des modèles d'apprentissage machine

Country Status (13)

Country Link
US (1) US11288602B2 (fr)
EP (2) EP4022532A1 (fr)
JP (2) JP7399269B2 (fr)
KR (2) KR20220066924A (fr)
CN (2) CN114556382A (fr)
AU (1) AU2021343372A1 (fr)
BR (1) BR112022005003A2 (fr)
CA (2) CA3154671A1 (fr)
DE (1) DE112021004908T5 (fr)
GB (4) GB2617045B (fr)
MX (1) MX2023003217A (fr)
NO (1) NO20230419A1 (fr)
WO (2) WO2021055847A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116384257A (zh) * 2023-05-29 2023-07-04 浙江大学 一种空分整装冷箱装配误差预测与公差优化方法
EP4239463A1 (fr) * 2022-03-02 2023-09-06 Ricoh Company, Ltd. Mises à jour de modèle d'estimation d'encre pour imprimantes de production
EP4246304A1 (fr) * 2022-03-02 2023-09-20 Ricoh Company, Ltd. Estimation d'encre parallèle en nuage pour imprimantes de production
US11803612B2 (en) 2018-09-28 2023-10-31 Hartford Steam Boiler Inspection And Insurance Company Systems and methods of dynamic outlier bias reduction in facility operating data
US11868425B2 (en) 2011-08-19 2024-01-09 Hartford Steam Boiler Inspection And Insurance Company Dynamic outlier bias reduction system and method

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102357659B1 (ko) 2014-04-11 2022-02-04 하트포드 스팀 보일러 인스펙션 앤드 인슈어런스 컴퍼니 시스템 조작 및 수행 데이터 모델링에 기초한 미래 신뢰도 예측 개선
US11615348B2 (en) 2019-09-18 2023-03-28 Hartford Steam Boiler Inspection And Insurance Company Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
US11328177B2 (en) * 2019-09-18 2022-05-10 Hartford Steam Boiler Inspection And Insurance Company Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
US11657323B2 (en) * 2020-03-10 2023-05-23 International Business Machines Corporation Machine learning model accuracy fairness
US11501239B2 (en) * 2020-03-18 2022-11-15 International Business Machines Corporation Metric specific machine learning model improvement through metric specific outlier removal
US20220189630A1 (en) * 2020-12-14 2022-06-16 Medtronic Minimed, Inc. Machine learning models for detecting outliers and erroneous sensor use conditions and correcting, blanking, or terminating glucose sensors
CN113176761B (zh) * 2021-04-28 2022-09-06 西安电子科技大学 多加工特征薄板零件质量预测与工艺参数优化方法
CN113408742B (zh) * 2021-06-24 2023-06-02 桂林理工大学 一种基于机器学习的高精度海表温度反演方法
CN113722813A (zh) * 2021-07-20 2021-11-30 中科云谷科技有限公司 用于泵车臂架状况监测的方法、装置及泵车
WO2023049128A1 (fr) * 2021-09-21 2023-03-30 Genentech, Inc. Routage de modèle et détection de valeur aberrante robuste
US20230116246A1 (en) * 2021-09-27 2023-04-13 Indian Institute Of Technology Delhi System and method for optimizing data transmission in a communication network
US20230229976A1 (en) * 2022-01-20 2023-07-20 Visa International Service Association System, method, and computer program product for time-based ensemble learning using supervised and unsupervised machine learning models
WO2023205445A1 (fr) * 2022-04-22 2023-10-26 Iqvia Inc. Apprentissage machine pour anonymisation de données
CN115460567B (zh) * 2022-11-09 2023-03-24 清华大学 数据处理方法、装置、计算机设备和存储介质
CN116625827B (zh) * 2023-06-17 2024-01-23 广州市盛通建设工程质量检测有限公司 含钢渣细集料的混凝土抗压测试方法、装置、设备及介质
CN116843080B (zh) * 2023-07-12 2024-03-19 济南明泉数字商务有限公司 一种基于机器学习的尿素生产碳元素足迹预测方法及系统
CN117302236B (zh) * 2023-09-27 2024-03-26 湖北天凯风林电子有限公司 一种基于深度学习的车辆状态监测方法及系统
CN117251816A (zh) * 2023-10-26 2023-12-19 南方电网能源发展研究院有限责任公司 碳排放数据的验证方法、装置、计算机设备和存储介质

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8195484B2 (en) 2005-06-15 2012-06-05 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
WO2013028532A1 (fr) * 2011-08-19 2013-02-28 Hsb Solomon Associates, Llc Système et procédé de réduction dynamique du biais d'observations aberrantes
US8812331B2 (en) 2012-04-27 2014-08-19 Richard B. Jones Insurance product, rating and credit enhancement system and method for insuring project savings
US9646262B2 (en) 2013-06-17 2017-05-09 Purepredictive, Inc. Data intelligence using machine learning
US10317854B2 (en) 2016-04-04 2019-06-11 Fanuc Corporation Machine learning device that performs learning using simulation result, machine system, manufacturing system, and machine learning method
US10339695B2 (en) 2016-03-10 2019-07-02 Siemens Healthcare Gmbh Content-based medical image rendering based on machine learning
US10452992B2 (en) 2014-06-30 2019-10-22 Amazon Technologies, Inc. Interactive interfaces for machine learning model evaluations

Family Cites Families (114)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS58160241U (ja) 1982-04-21 1983-10-25 呉羽化学工業株式会社 ｂ型バツフル
JPS59180731U (ja) 1983-05-23 1984-12-03 バブコツク日立株式会社 板状触媒水洗装置
JPH0336071Y2 (fr) 1985-10-04 1991-07-31
JPH0227976Y2 (fr) 1986-03-19 1990-07-27
JPH0313329Y2 (fr) 1987-04-27 1991-03-27
US5339392A (en) 1989-07-27 1994-08-16 Risberg Jeffrey S Apparatus and method for creation of a user definable video displayed document showing changes in real time data
CA2231652C (fr) 1995-10-12 2001-07-31 Yazaki Corporation Dispositif pour calculer une mauvaise repartition d'une charge supportee par un vehicule, et dispositif pour calculer une charge supportee par un vehicule
US7010336B2 (en) 1997-08-14 2006-03-07 Sensys Medical, Inc. Measurement site dependent data preprocessing method for robust calibration and prediction
US6085216A (en) 1997-12-31 2000-07-04 Xerox Corporation Method and system for efficiently allocating resources for solving computationally hard problems
JP2001318745A (ja) 2000-05-11 2001-11-16 Sony Corp データ処理装置およびデータ処理方法、並びに記録媒体
US20040172401A1 (en) 2000-06-15 2004-09-02 Peace Terrence B. Significance testing and confidence interval construction based on user-specified distributions
US6847976B1 (en) 2000-06-15 2005-01-25 Terrence B. Peace Method and apparatus for significance testing and confidence interval construction based on user-specified distribution
US6832205B1 (en) 2000-06-30 2004-12-14 General Electric Company System and method for automatically predicting the timing and costs of service events in a life cycle of a product
US7124059B2 (en) 2000-10-17 2006-10-17 Accenture Global Services Gmbh Managing maintenance for an item of equipment
US6988092B1 (en) 2000-12-28 2006-01-17 Abb Research Ltd. Method for evaluation of energy utilities
US7043461B2 (en) 2001-01-19 2006-05-09 Genalytics, Inc. Process and system for developing a predictive model
US7039654B1 (en) 2002-09-12 2006-05-02 Asset Trust, Inc. Automated bot development system
US20030171879A1 (en) 2002-03-08 2003-09-11 Pittalwala Shabbir H. System and method to accomplish pipeline reliability
US7313550B2 (en) 2002-03-27 2007-12-25 Council Of Scientific & Industrial Research Performance of artificial neural network models in the presence of instrumental noise and measurement errors
US20070219741A1 (en) 2005-05-20 2007-09-20 Emilio Miguelanez Methods and apparatus for hybrid outlier detection
US20040122625A1 (en) 2002-08-07 2004-06-24 Nasser Loren A. Apparatus and method for predicting total ownership cost
JP4042492B2 (ja) 2002-08-07 2008-02-06 トヨタ自動車株式会社 エンジン制御パラメータの適合方法及び適合システム
JP4109527B2 (ja) * 2002-10-01 2008-07-02 新日本製鐵株式会社 制御モデルの学習方法、制御モデルの学習装置、コンピュータプログラム、及びコンピュータ読み取り可能な記憶媒体
US7634384B2 (en) 2003-03-18 2009-12-15 Fisher-Rosemount Systems, Inc. Asset optimization reporting in a process plant
US8478534B2 (en) 2003-06-11 2013-07-02 The Research Foundation For The State University Of New York Method for detecting discriminatory data patterns in multiple sets of data and diagnosing disease
DE10331207A1 (de) 2003-07-10 2005-01-27 Daimlerchrysler Ag Verfahren und Vorrichtung zur Vorhersage einer Ausfall-Häufigkeit
WO2005015476A2 (fr) 2003-08-07 2005-02-17 Hsb Solomon Associates, Llc Systeme et procede pour determiner des coefficients d'equivalence a utiliser en analyse de performance comparative d'installations industrielles
US20050125322A1 (en) 2003-11-21 2005-06-09 General Electric Company System, method and computer product to detect behavioral patterns related to the financial health of a business entity
US20050131794A1 (en) 2003-12-15 2005-06-16 Lifson Kalman A. Stock portfolio and method
EP1548623A1 (fr) 2003-12-23 2005-06-29 Sap Ag Correction de déviations
JP4728968B2 (ja) 2004-02-06 2011-07-20 テスト アドバンテージ， インコーポレイテッド データ解析の方法および装置
JP4719694B2 (ja) 2004-02-13 2011-07-06 ウオーターズ・テクノロジーズ・コーポレイシヨン 化学物質を追跡し、定量化するためのシステムおよび方法
US7469228B2 (en) 2004-02-20 2008-12-23 General Electric Company Systems and methods for efficient frontier supplementation in multi-objective portfolio analysis
CA2501003C (fr) 2004-04-23 2009-05-19 F. Hoffmann-La Roche Ag Analyse d'echantillons pour obtenir des donnees de caracterisation
WO2005122881A1 (fr) 2004-06-21 2005-12-29 Aorora Technologies Pty Ltd Systeme de controle cardiaque
DE102004032822A1 (de) 2004-07-06 2006-03-23 Micro-Epsilon Messtechnik Gmbh & Co Kg Verfahren zur Verarbeitung von Messwerten
US20060069667A1 (en) 2004-09-30 2006-03-30 Microsoft Corporation Content evaluation
US7536364B2 (en) 2005-04-28 2009-05-19 General Electric Company Method and system for performing model-based multi-objective asset optimization and decision-making
US20060247798A1 (en) 2005-04-28 2006-11-02 Subbu Rajesh V Method and system for performing multi-objective predictive modeling, monitoring, and update for an asset
US7966150B2 (en) 2005-11-17 2011-06-21 Florida Power & Light Company Data analysis applications
US8290753B2 (en) 2006-01-24 2012-10-16 Vextec Corporation Materials-based failure analysis in design of electronic devices, and prediction of operating life
EP2013844A4 (fr) 2006-04-07 2010-07-07 Hsb Solomon Associates Llc Produit et procédé d'échange de droits d'émissions
US7447611B2 (en) 2006-05-09 2008-11-04 Hsb Solomon Associates, Llc Power generation performance analysis system and method
US20100262442A1 (en) 2006-07-20 2010-10-14 Standard Aero, Inc. System and method of projecting aircraft maintenance costs
EP2061374A4 (fr) 2006-08-31 2011-11-02 Nonlinear Medicine Inc Système automatisé de réduction du bruit permettant de prévoir des morts arythmiques
WO2008033439A2 (fr) * 2006-09-13 2008-03-20 Aurilab, Llc Systeme de reconnaissance de formes robustes et procede mettant en œuvre des agents socratiques
US20080104624A1 (en) 2006-11-01 2008-05-01 Motorola, Inc. Method and system for selection and scheduling of content outliers
JP5116307B2 (ja) 2007-01-04 2013-01-09 ルネサスエレクトロニクス株式会社 集積回路装置異常検出装置、方法およびプログラム
US8346691B1 (en) 2007-02-20 2013-01-01 Sas Institute Inc. Computer-implemented semi-supervised learning systems and methods
WO2008126209A1 (fr) 2007-03-27 2008-10-23 Fujitsu Limited Procédé, dispositif et programme pour réaliser un modèle de prédiction par analyse à régression multiple
US20080300888A1 (en) 2007-05-30 2008-12-04 General Electric Company Systems and Methods for Providing Risk Methodologies for Performing Supplier Design for Reliability
US8054177B2 (en) 2007-12-04 2011-11-08 Avaya Inc. Systems and methods for facilitating a first response mission at an incident scene using patient monitoring
JP5003566B2 (ja) 2008-04-01 2012-08-15 三菱電機株式会社 ネットワーク性能予測システム、ネットワーク性能予測方法およびプログラム
JP4991627B2 (ja) 2008-05-16 2012-08-01 株式会社日立製作所 計画執行管理装置およびそのプログラム
US8386412B2 (en) 2008-12-12 2013-02-26 At&T Intellectual Property I, L.P. Methods and apparatus to construct histogram and wavelet synopses for probabilistic data
US8509990B2 (en) 2008-12-15 2013-08-13 Panasonic Avionics Corporation System and method for performing real-time data analysis
US9111212B2 (en) 2011-08-19 2015-08-18 Hartford Steam Boiler Inspection And Insurance Company Dynamic outlier bias reduction system and method
JP5297855B2 (ja) 2009-03-26 2013-09-25 Ｄｏｗａメタルテック株式会社 銅合金板材およびその製造方法
JP2010250674A (ja) 2009-04-17 2010-11-04 Nec Corp 作業時間予測装置、方法、およびプログラム
US10739741B2 (en) 2009-06-22 2020-08-11 Johnson Controls Technology Company Systems and methods for detecting changes in energy usage in a building
US8311772B2 (en) 2009-12-21 2012-11-13 Teradata Us, Inc. Outlier processing
KR101733393B1 (ko) 2009-12-31 2017-05-10 에이비비 리써치 리미티드 전력 플랜트의 부하 스케줄링을 위한 방법 및 제어 시스템
JP5581965B2 (ja) 2010-01-19 2014-09-03 オムロン株式会社 Ｍｐｐｔ制御器、太陽電池制御装置、太陽光発電システム、ｍｐｐｔ制御プログラム、およびｍｐｐｔ制御器の制御方法
US20110246409A1 (en) 2010-04-05 2011-10-06 Indian Statistical Institute Data set dimensionality reduction processes and machines
CN102081765A (zh) 2011-01-19 2011-06-01 西安交通大学 输电设备状态检修的系统性控制方法
JP5592813B2 (ja) 2011-01-28 2014-09-17 株式会社日立ソリューションズ東日本 生涯需要予測方法、プログラムおよび生涯需要予測装置
US10557840B2 (en) 2011-08-19 2020-02-11 Hartford Steam Boiler Inspection And Insurance Company System and method for performing industrial processes across facilities
WO2013082724A1 (fr) 2011-12-08 2013-06-13 Copperleaf Technologies Inc. Appareil, systèmes et procédés de planification d'investissements sur des immobilisations
US9158303B2 (en) 2012-03-27 2015-10-13 General Electric Company Systems and methods for improved reliability operations
US8686364B1 (en) 2012-09-17 2014-04-01 Jp3 Measurement, Llc Method and system for determining energy content and detecting contaminants in a fluid stream
EP2770442A3 (fr) 2013-02-20 2014-09-17 Hartford Steam Boiler Inspection and Insurance Company Système et procédé de réduction dynamique du biais provenant de valeurs aberrantes
US9536364B2 (en) 2013-02-25 2017-01-03 GM Global Technology Operations LLC Vehicle integration of BLE nodes to enable passive entry and passive start features
KR102357659B1 (ko) 2014-04-11 2022-02-04 하트포드 스팀 보일러 인스펙션 앤드 인슈어런스 컴퍼니 시스템 조작 및 수행 데이터 모델링에 기초한 미래 신뢰도 예측 개선
US9568519B2 (en) 2014-05-15 2017-02-14 International Business Machines Corporation Building energy consumption forecasting procedure using ambient temperature, enthalpy, bias corrected weather forecast and outlier corrected sensor data
US9489630B2 (en) 2014-05-23 2016-11-08 DataRobot, Inc. Systems and techniques for predictive data analytics
US20190050510A1 (en) 2017-08-10 2019-02-14 Clearag, Inc. Development of complex agricultural simulation models from limited datasets
US9996933B2 (en) 2015-12-22 2018-06-12 Qualcomm Incorporated Methods and apparatus for outlier detection and correction of structured light depth maps
US10198339B2 (en) 2016-05-16 2019-02-05 Oracle International Corporation Correlation-based analytic for time-series data
US20190213446A1 (en) 2016-06-30 2019-07-11 Intel Corporation Device-based anomaly detection using random forest models
US11429859B2 (en) 2016-08-15 2022-08-30 Cangrade, Inc. Systems and processes for bias removal in a predictive performance model
WO2018075945A1 (fr) 2016-10-20 2018-04-26 Consolidated Research, Inc. Système et procédé d'évaluation des performances de prestataires de services
US11315045B2 (en) 2016-12-29 2022-04-26 Intel Corporation Entropy-based weighting in random forest models
CN107391569B (zh) 2017-06-16 2020-09-15 阿里巴巴集团控股有限公司 数据类型的识别、模型训练、风险识别方法、装置及设备
US10638979B2 (en) 2017-07-10 2020-05-05 Glysens Incorporated Analyte sensor data evaluation and error reduction apparatus and methods
US10474667B2 (en) 2017-07-29 2019-11-12 Vmware, Inc Methods and systems to detect and correct outliers in a dataset stored in a data-storage device
JP6837949B2 (ja) 2017-09-08 2021-03-03 株式会社日立製作所 予測システム及び方法
US20190108561A1 (en) 2017-10-05 2019-04-11 Mindtree Ltd. Purchase Intent Determination And Real Time In-store Shopper Assistance
EP3483797A1 (fr) 2017-11-13 2019-05-15 Accenture Global Solutions Limited Apprentissage, validation et surveillance de l'intelligence artificielle et de modèles d'apprentissage machine
US10521654B2 (en) 2018-03-29 2019-12-31 Fmr Llc Recognition of handwritten characters in digital images using context-based machine learning
US20190313963A1 (en) * 2018-04-17 2019-10-17 VideaHealth, Inc. Dental Image Feature Detection
CN109299156A (zh) 2018-08-21 2019-02-01 平安科技（深圳）有限公司 电子装置、基于XGBoost的电力数据异常预测方法及存储介质
US11636292B2 (en) 2018-09-28 2023-04-25 Hartford Steam Boiler Inspection And Insurance Company Dynamic outlier bias reduction system and method
US20200160229A1 (en) 2018-11-15 2020-05-21 Adobe Inc. Creating User Experiences with Behavioral Information and Machine Learning
US11204847B2 (en) 2018-12-21 2021-12-21 Microsoft Technology Licensing, Llc Machine learning model monitoring
US11797550B2 (en) 2019-01-30 2023-10-24 Uptake Technologies, Inc. Data science platform
WO2020197601A1 (fr) 2019-03-26 2020-10-01 Hrl Laboratories, Llc Systèmes et procédés de prévision d'alertes utilisant un apprentissage d'ensemble hybride homme-machine programmable
US11593650B2 (en) 2019-03-27 2023-02-28 GE Precision Healthcare LLC Determining confident data samples for machine learning models on unseen data
US11210587B2 (en) 2019-04-23 2021-12-28 Sciencelogic, Inc. Distributed learning anomaly detector
US11354602B2 (en) 2019-06-04 2022-06-07 Bank Of America Corporation System and methods to mitigate poisoning attacks within machine learning systems
US20200387836A1 (en) 2019-06-04 2020-12-10 Accenture Global Solutions Limited Machine learning model surety
US20200402665A1 (en) 2019-06-19 2020-12-24 GE Precision Healthcare LLC Unplanned readmission prediction using an interactive augmented intelligent (iai) system
CN110378386A (zh) 2019-06-20 2019-10-25 平安科技（深圳）有限公司 基于有监督的无标记异常识别方法、装置及存储介质
US20220351024A1 (en) 2019-06-24 2022-11-03 Telefonaktiebolaget Lm Ericsson (Publ) Method for detecting uncommon input
US11954610B2 (en) 2019-08-09 2024-04-09 GE Precision Healthcare LLC Active surveillance and learning for machine learning model authoring and deployment
CN110458374A (zh) 2019-08-23 2019-11-15 山东浪潮通软信息科技有限公司 一种基于arima和svm的企业用电最大需量预测方法
CN110411957B (zh) 2019-08-28 2021-11-19 北京农业质量标准与检测技术研究中心 水果货架期及新鲜程度的无损快速预测方法及装置
CN110543618A (zh) 2019-09-05 2019-12-06 上海应用技术大学 基于概率密度函数估计的圆度不确定度评定方法
CA3222363A1 (fr) 2019-11-06 2021-05-14 Centurylink Intellectual Property Llc Attribution predictive de ressources dans un reseau de calcul informatise en peripherie de reseau
CN110909822B (zh) 2019-12-03 2022-11-11 中国科学院微小卫星创新研究院 一种基于改进的高斯过程回归模型的卫星异常检测方法
CN111080502B (zh) 2019-12-17 2023-09-08 清华苏州环境创新研究院 一种区域企业数据异常行为的大数据识别方法
CN111157698B (zh) 2019-12-24 2022-10-21 核工业北京地质研究院 一种利用发射率数据获取黑土土壤全钾含量的反演方法
CN111709447A (zh) 2020-05-14 2020-09-25 中国电力科学研究院有限公司 电网异常检测方法、装置、计算机设备和存储介质
US11007891B1 (en) 2020-10-01 2021-05-18 Electricfish Energy Inc. Fast electric vehicle charging and distributed grid resource adequacy management system
CN112257963B (zh) 2020-11-20 2023-08-29 北京轩宇信息技术有限公司 基于航天软件缺陷数据分布离群点的缺陷预测方法及装置

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8676610B2 (en) 2005-06-15 2014-03-18 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8548833B2 (en) 2005-06-15 2013-10-01 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8554588B2 (en) 2005-06-15 2013-10-08 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8554589B2 (en) 2005-06-15 2013-10-08 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8595036B2 (en) 2005-06-15 2013-11-26 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8195484B2 (en) 2005-06-15 2012-06-05 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
US8719059B2 (en) 2005-06-15 2014-05-06 Hartford Steam Boiler Inspection And Insurance Company Insurance product, rating system and method
WO2013028532A1 (fr) * 2011-08-19 2013-02-28 Hsb Solomon Associates, Llc Système et procédé de réduction dynamique du biais d'observations aberrantes
US8812331B2 (en) 2012-04-27 2014-08-19 Richard B. Jones Insurance product, rating and credit enhancement system and method for insuring project savings
US9646262B2 (en) 2013-06-17 2017-05-09 Purepredictive, Inc. Data intelligence using machine learning
US10452992B2 (en) 2014-06-30 2019-10-22 Amazon Technologies, Inc. Interactive interfaces for machine learning model evaluations
US10339695B2 (en) 2016-03-10 2019-07-02 Siemens Healthcare Gmbh Content-based medical image rendering based on machine learning
US10317854B2 (en) 2016-04-04 2019-06-11 Fanuc Corporation Machine learning device that performs learning using simulation result, machine system, manufacturing system, and machine learning method

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11868425B2 (en) 2011-08-19 2024-01-09 Hartford Steam Boiler Inspection And Insurance Company Dynamic outlier bias reduction system and method
US11803612B2 (en) 2018-09-28 2023-10-31 Hartford Steam Boiler Inspection And Insurance Company Systems and methods of dynamic outlier bias reduction in facility operating data
EP4239463A1 (fr) * 2022-03-02 2023-09-06 Ricoh Company, Ltd. Mises à jour de modèle d'estimation d'encre pour imprimantes de production
EP4246304A1 (fr) * 2022-03-02 2023-09-20 Ricoh Company, Ltd. Estimation d'encre parallèle en nuage pour imprimantes de production
CN116384257A (zh) * 2023-05-29 2023-07-04 浙江大学 一种空分整装冷箱装配误差预测与公差优化方法
CN116384257B (zh) * 2023-05-29 2023-09-29 浙江大学 一种空分整装冷箱装配误差预测与公差优化方法

Also Published As

Publication number Publication date
BR112022005003A2 (pt) 2022-09-06
JP7399269B2 (ja) 2023-12-15
CN116569189A (zh) 2023-08-08
US20210110313A1 (en) 2021-04-15
JP2022548654A (ja) 2022-11-21
US11288602B2 (en) 2022-03-29
GB202305640D0 (en) 2023-05-31
JP2024026276A (ja) 2024-02-28
KR20220066924A (ko) 2022-05-24
EP4214652A1 (fr) 2023-07-26
GB2617045A (en) 2023-09-27
GB2603358B (en) 2023-08-30
GB2614849A (en) 2023-07-19
AU2021343372A1 (en) 2023-05-04
NO20230419A1 (en) 2023-04-18
GB2603358A (en) 2022-08-03
KR20230070272A (ko) 2023-05-22
CN114556382A (zh) 2022-05-27
GB2617045B (en) 2024-05-01
DE112021004908T5 (de) 2023-07-06
WO2022060411A1 (fr) 2022-03-24
GB202204238D0 (en) 2022-05-11
MX2023003217A (es) 2023-06-22
CA3195894A1 (fr) 2022-03-24
CA3154671A1 (fr) 2021-03-25
EP4022532A1 (fr) 2022-07-06
GB202402945D0 (en) 2024-04-17

Similar Documents

Publication Publication Date Title
US11288602B2 (en) Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
US11914680B2 (en) Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
US11615348B2 (en) Computer-based systems, computing components and computing objects configured to implement dynamic outlier bias reduction in machine learning models
US20210326782A1 (en) Systems and techniques for predictive data analytics
Budennyy et al. Eco2ai: carbon emissions tracking of machine learning models as the first step towards sustainable ai
KR20230078764A (ko) 자동화된 피처 엔지니어링을 운영하는 시스템 및 방법
CA3189593A1 (fr) Apprentissage machine hybride
RU2813245C1 (ru) Компьютерные системы, вычислительные компоненты и вычислительные объекты, выполненные с возможностью реализации уменьшения обусловленного выбросовыми значениями динамического отклонения в моделях машинного обучения
JP2024504179A (ja) 人工知能推論モデルを軽量化する方法およびシステム
US11734157B2 (en) Systems with software engines configured for detection of high impact scenarios with machine learning-based simulation and methods of use thereof
CN116304607A (zh) 用于使用深度强化学习进行预测建模的自动特征工程

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20785887

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 3154671

Country of ref document: CA

ENP Entry into the national phase

Ref document number: 2022517214

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 202204238

Country of ref document: GB

Kind code of ref document: A

Free format text: PCT FILING DATE = 20200918

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112022005003

Country of ref document: BR

ENP Entry into the national phase

Ref document number: 2020785887

Country of ref document: EP

Effective date: 20220329

ENP Entry into the national phase

Ref document number: 20227012917

Country of ref document: KR

Kind code of ref document: A

REG Reference to national code

Ref country code: BR

Ref legal event code: B01E

Ref document number: 112022005003

Country of ref document: BR

Free format text: APRESENTE O COMPLEMENTO DO PEDIDO (RELATORIO OU DESENHOS, SE HOUVER), POIS O MESMO AINDA NAO FOI APRESENTADO.

ENP Entry into the national phase

Ref document number: 112022005003

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20220317

WWE Wipo information: entry into national phase

Ref document number: 522431988

Country of ref document: SA