WO2020221460A1

WO2020221460A1 - Method for training an artificial intelligence model and computer-implemented method for automatically determining a product quality

Info

Publication number: WO2020221460A1
Application number: PCT/EP2019/061296
Authority: WO
Inventors: Prof. Dr. Heiko BEIER; Dr. Anna-Kristine WIPPER
Original assignee: Moresophy Gmbh; Pricewaterhousecoopers Legal Ag
Priority date: 2019-05-02
Filing date: 2019-05-02
Publication date: 2020-11-05

Abstract

A method (1000, 1000') for training an artificial intelligence model (40) on a data processing system (100) includes providing several training data (10, 10a, 10b, 20, 20a, 20b) referring to a respective product of a product class, the training data (10, 10a, 10b, 20, 20a, 20b) comprising primary training data (10, 10a, 10b) referring to a respective primary product of a first product group, and secondary training data (20, 20a, 20b) referring to a respective secondary product of a second product group. A pre-processing module (110) running on the data processing system (100) is used to determine from the training data (10, 10a, 10b, 20, 20a, 20b) respective training information (10i, 20i) including a chemical property of the respective product and/or a physical property of the respective product. The pre-processing module includes a quantitative module (111) and at least one of a linguistic module (113) referring to the product class and a semantic module (112) referring to the product class. The artificial intelligence model (40) is trained to determine from the training information (10i, 20i) a quality parameter (p) referring to a product group allocation of the respective product.

Description

METHOD FOR TRAINING AN ARTIFICIAL INTELLIGENCE MODEL AND COMPUTER-IMPLEMENTED METHOD FOR AUTOMATICALLY DETERMINING A PRODUCT QUALITY

TECHNICAL FIELD

[001] Embodiments of the present invention relate to a method for training an artificial intelligence model on a data processing system, in particular a method for training an artificial intelligence model with product related training data, and a method for automatically determining a product quality.

BACKGROUND

[002] Artificial intelligence (AI) is generally known as the intelligence exhibited by machines and/or software. Recently, AI techniques have experienced a revival which is apart from improved understanding mainly due to more suitable hardware architectures such as modem graphics processing units (GPUs), general advances in computational power and the availability of and desire to analyze large amounts of data. Besides notable breakthroughs in playing strategic games such as Chess or Go (AlphaZero), AI systems are increasingly applied in many technological fields including image recognition, machine control, driving autonomous vehicles, diagnosis, or predicting of real systems, for example in weather forecasting, but also for natural language processing and the like.

[003] Although AI systems may be divided into two types, namely in classifiers, e.g. for feature detection / image recognition, and controllers, controllers also have to classify prior to inferring an action. Therefore, classifying typically forms a central part of AI systems.

[004] Since AI systems such as artificial neural networks (ANNs) are comparatively complex, training of AI systems tends to be cumbersome and time consuming, respectively. The quality and reliability of AI systems generally depend on the chosen architecture and on the training or learning phase, typically to a large extent. This may be of particular importance in situations with comparatively sparse training data.

[005] Accordingly, there is a need to further improve training of Al-systems, in particular Al-classifiers. SUMMARY

[006] According to an embodiment of a method for training an artificial intelligence model on a data processing system, the method includes providing several training data referring to a respective product of a product class. The training data include primary training data referring to a respective primary product of a first product group, and secondary training data referring to a respective secondary product of a second product group. A pre-processing module running on the data processing system is used to determine from the training data respective training information including a chemical property of the respective product and/or a physical property of the respective product. The pre-processing module includes a quantitative module and at least one of a linguistic module referring to the product class and a semantic module referring to the product class. The artificial intelligence model is trained to determine from the training information a quality parameter referring to a product group allocation of the respective product.

[007] According to an embodiment of a method for automatically determining a product quality, the method includes initializing an instance of an artificial intelligence model on a data processing system. The artificial intelligence model refers to a product class. The product information refers to a further product of the product class, in particular a not-yet classified product. The product information includes a chemical property of the further product and/or a physical property of the further product. A quality parameter of the further product is determined using the product information as input of the instance of the artificial intelligence model. The artificial intelligence model is obtainable by providing several training data referring to respective products of the product class, using a pre-processing module which includes a quantitative module and at least one of a linguistic module referring to the product class and a semantic module referring to the product class to determine from the training data respective training information including a chemical property of the respective product and/or a physical property of the respective product, and training the artificial intelligence model to determine from the training information a quality parameter referring to a product group allocation of the respective product. The training data include primary training data referring to a respective primary product of a first product group of the product class, and secondary training data referring to a respective secondary product of a second product group of the product class. [008] Other embodiments include corresponding computer systems, in particular data processing system (also referred to as information processing system) connectable to network, computer-readable storage media or devices, and computer programs recorded on one or more computer-readable storage media or computer storage devices, each configured to perform the processes of the methods described herein.

[009] A system of and/or including one or more computers and/or processors can be configured to perform particular operations or processes by virtue of software, firmware, hardware, or any combination thereof installed on the one or more computers and/or processors that in operation may cause the a data processing system to perform the processes. One or more computer programs can be configured to perform particular operations or processes by virtue of including instructions that, when executed by a one or more computers and/or processors of the system, cause the system to perform the processes.

[0010] Those skilled in the art will recognize additional features and advantages upon reading the following detailed description, and upon viewing the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The components in the figures are not necessarily to scale, instead emphasis being placed upon illustrating the principles of the invention. Moreover, in the figures, like reference numerals designate corresponding parts. In the drawings:

[0012] Fig. 1A illustrates a flow chart of a method for training an artificial intelligence model on a data processing system according to an embodiment;

[0013] Fig. IB illustrates a flow chart of a method for automatically determining a product quality to an embodiment;

[0014] Fig. 2 is a block diagram schematically illustrating a data processing system and processes of a method for training an artificial intelligence model on the data processing system according to embodiments;

[0015] Fig. 3 is a block diagram schematically illustrating the data processing system shown in Fig. 2 and further processes of the method for training the artificial intelligence model on the data processing system according to embodiments; [0016] Fig. 4 is a block diagram schematically illustrating the data processing system shown in Fig. 2 and even further processes of the method for training the artificial intelligence model on the data processing system according to embodiments;

[0017] Fig. 5 is a block diagram schematically illustrating the data processing system shown in Fig. 2 and further processes of the method for training the artificial intelligence model on the data processing system according to embodiments; and

[0018] Fig. 6 is a block diagram schematically illustrating the data processing system shown in Fig. 2 and further processes of the method for training the artificial intelligence model on the data processing system and processes of a method for automatically determining a product property according to embodiments.

DETAILED DESCRIPTION

[0019] In the following Detailed Description, reference is made to the accompanying drawings, which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. In this regard, directional terminology, such as "top," "bottom," "front," "back," "leading," "trailing," etc., is used with reference to the orientation of the Figure(s) being described. Because components of embodiments can be positioned in a number of different orientations, the directional terminology is used for purposes of illustration and is in no way limiting. It is to be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0020] Fig. 1A illustrates a flow chart of a method 1000 for training an artificial intelligence model on a data processing system.

[0021] In a block 1100, several training data referring to a respective product of a given product class are provided. The (set of) training data (training data set) includes primary training data referring to a respective primary product of a first product group, and secondary training data referring to a respective secondary product of a second product group.

[0022] In a subsequent block 1200, a pre-processing module running on the data processing system is used to determine from the training data respective training information including a chemical property of the respective product and/or a physical property of the respective product. The pre-processing module has a quantitative module and at least one of a linguistic module referring to the product class and a semantic module referring to the product class.

[0023] In a subsequent block 1300, the artificial intelligence model is trained to determine from the training information a quality parameter referring to a product group allocation of the respective product.

[0024] As indicated by the dashed arrow in Fig. 1A, training the artificial intelligence model may include several, typically many training cycles.

[0025] Because the training data are pre-processed using linguistic and/or semantic methods implemented by the respective modules, the training of the artificial intelligence model (AIM) is much more efficient.

[0026] One main reason is that the artificial intelligence model (AIM) has to learn substantially less during the training phase when the training data are suitably pre-processed. In particular, the artificial intelligence model does not have to learn correlations which are “hidden” in the training data but revealed by the pre-processing. Due to the pre-processing, not only linguistic mistakes such as misspelling and the like may be corrected. Often more importantly, different terms used in the training data may be identified to refer to the same meaning.

[0027] For example, the training data may use different terms to describe the same chemical property of the product. More particular, different names (e.g. trade names) and/or symbols may be used for the same chemical compounds, and/or different terms and/or symbols may be used to describe the same physical property.

[0028] Likewise, the same term or the same symbol such as s for conductivity may be determined to have different meanings in the training data (thermal conductivity, electric conductivity).

[0029] Note that even physical terms with an apparently clear meaning like“mass” and “weight” or“power” and“force” are often mixed up and/or used differently depending on the context, in different fields and /or languages.

[0030] Therefore, the language of the respective data may be determined and a linguistic model may be selected for the linguistic module and/or a semantic model may be selected for the semantic module in accordance with the language. Likewise, the linguistic module and/or the semantic model may be selected in accordance with the product class.

[0031] Another example is the apparently clear term“metal” which is often used quite differently in astronomy and astrophysics compared to other fields like chemistry, namely as short term for "all elements except hydrogen and helium". From a chemical point of view, hydrogen is like helium a non-metal, but this also applies to the other noble gases and many other elements. On the other hand, hydrogen may have metallic properties at high enough pressure.

[0032] Training an AIM to learn such“hidden” information from training data requires a huge dataset of training data which is often not available. Even if, the training effort would be at least immense, more likely unrealistically high.

[0033] Different thereto, pre-processing based on linguistic analysis and semantic analysis can comparatively easy provide the required “understanding” to extract the “hidden” information / correlation(s) from the training data and provide suitably enriched training information for training AIMs.

[0034] Further, different units of chemical and/or physical properties may be normalized during the pre-processing by quantitative analysis and using the quantitative module, respectively.

[0035] Furthermore, properties of the training data which are determined during the pre processing such as the mentioned number or a rate of linguistic mistakes may be used as input of the AIM for determining the quality parameter as they may correlate with the trustworthiness of the training data and the source thereof, respectively.

[0036] Furthermore, based on the normalized data the process of anomaly detection can be used to improved or even optimized providing the secondary data. In particular, the anomalies detected can be used for effectively identifying secondary product data from an open network (WWW). For example, the detected anomaly(ies) may be used for determining and/or improving a search query used during searching the network for secondary training data (and later for searching for further product data). Thus, the process of providing secondary training data for later training of the AIM can be facilitated and/or even completely automated, at least after one, typically some secondary training data are found and pre-processed. [0037] All this may substantially facilitate training of artificial intelligence models significantly.

[0038] Likewise, classifying product data with the trained AIM and an instance thereof, respectively, may be substantially facilitated if the product data are also pre-processed in the same manner, i.e. using linguistic analysis and/or semantic analysis in combination with a typically subsequent quantitative analysis.

[0039] Classifying may be done on the data processing system used for training or on a different one initialized with an instance of the trained artificial intelligence model.

[0040] Improved training of AIMs as well as the use of accordingly trained AIMs may facilitate efficiency of AI systems in many fields.

[0041] For example, the training data (and likewise the product data to be classified) may include measurement data of chemical properties like concentrations of chemical elements and/or chemical compounds, physical properties like material characteristics, structural characteristics, amounts and/or dimensions, and/or process parameters like temperature or pressure referring to the product during manufacturing the product such as cars, medical devices or drugs. The training data set may include primary training data of primary products of a first product group which fulfill a quality criterion and secondary training data of secondary products of a second product group which do not fulfill the quality criterion. The training data may be pre-processed and the resulting training information may be used to teach an artificial intelligence model to properly classify the products into the two groups. Feeding the respective product information as input to the trained AIM, a quality parameter may be determined for later manufactured products of the same product class, e.g. a drug class referring to headache tablets, and may the products may be classified accordingly. For example, it may be automatically decided if the later manufactured products were manufactured in accordance with a desired quality standard such as good manufacturing practices (GMP) or not. In the latter case, the products may be withheld from sale, destroyed or recycled. If desired or necessary, the production process may even be influenced.

[0042] Monitoring manufacturing of products is however only one example, even if it is an important example.

[0043] Another example refers to classifying products offered on the internet. This example is explained in more detail below with regard to Fig. 2 o Fig. 6. [0044] For sake of clarity, classifying is mainly described as binary classification. The quality parameter (p) may however also refer to more than two product group allocations.

[0045] The training data as well as the product data to be classified using the trained AIM are typically retrieved (downloaded) via a network, in particular from the Internet.

[0046] The training data as well as the product data may be retrieved as/or from websites, html-pages, pdf-documents and the like. The data may also be retrieved as/or from brochures or package inserts referring to the product.

[0047] In other words, the training data (and likewise the product data of products to be classified) are typically downloaded from one or more servers in data connection with the respective data processing system.

[0048] Within this specification the terms“product to be classified” and“further product” are used synonymously. Likewise, the terms“data of product to be classified” and“further data” are used synonymously herein. Further, the phrase “information referring to a further product“is also referred to as“product information”.

[0049] The term "artificial intelligence model" (AIM) as used in this specification intends to describe a system which is operable in software and/or hardware, implements a mathematical model with a large number of parameters and is capable to be trained from examples (training data or information).

[0050] The AIMs as described herein are typically implemented as respective (artificial) neural networks.

[0051] The term "neural network" (NN) as used in this specification intends to describe an artificial neural network (ANN) including a plurality of connected units or nodes called artificial neurons. The output signal of an artificial neuron is calculated by a (non-linear) activation function of the sum of its inputs signal(s). The connections between the artificial neurons typically have respective weights (gain factors for the transferred output signal(s)) that are adjusted during one or more learning phases. Other parameters of the NN that may or may not be modified during learning may include parameters of the activation function of the artificial neurons such as a threshold. Typically, the artificial neurons are organized in layers. The most basic NN architecture, which is known as a“Multi-Layer Perceptron”, is a sequence of so called fully connected layers. A layer consists of multiple distinct units (neurons) each computing a linear combination of the input followed by a nonlinear activation function. Different layers (of neurons) may perform different kinds of transformations on their respective inputs. Neural networks may be implemented in software, firmware, hardware, or any combination thereof. In the learning phase(s), a machine learning method, in particular a supervised, unsupervised or semi- supervised (deep) learning method may be used. For example, a deep learning technique, in particular a gradient descent technique such as backpropagation may be used for training of (feedforward) NNs having a layered architecture. Modern computer hardware, e.g. GPUs makes backpropagation efficient for many-layered neural networks. A convolutional neural network (CNN) is a feed-forward artificial neural network that includes like most other NNs an input (neural network) layer, an output (neural network) layer, and one or more hidden (neural network) layers arranged between the input layer and the output layer. The speciality of CNNs is the usage of convolutional layers performing the mathematical operation of a convolution of the input with a kernel. The hidden layers of a CNN may include convolutional layers as well as optional pooling layers (for down sampling the output of a previous layer before inputting it to the next layer), fully connected layers and normalization layers. At least one of the hidden layers of a CNN is a convolutional neural network layer, in the following also referred to as convolutional layer. The usage of convolutional layer(s) can help to compute recurring features in the input more efficiently than fully connected layers. Accordingly, memory footprint may be reduced and performance improved. Due to the shared-weights architecture and translation invariance characteristics, CNNs are also known as shift invariant or space invariant artificial neural networks (SIANNs).

[0052] In a training phase, the NN (AIM) learns to interpret the input(s) automatically in order to output the correct result. Supervised learning algorithms, such as those employing neural networks (NN) rely on the existence of so-called labelled training data, i.e. some kind of input data in combination with expected output connected to that input data (primary and secondary training data of known classification).

[0053] Initially, the parameters (weights) of the NN may be randomly initialized. The goal of the training procedure is then to optimize the parameters such that when a training example is input, its corresponding expected output value is correctly output by the NN. The training procedure is done by feeding training examples into the network and for example summing the absolute deviations of the output predictions from the expected outputs, which yields a cost value or function. Numerical methods are used to minimize this cost in an iterative fashion which updates the parameters of the network model (backpropagation, gradient descent). A learning rate parameter is part of such methods in order to update the parameters. During the neural network training, the learning rate (learning rate decay) may be (incrementally) decreased, i.e. the step size of the gradient descent algorithm.

[0054] The training process can for example be stopped once a desired prediction accuracy based on a set of reference data (not used for training) is achieved or once the cost does not substantially decrease further. The final values of the parameters (weights) can then be digitally stored as a“trained model” and applied to a new input data example to generate a prediction. Depending on the amount of training data and the complexity of the model, training may be a process requiring several hours or days of calculation time, even when parallelized on many suitable units such as GPUs, while applying the trained NN for classifying input data may be quasi instantaneous.

[0055] As already explained above, training time and memory usage both during training and classifying may be substantially reduced, for example by a factor of at least two or even five, if not the training data and product data_s respectively, but semantically and/or linguistically pre-processed data (information) are used for training. The reduced memory usage is typically due to the lower requirements for the network architecture (less layers, less neurons) if enriched training information is used. Pre-processing typically combines rule-based and dictionary-based procedures for linguistic and semantic analysing (processing) of the training data and product data_s respectively, and by this achieves a high level of precision.

[0056] If a comparable high accuracy with a CNN but without pre-processing as described herein is to be achieved, a multiple of training data is required. Linguistic and semantic analysing during pre-processing may even mimic human language learning, which includes vocabulary, grammar and rules to be leamt, to be able to detect the meaning of a word or a certain phase in a sentence, not just be repetition of having seen this exact phrase a huge number of times, but actually knowing that this word or phrase must be constructed and derived from (a) specific base word(s).

[0057] In addition, the output of CNNs is lacking normalisation and semantic mapping and therefore hinders evaluating and/or assessing the data processing of the CNN, in particular for humans because of being flooded with very fuzzy data. Thus, linguistic and semantic analysing the data during pre-processing may facilitate presenting intermediate and/or final results. In particular, only a fraction, typically only a tiny fraction of results may be presented to humans, e.g. on a screen or printout, if pre-processing is used. Thus, humans may be enabled to get a (quick) overview.

[0058] The training information determined from the primary training data is in the following also referred to as primary training information.

[0059] Likewise, the training information determined from the secondary training data is in the following also referred to as secondary training information.

[0060] To facilitate training of different AIMs or a re-training of an AIM, the primary training information and the secondary training information may be stored in a respective database, i.e. in a primary product database and a secondary product database and/or in an aggregated product database for both the primary training information and the secondary training information.

[0061] The determined chemical property and/or the determined physical property typically represent an intrinsic property of the respective product and are therefore typically stored as intrinsic property in the respective database(s).

[0062] The intrinsic property may include several chemical properties and/or several physical properties referring to a final product property and/or a product property during manufacturing and/or process parameter(s) during manufacturing of the product.

[0063] The chemical property may include, represent and/or be a chemical composition of the respective product, a concentration of a chemical compound of the respective product, an amount of the chemical compound of the respective product, a health hazard of the chemical compound, the chemical composition and/or the respective product, an environmental hazard of the chemical compound, the chemical composition and/or the respective product, an LD50 value of the chemical compound, the chemical composition and/or the respective product, a biological effectiveness of the chemical compound, the chemical composition and/or the respective product, a medical or pharmaceutical effectiveness or efficacy of the chemical compound, the chemical composition and/or the respective product, a medical intolerance of the chemical compound, the chemical composition and/or the respective product, and/or a medical compatibility of the chemical compound, the chemical composition and/or the respective product.

[0064] The physical property may include, represent and/or be a structure of the respective product, a geometric measure of the respective product, a size of the respective product, a physical hazard of the respective product, and/or a form of the respective product, e.g. a form of administration of the respective product.

[0065] Furthermore, the determined training information (and likewise the product information of a product to be classified / product information) may include information referring to an extrinsic property of the respective product, in particular information referring to a use of the respective product.

[0066] Note that beside physical and chemical product information, product use information may also facilitate training of AIMs and classifying. For example, the same use of products may be useful for classifying the products as similar even if there is a certain mismatch of physical and/or chemical product properties. Thus uses of the products may be determined and fed to the AIM if available.

[0067] For similar reasons, the determined training information (and likewise the product information of a product to be classified / product information) may include information regarding a manufacturer of the respective product, a vendor of the respective product, a brand name of the respective product, and/or further technical information referring to the respective data and/or the respective product.

[0068] The further technical information may include, an identifier, in particular a serial number, a batch number and/or a product number of the respective product, an IP-address or an DOI (Digital Object Identifier) of the data source, information regarding a provider of the respective product, information regarding a provider of the data, information regarding a certificate of the data, and/or a price of the respective product.

[0069] The quality parameter may be indicative for the allocation of the respective product to one of the first product group and the second product group (or yet a further product group).

[0070] During training the artificial intelligence model, the training information is typically used as input of the AIM, and the quality parameter may be used as output of the artificial intelligence model.

[0071] In particular, the AIM may be trained to allocate the training information to a corresponding one of the product groups.

[0072] Prior to performing the training, a mapping between the training information may be performed to further facilitate the training. [0073] In particular, terms used in the training data may be mapped or even adjusted/amended. The mapping may refer to product names, chemical property(ies) and/or physical property(ies) of the products.

[0074] Thereafter, the training information may be updated / stored as amended training information in the respective database(s).

[0075] Furthermore, features of the determined training information or the amended training information may be compared and subsequently stored as further amended training information in the respective database(s).

[0076] For example, a quality of the training information may be determined based on a number or a rate of linguistic mistakes in the corresponding training data.

[0077] Alternatively or in addition, normalized values and/or ratios may be determined from the training information and compared. For example, prices per unit or dose and/or price volatility may be determined and compared.

[0078] Alternatively or in addition, images embedded in the training data (or the training information) may be compared.

[0079] Based on the comparison, an anomaly may be detected in the training information.

[0080] Fig. IB illustrates a flow chart of a method 200 for automatically determining a product quality.

[0081] In a block 2100, an instance of a trained artificial intelligence model for a product class is initialized on a data processing system.

[0082] In a subsequent block 2200, (enriched) product information referring to a further product of the product class is provided. The product information include a chemical property of the further product and/or a physical property of the further product, and are derived from product data referring to the further product similar as explained above with regard to Fig. 1A for the secondary training data. In particular, the product information may be obtained by pre processing respective product data found during a search in a network such as the WWW.

[0083] In a subsequent block 2300, a quality parameter of the further product is determined using the product information as input of the instance of the trained artificial intelligence model. The quality parameter may be a classifier or a probability referring to a product group of the further product. [0084] Thereafter, the product information and optionally the quality parameter and/or the product data (of the further product) may be stored in a target database, typically if the quality parameter fulfils a given criterion.

[0085] Further, the product information, in particular the physical property of the further product, and/or the chemical property of the further product may be further processed and/or a further action may be triggered.

[0086] Further processing the product information and/or triggering the further action typically depends on the quality parameter.

[0087] Further processing and/or the further action may be triggered when the quality parameter fulfills a predetermined criterion, for example when the quality parameter is in a given range and/or above or below a respective given threshold.

[0088] For example, a warning message may be sent and/or displayed when the determined quality parameter of the further product is too low.

[0089] Further processing may include weighting the physical property and/or the chemical property in accordance with the quality parameter and/or any further technical information of the product information such as measurement accuracy of physical and/or chemical properties.

[0090] For example, the physical property and/or the chemical property may be weighed in accordance with (measurement) accuracy and/or trustworthiness of the training data. For example, values obtained of reference books may be taken into account with higher significance than corresponding values from websites. Further, a trustworthiness of the website may be taken into account. For example, values of a manufactures website or brochure may be attributed a higher significance than values give in a user forum.

[0091] Further processing may include filling the physical property of the further product and/or the chemical property of the further product into a template, for example a template of a spreadsheet or word processing program.

[0092] Alternatively or in addition, further data retrieved from the product information, the product data and/or the target database are filled into the template.

[0093] For example, a template of a word processing program may be filled to generate a document referring to the further product. [0094] The document may be a draft document for review, for example a draft legal document such as a draft certificate or a draft lawyer's letter. Accordingly, people can be relieved of tedious routine work.

[0095] Prior to classifying, a search query referring to further products may be determined and the search query may be used for searching a network for available data referring to the further product. This may also facilitate relieving of tedious routine work.

[0096] Note that providing the product information may substantially or even completely be automated. In particular anomalies detected in the training information (between the primary and secondary training data, and thus in the training data) may be taken into account for defining a search query (or modifying a search query previously used for finding the secondary training data).

[0097] In particular, a web crawler may be used to search webpages / websites for further product.

[0098] To preserve evidence, a snapshot of found data may be stored, typically in the target data base.

[0099] Typically, the search query is determined using data stored in the aggregated product database and/or obtained during pre-processing the (training) data. Defining the search query may be rule-based.

[00100] In the following further embodiments referring to training AIMs and determining a product quality using the trained AIMs are explained.

[00101] Fig. 2 is a block diagram schematically illustrating a data processing system 100 which is connected to a network and has via the network access to data 10 referring to products of a product class.

[00102] The data 10 may be provided by one or more server of the world wide web (WWW).

[00103] In the exemplary embodiment, the product class covers (medical) drugs. This is however only an example. Alternatively, the product class may cover other manufactured products, for example medical devices, electronic devices, consumer products or even clothes.

[00104] In a first step primary data 10, 10a, 10b describing the primary products of the product class may be identified in the WWW. [00105] The primary data 10 may describe chemical, physical and other properties of the primary products.

[00106] Further, the primary data 10 may include primary data 10a provided by the manufacture or vendor and other primary data 10a, for example product reviews, customer reviews, reports of users and the like.

[00107] The primary products typically belong to a first product group of the product class.

[00108] Typically, the primary products fulfil a quality criterion. Accordingly, the first product group may be a group of quality products.

[00109] Alternatively or in addition, the primary products may be protected by intellectual property rights, in particular patents, industrial design rights, trademarks and/or copyright. In other words, the first group may be a group of intellectual property rights protected products.

[00110] The content of the primary data 10 is downloaded to the data processing system 100 and pre-processed by a pre-processing module 110 running on the data processing system 100.

[00111] As illustrated in Fig. 2, the pre-processing module 110 typically includes a linguistic module 113 covering the product class, a semantic module 112 covering the product class and a quantitative module 111 for analysing values such as measuring values.

[00112] As already explained above, pre-processing is used to extract primary information lOi, in particular intrinsic properties of the primary product such as chemical and/or a physical properties of the primary product, but also extrinsic properties such as product use information from the primary data 10.

[00113] The primary information lOi and optionally the primary data 10 are typically stored in a primary product data base 15 hosted on the data processing system 100 or in a cloud.

[00114] As illustrated in Fig. 3, secondary data 20, 20a, 20b referring to secondary products of the product class may be downloaded to the data processing system 100 and pre- processed by the pre-processing module 110 similarly or even in the same way as the primary data 10, 10a, 10b. [00115] However, the secondary information 20i and optionally the secondary data 10 are typically stored in a secondary product data base 25 hosted on the data processing system 100 or in the cloud. This is because the secondary products belong to a second product group disjunctive to the first product group. For example, the second product group may be a group of low quality products, a group of non-compliant products, a group of imitation products and/or a group of (non-protected) infringer products.

[00116] As shown in Fig. 3, further technical information and/or vendor information and the like may be determined and also stored in the secondary product data base 25.

[00117] Further technical information and/or vendor information may also be determined of the primary products and stored in the primary product data base 15.

[00118] Prior to using the primary and secondary information lOi, 20i as respective training information during training an AIM (as primary and secondary training information lOi, 20i), a mapping may be performed between the primary information lOi and the secondary information 20i.

[00119] As illustrated in Fig. 4, a mapping module 120 running on the data processing system 100 may be used for this purpose.

[00120] Typically, the mapping module 120 maps individual product IDs of primary and secondary products by means of normalizing intrinsic and/or extrinsic properties.

[00121] For example, product IDs of primary and secondary products with different names but at least substantially equal intrinsic properties may be mapped.

[00122] In an exemplary embodiment, the product IDs of an illegally traded medical drug with a different name but exactly the same active ingredient may be mapped to the ID of a legal drug, at least when the quantitative analysis module 111 determines that concentrations of active ingredient are at least substantially equal or comparable and/or the semantic analysis module 112 determines the same medical use.

[00123] Note that the mapping module 120 may not require linguistic capabilities. Some semantic capabilities may however be used, e.g. for mapping the product IDs etc. For example, different expressions of chemical substances should be mapped like the common name alcohol, its systematic name ethanol, the corresponding chemical formula, a chemical structure descriptor, a CAS registry number or any other standardized identifier. [00124] The respective secondary information may be amended accordingly, and the primary information and the amended secondary information 20i’ may be stored in an aggregated product database 55 hosted on the data processing system 100 or in the cloud.

[00125] Storing the mapping of product IDs in the aggregated product database 55 may facilitate later evaluation and graphical representation of correlations.

[00126] Further, an AIM may be trained to determine the correct mappings of secondary products to respective primary products. Accordingly, the quality parameter to be determined for a secondary product may have more than one dimension, for example a vector. A first element of the quality parameter may refer to the product group allocation and a second element of the quality parameter may refer to the product ID of a primary product.

[00127] Again, the primary data and the secondary data may be additionally stored in the aggregated product database 55.

[00128] Alternatively or in addition, a feature comparison between the first information lOi and the second information 20i (dashed arrows) or the amended second information 20i’ may be carried out, in particular for situations with comparatively few available product data.

[00129] As illustrated in Fig. 5, a feature comparison module 130 running on the data processing system 100 may be used for this purpose.

[00130] In particular, normalized values and/or ratios may be determined from the respective information lOi, 20i, 20i’ and compared, such as a price pre daily medication dose, or a relative price volatility. Note that a comparatively low price pre daily medication and a comparatively larger price volatility (compared to the primary product) may be useful indicators of low quality products and imitation products.

[00131] Likewise, a quality/trustworthiness of the primary and secondary information and/or respective images in the product data may be determined and compared.

[00132] Similar as explained above for the mapping module 120, the feature comparison module typically performs a mapping of product IDs of primary and secondary products with different names but at least substantially equal intrinsic properties may be mapped.

[00133] However, the feature comparison module 130 is typically more powerful than the mapping module 120 and may perform a more powerful semantic analysis. This is because the feature comparison module assists to evaluate how and why secondary product data differs from the primary one. The feature comparison module 130 is typically used if only comparatively data are available and/or if the mapping module 120 does find less than an expected number of product ID mappings. However, an in-depth semantic analysis may already be performed by the semantic analysis module 112 for all secondary products and not required in the feature comparison module 130.

[00134] In particular, anomalies and/or significant differences of the secondary products with respect to the primary products may be detected by the feature comparison module 130 and used for the product ID mapping.

[00135] For example, the feature comparison module 130 may help to detect an unusual combination of (the semantically normalized names of) chemical substances in secondary product data with respect to the primary data.

[00136] In a similar way feature comparison module 130 can be used to detect anomalies on normalized data.

[00137] For example, feature comparison module 130 may be used to detect an unusual pricing of the secondary product (based on the normalized out of the quantitative analysis module 111, like e.g. a price per unit).

[00138] Alternatively or in addition, the output of the feature comparison module 130 can be used to automate the process of identifying secondary product data on an open network.

[00139] For example, the feature comparison module 130 can show the anomaly of characteristic misspellings of products or certain keywords that are used to make a productive more attractive regardless of its actual intrinsic and extrinsic properties.

[00140] As illustrated by the dashed-dotted arrows in Fig. 5, in particular any detected anomaly(ies) may be used to further specify a search query in a search module 125 of the data processing system 100. Determining the search query is typically rule based. The search query may be transferred to one or more search engines (e.g. web crawlers) to search for product data that are subsequently pre-processed by the pre-processing module 110. Eventually, the derived product information and optionally the product data (showing the anomalies) may be stored in the secondary data base 25 and or the aggregated data base 55. [00141] The aggregated product data base 55 may also be updated in accordance with the detected anomalies, in particular the secondary data 20”.

[00142] Fig. 6 illustrates a data processing system 100 as explained above with regard to Fig. 2 to Fig. 5. Further, processes of the method 1000’ for training an artificial intelligence model 40 on the data processing system 10 and processes of a method 2000’ for automatically determining a product property are indicated as flow chart in Fig. 6.

[00143] The methods 1000’ and 2000’ are typically similar to the methods 1000, 2000 explained above with regard to Fig. 1A, and Fig IB, respectively. Further, the exemplary server 100’ used as data processing system for classifying may be the same as the exemplary server 100 used as data processing system during training the AIM 40. However, the server 100’ may also be a different one, in particular a computational less or more powerful one than the server 100 used for training.

[00144] For training, primary and secondary training data of a product class may be selected from the aggregated product data base 55 or alternatively from the first and second product data bases 15, 25 (dashed arrows), and sent to an AIM (classification) training module 140 running on the server 100. Using training data lOi, 20i of the aggregated product data base 55 is however preferred as mapping of product IDs may be stored more conveniently in the aggregated product data base 55.

[00145] The classification training module builds an AIM 40. This may require many training cycles.

[00146] The trained AIM 40 may be used to classify further product of the product class based on product information 30i determined using product data referring to the respective further product. The product information 30i is typically also determined by pre processing of respective product data which may be downloaded from connected servers of the WWW. The product information 30i may be provided from the aggregated product data base 55 (dashed dotted arrow in Fig. 6) or from a further product database.

[00147] However, storing the product information 30i in the aggregated product data base 55 is preferred. Due to the thus enlarged aggregated product data base 55, later retraining of the trained AIM 40 or training of a new AIM with a different architecture may be facilitated. [00148] The product information 30i is fed to the trained AIM (instance) 40 of a classification module 150 to determine a quality parameter p for the respective further product.

[00149] Depending on the quality parameter p (classification) some of the further products are stored in a target product database 75.

[00150] Alternatively and / or in addition, a further action or further processing may be triggered.

[00151] In one example, the target product database 75 refers to products not complying with a standard such as GMP. In this example, further action may include stopping the manufacturing or amending the manufacturing, e.g. to address a detected anomaly.

[00152] In another example, the target product database 75 refers to counterfeit products of legally protected products. In this example, draft document(s) referring to the counterfeit product(s) may be automatically generated and provided for review of a lawyer.

[00153] The draft document(s) are typically generated using a template filled with respective data from the target database 75.

[00154] Further, the draft document(s) may refer to infringement of respective intellectual property rights and/or may be addressed to a manufacturer of the respective counterfeit product, a supplier of the respective counterfeit product, e. g. in both cases in the form of a warning letter, the customs authorities, a department of public prosecution or even a competent court, e. g. in form of an application for an injunction or a lawsuit.

[00155] According to an embodiment of a method for training an artificial intelligence model on a data processing system, the method includes providing several training data of products of a (common) product class. Each of the training data refers to a respective product of the product class. The training data includes both primary training data referring to a respective primary product of a first product group and secondary training data referring to a respective secondary product of a second product group disjunctive to the first product group. The training data are pre-processed using a linguistic analysis and/or a semantic analysis (typically both) in combination with a quantitative analysis to determine a corresponding training information including at least one intrinsic property of the respective product, in particular a chemical property of the respective product and/or a physical property of the respective product. The artificial intelligence model is trained to classify the training information, in particular to determine a parameter referring to a product group allocation of the respective product from the training information.

[00156] Typically, a pre-processing module including a quantitative module and at least one of, typically both of a linguistic module referring to the product class and a semantic module referring to the product class and running on the data processing system is used for pre-processing.

[00157] According to an embodiment of a data processing system for training an artificial intelligence model and / for determining a quality of products of a product class, the data processing system includes one or more processors, a pre-processing module which is executable by at least one of the one or more processors, and configured to determine from product data referring to respective products corresponding product information including a chemical property of the respective product and/or a physical property of the respective product, the pre-processing module includes a quantitative module and at least one of a linguistic module referring to the product class and a semantic module referring to the product class, and at least one of a training module which is executable by at least one of the one or more processors, and configured to train the artificial intelligence model to determine from training information, which are determined by the pre-processing module from product training data, a quality parameter referring to a known product group allocation of the respective product, and a classification module which is executable by at least one of the one or more processors, and configured to determine a quality parameter of a further product using a product information, which is determined by the pre-processing module from product data referring to the further product, as input of an instance of the trained artificial intelligence model.

[00158] Typically, the data processing system is configured to perform any of the processes of the methods described herein.

[00159] Although various exemplary embodiments of the invention have been disclosed, it will be apparent to those skilled in the art that various changes and modifications can be made which will achieve some of the advantages of the invention without departing from the spirit and scope of the invention. It will be obvious to those reasonably skilled in the art that other components performing the same functions may be suitably substituted. It should be mentioned that features explained with reference to a specific figure may be combined with features of other figures, even in those cases in which this has not explicitly been mentioned. Such modifications to the inventive concept are intended to be covered by the appended claims.

[00160] While processes may be depicted in the figures in a particular order, this should not be understood as requiring, if not stated otherwise, that such operations have to be performed in the particular order shown or in sequential order to achieve the desirable results. In certain circumstances, multitasking and/or parallel processing may be advantageous.

[00161] Spatially relative terms such as“under”,“below”,“lower”,“over”,“upper” and the like are used for ease of description to explain the positioning of one element relative to a second element. These terms are intended to encompass different orientations of the device in addition to different orientations than those depicted in the figures. Further, terms such as“first”,“second”, and the like, are also used to describe various elements, regions, sections, etc. and are also not intended to be limiting. Like terms refer to like elements throughout the description.

[00162] As used herein, the terms“having”,“containing”,“including”,“comprising” and the like are open ended terms that indicate the presence of stated elements or features, but do not preclude additional elements or features. The articles“a”,“an” and“the” are intended to include the plural as well as the singular, unless the context clearly indicates otherwise.

[00163] With the above range of variations and applications in mind, it should be understood that the present invention is not limited by the foregoing description, nor is it limited by the accompanying drawings. Instead, the present invention is limited only by the following claims and their legal equivalents.

Claims

1. A method (1000, 1000’) for training an artificial intelligence model (40) on a data processing system (100), the method comprising:

- providing several training data (10, 10a, 10b, 20, 20a, 20b) referring to a respective product of a product class, the training data (10, 10a, 10b, 20, 20a, 20b) comprising primary training data (10, 10a, 10b) referring to a respective primary product of a first product group, and secondary training data (20, 20a, 20b) referring to a respective secondary product of a second product group;

- using a pre-processing module (110) running on the data processing system (100) to determine from the training data (10, 10a, 10b, 20, 20a, 20b) respective training information (lOi, 20i) comprising a chemical property of the respective product and/or a physical property of the respective product, the pre-processing module (110) comprising a quantitative module (111) and at least one of a linguistic module (113) referring to the product class and a semantic module (112) referring to the product class; and

- training the artificial intelligence model (40) to determine from the training information (lOi, 20i) a quality parameter (p) referring to a product group allocation of the respective product.

2. The method of claim 1, further comprising at least one of:

- storing the training information (lOi) determined from the primary training data (10, 10a, 10b) in a primary product database (15) and/or an aggregated product database (50);

- storing the primary training data (10, 10a, 10b) in the primary product database (15) and/or the aggregated product database (55);

- storing the secondary training data (10, 10a, 10b) in a secondary product database (25) and/or the aggregated product database (55); and

- storing the training information (20i) determined from the secondary training data (20, 20a, 20b) in the secondary product database (25) and/or the aggregated product database (50).

3. The method of claim 1 or 2, wherein the training information comprises information referring to an extrinsic property of the respective product, in particular information referring to a use of the respective product.

4. The method of claim 2 or 3, wherein the chemical property of the respective product and/or the physical property of the respective product comprise a corresponding measured value, and/or wherein the chemical property of the respective product and/or the physical property of the respective product are stored as intrinsic property in the respective database.

5. The method of any preceding claim, wherein providing the training data comprises downloading data (10a, 10b, 20a, 20b) referring to the respective product from at least one server in data connection with the data processing system (100).

6. The method of any preceding claim, wherein the training information further comprises at least one of:

- information regarding a vendor of the respective product;

- a brand name of the respective product; and

- further technical information referring to the respective training data and/or the respective product.

7. The method of claim 6, wherein the further technical information comprises at least one of:

- an identifier, in particular an IP-address or an DOI;

- information regarding a provider of the respective product;

- information regarding a provider of the training data;

- information regarding a certificate of the training data; and

- a price of the respective product.

8. The method of any preceding claim, wherein the quality parameter (p) refers to manufacturing and/or a quality standard, wherein the quality parameter (p) is indicative for the allocation of the respective product to one of the first product group and the second product group, wherein training the artificial intelligence model (40) comprises using the training information (lOi, 20i) as input of the artificial intelligence model (40), and/or using the quality parameter (p) and/or a corresponding one of the first product group and the second product group or a representation thereof as output of the artificial intelligence model (40), and/or wherein the artificial intelligence model (40) is trained to allocate the training information (lOi, 20i) to a corresponding one of the first product group and the second product group.

9. The method of any preceding claim, prior to performing the training comprising at least one of:

- performing a mapping between one of the training information (lOi) determined from the primary training data (10, 10a, 10b) and one of the training information (20i) determined from the secondary training data (20, 20a, 20b);

- amending the one of the training information (20i) determined from the secondary training data in accordance with the mapping; and

- storing the one of the training information (20i) as amended training information (20i’) in the aggregated product database (55).

10. The method of claim 9, wherein the mapping refers to at least one of the name of the product, the chemical property of the respective products and/or the physical property of the respective products.

11. The method of any preceding claim, wherein the artificial intelligence model comprises an artificial neural network, in particular a CNN.

12. The method of any preceding claim, further comprising at least one of:

- performing a feature comparison between one of the training information (lOi) determined from the primary training data (10, 10a, 10b) and one of the training information (20i) determined from the secondary training data (20, 20a, 20b) or the respective amended training information (20i’); - amending the one of the training information (20i) determined from the secondary training data (20, 20a, 20b) or the respective amended training information (20i’) in accordance with the feature comparison; and

- storing the one of the training information (20i) determined from the secondary training data (20, 20a, 20b) or the respective amended training information (20i’) as further amended training information (20i”) in the aggregated product database (55).

13. The method of claim 12, wherein the performing the feature comparison comprises at least one of:

- comparing a quality of the training information (lOi, 20i, 20i’);

- determining normalized values and/or ratios from the training information (lOi, 20i, 20i’);

- comparing the normalized values and/or the ratios;

- comparing respective images; and

- detecting an anomaly in the training information (lOi, 20i, 20i’).

14. A computer-implemented method (2000, 2000’) for automatically determining a product quality, the method comprising:

- initializing an instance of an artificial intelligence model (40) on a data processing system (100), the artificial intelligence model (40) referring to a product class and being obtainable with the method of any preceding claim, in particular being trained in accordance with the method of any preceding claim;

- providing product information (30i) referring to a further product of the product class, the product information (30i) comprising a chemical property of the further product and/or a physical property of the further product; and

- determining a quality parameter (p) of the further product using the product

information (30i) as input of the instance of the artificial intelligence model (40).

15. The method of claim 14, wherein providing the product information (30i) comprises at least one of: - downloading product data (30) referring to the further product from at least one server to a data processing system (100) in data connection with the at least one server; and

- using a pre-processing module (110) running on the data processing system (100) to determine from the product data (30) the product information (30i), the pre processing module (110) comprising a quantitative module (111) and at least one of a linguistic module (113) referring to the product class and/or a semantic module (112) referring to the product class.

16. The method of claim 14 or 15, wherein the product information further comprises at least one of:

- information referring to an extrinsic property of the further product, in particular information referring to a use of the further product;

- information regarding a vendor of the further product;

- a brand name of the further product; and

- further technical information referring to the product data and/or the further product.

17. The method of claim 16, wherein the further technical information comprises at least one of:

- an identifier, in particular an IP-address or an DOI;

- information regarding a provider of the further product;

- information regarding a provider of the product data;

- information regarding a certificate of the product data; and

- a price of the further product.

18. The method of any of the claims 14 to 16, further comprising at least one of storing the product information (30i) and optionally the quality parameter (p) and/or the product data (30) in a target database (75).

19. The method of any preceding claim, wherein the chemical property comprises at least one of a chemical composition of the respective product, a concentration of a chemical compound of the respective product, an amount of the chemical compound of the respective product, a health hazard of the chemical compound, the chemical composition and/or the respective product, an environmental hazard of the chemical compound , the chemical composition and/or the respective product, an LD50 value of the chemical compound , the chemical composition and/or the respective product, a biological effectiveness of the chemical compound , the chemical composition and/or the respective product, a medical/pharmaceutical effectiveness/efficacy of the chemical compound, the chemical composition and/or the respective product, a medical intolerance of the chemical compound, the chemical composition and/or the respective product, and a medical compatibility of the chemical compound, the chemical composition and/or the respective product

20. The method of any preceding claim, wherein the physical property comprises at least one of a structure of the respective product, a geometric measure of the respective product, a size of the respective product, a physical hazard of the respective product, and a form of the respective product.

21. The method of any of the claims 14 to 20, wherein the quality parameter (p) is a classifier or probability referring to a product group of the further product, and/or wherein the quality parameter (p) refers to manufacturing and/or a quality standard.

22. The method of any of the claims 14 to 21, further comprising:

- further processing the product information (30i), in particular the physical property of the further product, and/or the chemical property of the further product.

23. The method of claim 22, wherein the further processing depends on the quality parameter (p), and/or wherein the further processing comprises weighting the physical property and/or the chemical property in accordance with the quality parameter (p) and/or the further technical information of the product information (30i).

24. The method of claim 22 or 23, wherein the further processing comprises filling a template with the physical property of the further product, the chemical property of the further product, and optionally further data retrieved from the target database (75).

25. The method of any of the claims 22 to 24, further comprising triggering the further processing and/or a further action when the quality parameter (p) fulfills a predetermined criterion.

26. The method of any of the claims 13 to 25, further comprising at least one of:

- determining a search query referring to secondary products of the second product group;

- determining a search query referring to the further product;

- using the search query for searching a network for available data referring to the properties of the further product, in particular webpages referring to the further product; and

- saving a snapshot of the available data.

27. The method of claim 26, wherein determining the respective search query comprises at least one of:

- using a detected anomaly to define and/or amend the search query referring to secondary products; and

- using data stored in the aggregated product database (55) to define and/or amend the search query referring to the further product.

28. The method of any preceding claim, wherein the pre-processing module (110) is use to extract an image from the respective data, and wherein the extracted image is used as further input during training the artificial intelligence model (40) and/or determining the quality parameter (p).

29. The method of any preceding claim, further comprising at least one of:

- determining a language of the respective data; and - selecting a linguistic model for the linguistic module (113) and/or a semantic model for the semantic module (112) in accordance with the language.

30. A computer program product and/or a computer-readable storage medium comprising instructions which, when executed by a one or more processors of a computer, cause the computer to carry out the steps of the method according to any one of the preceding claims.