EP3444759A1

EP3444759A1 - Synthetic rare class generation by preserving morphological identity

Info

Publication number: EP3444759A1
Application number: EP18188616.9A
Authority: EP
Inventors: Arijit UKIL; Soma Bandyopadhyay; Chetanya Puri; Rituraj Singh; Arpan Pal
Original assignee: Tata Consultancy Services Ltd
Current assignee: Tata Consultancy Services Ltd
Priority date: 2017-08-14
Filing date: 2018-08-13
Publication date: 2019-02-20
Anticipated expiration: 2038-08-13
Also published as: US20190050673A1; EP3444759B1; US11531830B2

Abstract

In many real-life applications, ample amount of examples from one class are present while examples from other classes are rare for training and learning purposes leading to class imbalance problem and misclassification. Methods and systems of the present disclosure facilitate generation of an extended synthetic rare class super dataset that is further pruned to obtain a synthetic rare class dataset by maximizing similarity and diversity in the synthetic rare class dataset while preserving morphological identity with labeled rare class training dataset. Oversampling methods used in the art result in cloning of datasets and do not provide the needed diversity. The methods of the present disclosure can be applied to classification of noisy phonocardiogram (PCG) signals among other applications.

Description

Priority Claim

The present application claims priority from: Indian Patent Application No. 201721028875, filed on 14 August, 2017 . The entire contents of the aforementioned application are incorporated herein by reference.

Technical Field

The embodiments herein generally relate to data classification and more particularly to systems and methods for synthetic rare class generation by preserving morphological identity for facilitating data classification.

Background

Data-driven computational method is a challenging task in a scenario wherein rare class examples are scarce. For instance, examples or training datasets of disease class is very less in number compared to examples or training datasets of normal class. Again, fraud credit card events available for a certain type of transaction is very less in number compared to normal transaction events. Existing supervised learning methods perform poorly when one of the class examples is rare in number.

SUMMARY

Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems.
In an aspect, there is provided a processor implemented method comprising: analyzing a labeled abundant training dataset and labeled rare class training dataset; generating an extended synthetic rare class super dataset based on the analysis using an extended oversampling method; and extracting a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset by maximizing similarity and diversity in the synthetic rare class dataset, wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset such that morphological identity of the synthetic rare class dataset is preserved with respect to the labeled rare class training dataset while maintaining diversity.
In another aspect, there is provided a system comprising: one or more data storage devices operatively coupled to the one or more processors and configured to store instructions configured for execution by the one or more processors to: analyze a labeled abundant training dataset and labeled rare class training dataset; generate an extended synthetic rare class super dataset based on the analysis using an extended oversampling method; and extract a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset by maximizing similarity and diversity in the synthetic rare class dataset, wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset such that morphological identity of the synthetic rare class dataset is preserved with respect to the labeled rare class training dataset while maintaining diversity.
In yet another aspect, there is provided a computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to: analyze a labeled abundant training dataset and labeled rare class training dataset; generate an extended synthetic rare class super dataset based on the analysis using an extended oversampling method; and extract a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset by maximizing similarity and diversity in the synthetic rare class dataset, wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset such that morphological identity of the synthetic rare class dataset is preserved with respect to the labeled rare class training dataset while maintaining diversity.
In an embodiment of the present disclosure, the one or more hardware processors are further configured to perform extended oversampling method based on a Markov chain model.
In an embodiment of the present disclosure, the labeled abundant training dataset comprises labeled non-anomalous examples and the rare class training dataset comprises labeled anomalous examples.
In an embodiment of the present disclosure, the one or more hardware processors are further configured to perform the step of extracting a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset by: determining a similarity function pertaining to the extended synthetic rare class super dataset to obtain an extended rare class similar dataset; generating a ranked extended rare class similar dataset by ranking elements of the extended rare class similar dataset based on a similarity index associated thereof and sorting in descending order; and determining a diversity function for the ranked extended rare class similar dataset to obtain the synthetic rare class dataset with elements that are top ranked in the ranked extended rare class similar dataset and satisfies the diversity function.
In an embodiment of the present disclosure, the diversity function is based on l-diversity.
In an embodiment of the present disclosure, the synthetic rare class dataset is independent of dimensionality and is signal space rare class dataset.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the embodiments of the present disclosure, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments herein will be better understood from the following detailed description with reference to the drawings, in which:

FIG.1 illustrates an exemplary block diagram of a system for synthetic rare class generation by preserving morphological identity, in accordance with an embodiment of the present disclosure;
FIG.2 is an exemplary flow diagram illustrating a computer implemented method for synthetic rare class generation by preserving morphological identity, in accordance with an embodiment of the present disclosure;
FIG.3 is an illustration of a Markov chain model for generating an extended synthetic rare class super dataset, in accordance with an embodiment of the present disclosure; and
FIG.4A and FIG.4B illustrate synthetically generated rare class dataset in accordance with an embodiment of the present disclosure and physiological real-life labeled rare class data respectively.

It should be appreciated by those skilled in the art that any block diagram herein represent conceptual views of illustrative systems embodying the principles of the present subject matter. Similarly, it will be appreciated that any flow charts, flow diagrams, state transition diagrams, pseudo code, and the like represent various processes which may be substantially represented in computer readable medium and so executed by a computing device or processor, whether or not such computing device or processor is explicitly shown.

DETAILED DESCRIPTION OF EMBODIMENTS

The embodiments herein and the various features and advantageous details thereof are explained more fully with reference to the nonlimiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. The examples used herein are intended merely to facilitate an understanding of ways in which the embodiments herein may be practiced and to further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be construed as limiting the scope of the embodiments herein.
The words "comprising," "having," "containing," and "including," and other forms thereof, are intended to be equivalent in meaning and be open ended in that an item or items following any one of these words is not meant to be an exhaustive listing of such item or items, or meant to be limited to only the listed item or items.
It must also be noted that as used herein and in the appended claims, the singular forms "a," "an," and "the" include plural references unless the context clearly dictates otherwise. Although any systems and methods similar or equivalent to those described herein can be used in the practice or testing of embodiments of the present disclosure, the preferred, systems and methods are now described.
Some embodiments of this disclosure, illustrating all its features, will now be discussed in detail. The disclosed embodiments are merely exemplary of the disclosure, which may be embodied in various forms.
Before setting forth the detailed explanation, it is noted that all of the discussion below, regardless of the particular implementation being described, is exemplary in nature, rather than limiting.
Clinical decision making in data-driven computational methods is a challenging task due to scarcity of negative examples. The main drawback of prior-art is that simple over-sampling of available rare class examples are performed to generate synthetic rare class, which does not ensure diversity in the generated examples. Also, prior-art does not consider preserving morphological identities between the available rare class examples and generated rare class examples, thereby ignoring balancing of performance of the learning method. Systems and methods of the present disclosure ensure diversity (by not merely cloning as in the prior art) in the generated rare class examples while preserving morphological identity to overcome the class imbalance issue of the prior art.
Referring now to the drawings, and more particularly to FIGS. 1 through 4B, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and method.
FIG.1 illustrates an exemplary block diagram of a system 100 for synthetic rare class generation by preserving morphological identity, in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more processors 104, communication interface device(s) or input/output (I/O) interface(s) 106, and one or more data storage devices or memory 102 operatively coupled to the one or more processors 104. The one or more processors 104 that are hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, graphics controllers, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) are configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like.
The I/O interface device(s) 106 can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite. In an embodiment, the I/O interface device(s) can include one or more ports for connecting a number of devices to one another or to another server.
The memory 102 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. In an embodiment, one or more modules (not shown) of the system 100 can be stored in the memory 102.
In an embodiment, the system 100 comprises one or more data storage devices or memory 102 operatively coupled to the one or more processors 104 and is configured to store instructions configured for execution of steps of the method 200 by the one or more processors 104.
FIG.2 is an exemplary flow diagram illustrating a computer implemented method for synthetic rare class generation by preserving morphological identity, in accordance with an embodiment of the present disclosure. The steps of the method 200 will now be explained in detail with reference to the components of the system 100 of FIG.1. Table 1 herein below provides notation description used in the present disclosure. Table 1:

Notation Description

labeled abundant training dataset

labeled rare class training dataset

extended synthetic rare class super dataset

synthetic rare class dataset
In accordance with an embodiment of the present disclosure, the one or more processors 104 are configured to analyze, at step 202, a labeled abundant training dataset (
) and labeled rare class training dataset(
). In the context of the present disclosure, the expression "abundant training dataset" refers to a dataset for a class that is available in large numbers. For instance in case of a transaction scenario, positive training dataset in the form of normal transaction events may be available in large numbers. On the contrary, negative training dataset in the form of fraud credit card events may be available in small numbers only and may be referred to as "rare class training dataset". Again in a clinical decision making scenario, disease class examples may be referred to as "rare class training dataset" while normal class examples may be referred to as "abundant training dataset". Accordingly, in an embodiment, the labeled abundant training dataset may refer to labeled non-anomalous examples and the rare class training dataset may refer to labeled anomalous examples. Although this is a generally observed scenario, it may be true otherwise.
In accordance with an embodiment of the present disclosure, the one or more processors 104 are configured to generate, at step 204, an extended synthetic rare class super dataset (
) based on the analysis using an extended oversampling method. As explained above, a critical problem in anomaly detection is the limited availability of labeled rare class training dataset, which in an embodiment may be labeled anomalous examples. Let Π, π be the cardinality of the labeled abundant training dataset (
) and the labeled rare class training dataset (
) respectively and Π>>π.
Let $X = (X_{+}, X_{-}),$
$X_{+} = {\{x_{i}^{+}\}}_{i = 1}^{Π}, X_{-} = {\{x_{i}^{-}\}}_{i = 1}^{π}$
where $x_{i}^{+}, x_{i}^{-} \in R^{d},$
where
represents training instances.
At step 204, a generation function
generates the extended synthetic rare class super dataset (
) represented as {
} $\overset{G}{\to} X_{- - -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π + +} .$
One example of generation function
is that of permutated data generation in Markov chain model, an extended oversampling method. Alternatively, any deterministic model with a known function may be employed. Given the labeled rare class training dataset (
), some predicted number of states and associated state transition probabilities, the extended synthetic rare class super dataset $X_{- - -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π + +}$
is generated, where length of $[length ({\{x_{i}^{- - -}\}}_{i = 1}^{π + +}), \forall_{i}] \leq median (X_{-}) \mp 3 σ (X_{-}) .$
Let the cardinality Π of the labeled abundant training dataset (
) be 500 and the cardinality π of the labeled rare class training dataset be $X_{-} = {\{x_{i}^{-}\}}_{i = 1}^{π = 20} .$
In accordance with the present disclosure, the extended synthetic rare class super dataset (
) of 10000 instances is firstly generated by the generation function
. From (
), the synthetic rare class dataset $X_{- -} = {\{x_{i}^{- -}\}}_{i = 1}^{π}$
having 500 examples are extracted. Here, in the exemplary embodiment, Π=500, π=20 and Π++ = 10000.
In physiological signal space, typically a noisy signal consists of four segments: clean segment, motion artifact, random noise and power line interference segment, which correspond to measurement, instrumentation and interference plane respectively. In an embodiment, a Markov model based synthetic signal generation is provided, wherein the Markov model provides a systematic and stochastic method to model the time varying signals. In the Markov model, a future state only depends upon current states, not on predecessor states. This assumption and property makes the Markov model best suited for the generation of noisy/anomalous synthetic data or the rare class training dataset.
A stochastic process {X_n } is called a Markov chain if for all times n ≥ 0 and all states i ₀ , i ₁, ..... j ∈ S. $P = (x_{n + 1} = j | x_{n} = i, x_{n - 1} = i_{n - 1}, ... ... ..., x_{0} = i_{0}) = P (x_{n + 1 = j} | x_{n} = i) = P_{ij}$
wherein P_ij denotes a probability of moving from one state to another state, subject to ∑P_ij = 1 and is known as one state Markov chain. FIG.3 is an illustration of a Markov chain model for generating an extended synthetic rare class super dataset, in accordance with an embodiment of the present disclosure. The illustrated Markov model consists of five states. The state transition matrix is formed from noisy signal instances which are extracted from publicly available sources [Sardouie et al., IEEE journal of biomedical and health informatics 2015; Fasshauer and Zhang, Numerical Algorithms 2007]. In accordance with the present disclosure, for computation of state transition probabilities, it is assumed that state# 2 (S2=random noisy segment) and state# 3 (S3= motion artifact) may contribute maximum to noisy signals. Hence, corresponding probabilities are taken higher than other states. Table 2 herein below represents state transition probabilities (P_ij ). Table 2:

S₀ S₁ S₂ S₃ S₄

S₁ 0.05 0.05 0.05 0.05 0.1

S₂ 0.4 0.5 0.3 0.4 0.3

S₃ 0.5 0.4 0.6 0.5 0.5

S₄ 0.05 0.05 0.05 0.05 0.1
In accordance with an embodiment, generating the extended synthetic rare class super dataset (
) using the Markov model may be represented as given below.
Input:

(i) labeled rare class training dataset (
)
(ii) labeled abundant training dataset (
)
(iii) prior knowledge as represented in Table 2 above and FIG.3
1. (a) Different transition states $S = {\{S_{i}\}}_{i = 1}^{n}$
2. (b) State transition probabilities P_ij ; ∑P_ij = 1

Output: extended synthetic rare class super dataset (
)
Method:

1. Let length of the abundant training dataset (
) be $L_{p} = \{l_{1}^{p}, l_{2}^{p}, \dots \dots \dots l_{Π}^{p}\} .$
2. Construct the Markov chain model (Refer FIG.3)
3. S ₀,S ₁: Randomly selected from
- S ₂ : Additive white noisy signal from Gaussian distribution
- S ₃ : Typical motion artifact samples [Fraser et al., IEEE transactions on Instrumentation and Measurement 2014; Fasshauer and Zhang, Numerical Algorithms 2007].
- S ₄ : Powerline Interference (60 Hz).
4. Construct $L_{r} = \{l_{1}^{r}, l_{2}^{r}, \dots \dots l_{π}^{r}\},$
where $l_{i}^{r},$
i = 1,2, ... , Π be the length the generated extended synthetic rare class super dataset such that ∑L_p = ∑L_r.
5. $X_{i}^{- - -} = {\{S_{0}, S_{k}\}}_{i},$
order in which S_k ; k ∈ [1,4] selected, based on State Transition probabilities {Refer Table 2}; E.g. $x_{i}^{- - -} = {\{S_{0}, S_{2}, S_{3}, S_{1}, S_{4}\}}_{i}$
where length $(x_{i}^{- - -}) = l_{i}^{r} .$

It may be noted that the method described herein above, imposes restriction on the signal length (step 4) and arbitrary length is not permitted as in the case of the art.
In accordance with an embodiment of the present disclosure, the one or more processors 104 are configured to extract, at step 206, a subset of the extended synthetic rare class super dataset (
) to obtain the synthetic rare class dataset (
) by maximizing similarity
and diversity
in the synthetic rare class dataset, wherein maximizing similarity ensures maximizing mutual information (explained hereinafter) and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset such that morphological identity of the synthetic rare class dataset is preserved with respect to the labeled rare class training dataset while maintaining diversity. In the context of the present disclosure, morphological identity refers to characteristics associated with the available rare class dataset that are preserved in the synthetic rare class dataset while preserving diversity.
Dissimilarity property
with
may be represented as given below. ${maximize}_{x_{i}^{+}, x_{i}^{-} \in R^{d}} (S (X_{-}), D (X_{-}), B (X_{+}))$
with the assumption that the generated synthetic rare class dataset (
) would be similar yet not redundant with the labeled rare class training dataset (
) but distinct from the labeled abundant training dataset (
). In accordance with the present disclosure, it is further assumed that the labeled abundant training dataset (
) and the labeled rare class training dataset (
) are independent and subsequently, similarity
among
and dissimilarity
between (
) and (
) are equivalent. For simplicity of explanation, it is assumed that satisfying similarity is practically sufficient to satisfying dissimilarity and accordingly, the condition may be omitted from further consideration. The problem addressed in the present disclosure is therefore to generate the synthetic rare class dataset $X_{- -} =$
${\{x_{i}^{- -}\}}_{i = 1}^{π}$
from the labeled rare class training dataset $X_{-} = {\{x_{i}^{-}\}}_{i = 1}^{π},$
where Π>>π. Let, the extended synthetic rare class super dataset $X_{- - -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π + +},$
Π++>> Π be the universe of the synthetic rare class dataset (
) generated. The problem is to find (
) from the universe (
) such that equation (1) is satisfied.
In an embodiment, the step of extracting a subset of the extended synthetic rare class super dataset (
) to obtain a synthetic rare class dataset (
) comprises firstly determining a similarity function pertaining to the extended synthetic rare class super dataset (
) to obtain an extended rare class similar dataset and then generating a ranked extended rare class similar dataset by ranking elements of the extended rare class similar dataset based on a similarity index associated thereof and sorting in descending order. In accordance with the present disclosure, one example of the similarity function may be constructed as: Find mutual information $I (x; y) = = \sum_{x \in X_{- - -}} \sum_{y \in X_{-}} {p (x, y) \log}_{2} \frac{p (x, y)}{p (x) p (y)}$
for each of
with each of
that spawns II + + × π in
(
;
).. There are π number of $I (x_{i}^{- - -}; X_{-})$
for each i. Then, find the cluster centroid $C_{i}^{- - -}$
that contains higher number of cluster elements when performing k-means (k=2) clustering on $I (x_{i}^{- - -}; X_{-}) .$
For example, let there be 20 number of labeled rare class training dataset (
), $π = 20 : X_{-} = {\{x_{i}^{-}\}}_{i = 1}^{π = 20} .$
The generation function
generates 10000 extended synthetic rare class super dataset, where: $X_{- - -} =$
${\{x_{i}^{- - -}\}}_{i = 1}^{π = 10000},$
Π + + × π = 200000. For each computed $I (x_{i}^{- - -}; X_{-}),$
∀i = {1,2, ..., π = 20} total 20 mutual information values for each of the generated: $X_{- - -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π = 10000}$
are 2-means clusterd. The cluster centroid $C_{i}^{- - -}$
that contains higher number of cluster elements of each $I (x_{i}^{- - -}; X_{-})$
set is marked as the similarity index of $X_{i}^{- - -}$
on
. $X_{i}^{- - -} S$
are ranked in descending order sorting of the similarity indices. Let, the sorted order of
be $X_{- - -}^{similar} .$
.
After the ranked extended rare class similar dataset is generated, a diversity function is determined for the ranked extended rare class similar dataset to obtain the synthetic rare class dataset with elements that are top ranked in the ranked extended rare class similar dataset and satisfies the diversity function. Accordingly, the diversity function is determined on $X_{- - -}^{similar}$
and $X_{- - -}^{similar + diversed}$
is identified that are both ranked high in $X_{- - -}^{similar}$
and significantly diverse. Thus, $X_{- - -}^{similar + diversed} \subseteq X^{similar}$
and $X_{- - -}^{similar + diversed} = X_{- -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π},$
the generated synthetic rare class dataset. One example of the diversity function may be constructed as:
Find $X_{- -} = {\{x_{i}^{- -}\}}_{i = 1}^{π}$
set from $X_{- - -}^{similar}$
which are l-diverse (Machanavajjhala; 2007) and
contains the top-ranked in $X_{- - -}^{similar}$
in each of the l-diverse groups. Definition l-diversity: A group is l-diverse if each of the l different group
's entropy ≥ log₂
groups from $X_{- - -}^{similar},$
where l = Π^1/4, $X_{- - -}^{similar + diversed} = X_{- - -} = {\{x_{i}^{- - -}\}}_{i = 1}^{π}$
is formed taking the top ranked corresponds $X_{- - -}^{similar}$
from each of the l groups such that total number of elements chosen is Π.
FIG.4A and FIG.4B illustrate synthetically generated rare class dataset in accordance with an embodiment of the present disclosure and physiological real-life labeled rare class data respectively. As illustrated, the method of the present disclosure facilitates close capture of real-life noisy physiological signals, which confirms the assumption $X_{real - life noise} \sim P_{x}^{synthetic - noise} .$
It is observed that morphological identities are closely preserved in the generated synthetic rare class dataset (FIG.4A) when compared with the real-life noisy physiological signals (FIG 4B).
Thus in accordance with the present disclosure, systems and methods of the present disclosure facilitate addressing the class imbalance problem in applications such as identifying noisy phonocardiogram (PCG) signals. The present disclosure deals with signal (time-series) space rare-class dataset as compared to prior art that deal with feature space rare-class dataset. The subset of the extended synthetic rare class super dataset is extracted without reducing the dimensionality (feature space) of the dataset, thereby making the step of obtaining the synthetic rare class dataset independent of the dimensionality properties of the datasets and non-parametric.
The written description describes the subject matter herein to enable any person skilled in the art to make and use the embodiments of the present disclosure. The scope of the subject matter embodiments defined here may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language.
The scope of the subject matter embodiments defined here may include other modifications that occur to those skilled in the art. Such other modifications are intended to be within the scope if they have similar elements that do not differ from the literal language of the claims or if they include equivalent elements with insubstantial differences from the literal language.
It is, however to be understood that the scope of the protection is extended to such a program and in addition to a computer-readable means having a message therein; such computer-readable storage means contain program-code means for implementation of one or more steps of the method, when the program runs on a server or mobile device or any suitable programmable device. The hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof. The device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination of hardware and software means, e.g. an ASIC and an FPGA, or at least one microprocessor and at least one memory with software modules located therein. Thus, the means can include both hardware means and software means. The method embodiments described herein could be implemented in hardware and software. The device may also include software means. Alternatively, the embodiments of the present disclosure may be implemented on different hardware devices, e.g. using a plurality of CPUs.
The embodiments herein can comprise hardware and software elements. The embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc. The functions performed by various modules comprising the system of the present disclosure and described herein may be implemented in other modules or combinations of other modules. For the purposes of this description, a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The various modules described herein may be implemented as software and/or hardware modules and may be stored in any type of non-transitory computer readable medium or other storage device. Some nonlimiting examples of non-transitory computer-readable media include CDs, DVDs, BLU-RAY, flash memory, and hard disk drives.
Further, although process steps, method steps, techniques or the like may be described in a sequential order, such processes, methods and techniques may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously.
The preceding description has been presented with reference to various embodiments. Persons having ordinary skill in the art and technology to which this application pertains will appreciate that alterations and changes in the described structures and methods of operation can be practiced without meaningfully departing from the principle, spirit and scope.

Claims

A processor implemented method (200) comprising:
analyzing a labeled abundant training dataset (
) and labeled rare class training dataset (
) (202);

generating an extended synthetic rare class super dataset (
) based on the analysis using an extended oversampling method (204); and

extracting a subset of the extended synthetic rare class super dataset (
) to obtain a synthetic rare class dataset (
) by maximizing similarity and diversity in the synthetic rare class dataset (
), wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset (
) such that morphological identity of the synthetic rare class dataset (
) is preserved with respect to the labeled rare class training dataset (
) while maintaining diversity (206).
The processor implemented method of claim 1, wherein the extended oversampling method is based on Markov chain model.
The processor implemented method of claim 1, wherein the labeled abundant training dataset comprises labeled non-anomalous examples and the rare class training dataset comprises labeled anomalous examples.
The processor implemented method of claim 1, wherein the step of extracting a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset comprises:
determining a similarity function pertaining to the extended rare class dataset to obtain an extended rare class similar dataset;

generating a ranked extended rare class similar dataset by ranking elements of the extended rare class similar dataset based on a similarity index associated thereof and sorting in descending order; and

determining a diversity function for the ranked extended rare class similar dataset to obtain the synthetic rare class dataset with elements that are top ranked in the ranked extended rare class similar dataset and satisfies the diversity function.
The processor implemented method of claim 4, wherein the diversity function is based on l-diversity.
The processor implemented method of claim 1, wherein the synthetic rare class dataset is independent of dimensionality and is signal space rare class dataset.
A system (100) comprising:
one or more data storage devices (102) operatively coupled to one or more hardware processors (104) and configured to store instructions configured for execution by the one or more hardware processors to:
analyze a labeled abundant training dataset (
) and labeled rare class training dataset (
);

generate an extended synthetic rare class super dataset (
) based on the analysis using an extended oversampling method; and

extract a subset of the extended synthetic rare class super dataset (
) to obtain a synthetic rare class dataset (
), by maximizing similarity and diversity in the synthetic rare class dataset (
), wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset (
) such that morphological identity of the synthetic rare class dataset (
) is preserved with respect to the labeled rare class training dataset (
) while maintaining diversity.
The system of claim 7, wherein the one or more hardware processors are further configured to perform extended oversampling method based on Markov chain model.
The system of claim 7, wherein the labeled abundant training dataset comprises labeled non-anomalous examples and the rare class training dataset comprises labeled anomalous examples.
The system of claim 7, wherein the one or more hardware processors are further configured to perform the step of extracting a subset of the extended synthetic rare class super dataset to obtain a synthetic rare class dataset by:
determining a similarity function pertaining to the extended rare class dataset to obtain an extended rare class similar dataset;

generating a ranked extended rare class similar dataset by ranking elements of the extended rare class similar dataset based on a similarity index associated thereof and sorting in descending order; and

determining a diversity function for the ranked extended rare class similar dataset to obtain the synthetic rare class dataset with elements that are top ranked in the ranked extended rare class similar dataset and satisfies the diversity function.
The system of claim 10, wherein the diversity function is based on l-diversity.
The system of claim 7, wherein the synthetic rare class dataset is independent of dimensionality and is signal space rare class dataset.
A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
analyze a labeled abundant training dataset (
) and labeled rare class training dataset (
);

generate an extended synthetic rare class super dataset (
) based on the analysis using an extended oversampling method; and

extract a subset of the extended synthetic rare class super dataset (
) to obtain a synthetic rare class dataset (
), by maximizing similarity and diversity in the synthetic rare class dataset (
), wherein maximizing similarity ensures maximizing mutual information and maximizing diversity ensures minimum redundancy in the synthetic rare class dataset (
) such that morphological identity of the synthetic rare class dataset (
) is preserved with respect to the labeled rare class training dataset (
) while maintaining diversity.