JP2006012140A - Anomaly detection in data perspective - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system and a method for providing automatic detection of data anomalies in a data perspective. <P>SOLUTION: The utilization of curve fitting data techniques provides automatic detection of data anomalies in a "data tube" from the data perspective, and makes it possible to detect data anomalies such as on-screen, and drill down and drill across data anomalies in, for example, pivot tables and/or OLAP cubes. Whether or not data substantially deviates from a predicted value established by a curve fitting process such as, for example, a piece-wise linear function applied to the data base is determined. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates generally to data mining, and more particularly to a system and method for providing automatic data anomaly detection of data perspectives.

  By digitizing information, a huge amount of data can be stored in a surprisingly small amount of space. This approach, for example, allows library library storage to be captured on a single computer hard drive. In this state, data is converted into a binary state, and can be stored in various digital storage media such as a hard drive, a CD-ROM disk, and a floppy (registered trademark) disk through the digital encoding device. To be possible. With the development of digital storage technology, the density of storage devices has been able to store much more data in a given amount of space than before, and the density of data mainly depends on physical properties and manufacturing methods. Limited by.

  As the storage capacity increases, there are increasing issues of effective data retrieval, and it is most important that data can be easily accessed. For example, if a book is in a library but cannot find it, it will not help the user who wants to read the book. Similarly, simply digitizing data is not a step forward unless the data is easily accessible. As a result, a data structure that helps efficient data retrieval was created. Such a structure is commonly known as a “database”. A database holds data in a structured format and provides efficient access to the data. By structuring the data store, the efficiency of data retrieval can be increased compared to an unstructured data store. Indexing and other organization techniques can also be utilized. In addition to data, the relationship between data can be stored to increase the value of the data.

  In the early days of database development, it was common for users to see “raw data”, ie data displayed as entered into the database. Eventually, technologies were developed that allow data to be formatted, manipulated, and viewed in a more efficient manner. This allows the user to apply mathematical operators to the data and create reports, for example. Business users can now access information such as “total sales” from database data that includes only individual sales. User interface development continued, making it easier to search and display data in an easy-to-use format. Users eventually realized that they could obtain additional information from raw data in the database with different views of the data, such as total sales from individual sales. This collection of additional data is known as “data mining” and produces “metadata” (data about the data). Data mining can extract useful additional information from raw data. This is particularly useful in businesses where information can be found that explains the reasons for a company's sales or output, and is superior to results obtained from raw input data in a database alone.

  Therefore, extremely important information can be extracted from the raw data by manipulating the data. This data manipulation is possible because of the digital nature of the stored data. Enormous amounts of digitized data can be viewed from various aspects, much faster than if attempted manually. Each new perspective of data can allow the user to gain further insight into the data. This is a very influential concept that can lead to the success or failure of a business using this concept. For example, trend analysis, cause-and-effect analysis, impact research, and prediction can be obtained from raw data entered in the database, and its value and timeliness are intuitive and easy to use for digitized information To have access.

  Currently, data manipulation to enhance the data mining function requires considerable user input and user knowledge in order to prevent error data from being included in various data perspectives. This requires the user to have deep knowledge of the data and insight into the types of errors that can occur in the data. Without this prior knowledge, the user must try a “spot” approach in the hope of finding data anomalies buried in a given data perspective. This approach generally does not add up to occasional users and / or is too time consuming for advanced users. In general, the amount of stored data is too large and complex to allow a user to efficiently develop a usable scheme for reliably finding all data anomalies.

  The following presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an extensive overview of the invention. This summary is not intended to identify key / essential elements of the invention or to define the scope of the invention. Its sole purpose is to present some concepts of the invention in a simplified form as a prelude to the more detailed description that is presented later.

  The present invention relates generally to data mining and, more particularly, to a system and method for providing automatic data anomaly detection in a data perspective. Utilizes data curve fitting technology to provide automatic detection of “data tube” data anomalies from the data perspective. A data tube contains data having only one changing data dimension. This allows, for example, on-screen data anomaly, drill down data anomaly, and down-down data anomaly in spreadsheets and / or OLAP (online analytical processing) cubes, etc. It is possible to detect data anomalies such as laterally developed data anomalies (drill across data anomaly). By providing automatic data perspective analysis, the present invention allows inexperienced users to easily find erroneous data information in a database. This is accomplished by determining if the data deviates significantly from the predicted value established by a curve fitting process, such as a piece-wise linear function applied to the data tube. The The threshold can also be used in the present invention to help determine the degree of deviation required before a data value is considered abnormal. This threshold can be provided dynamically and / or statically, for example by the system and / or by the user, such as via a user interface. The present invention also allows the user to easily indicate the type of anomaly detected and the location from the top data perspective, eliminating the need for the user to search for data anomalies at a lower level.

  To the accomplishment of the foregoing and related ends, illustrative aspects of the invention are described herein in connection with the following description and the annexed drawings. However, these modes merely show some examples of various systems in which the principle of the present invention can be used, and the present invention includes all such modes and equivalents thereof. Other advantages and novel features of the invention will become apparent from the following detailed description of the invention when considered in conjunction with the drawings.

  Next, the present invention will be described with reference to the drawings. In all the drawings, like elements are referred to with like reference numerals. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the present invention.

  As used herein, the term “component” is intended to refer to an entity associated with a computer, whether hardware, a combination of hardware and software, software, or running software. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and / or a computer. By way of illustration, both an application running on a server and the server can be a computer component. There may be one or more components in a process and / or thread of execution, and the components may be centralized on one computer and / or distributed across two or more computers. A “thread” is an entity in the process that the operating system kernel schedules for execution. As is well known in the art, each thread has an associated “context”, which is volatile data associated with the execution of that thread. The context of a thread includes the contents of the system register and a virtual address belonging to the thread's process. Therefore, the actual data consisting of the thread context varies at runtime.

  The present invention aids data perspective analysis by automatically detecting anomalous data. The display is used to provide a notification to the user that there is erroneous data at any level of the particular data perspective. This level is, for example, within the screen, ie the top level, and / or currently not displayed, expanding the data downwards to reveal the value of the erroneous data and / or moving the data horizontally It is a level that needs to be expanded. In this way, the user can easily determine the presence of a data anomaly and the amount of effort and / or data view required to reveal the erroneous data. Users and / or systems can also set thresholds statically and / or dynamically to aid in automatic detection. The user can also select different thresholds for various types of data anomalies. The threshold determines how far the data value must deviate before the data is considered abnormal. This deviation is determined by comparing the data value with the predicted data value obtained from the curve fitting process applied to a data tube having only one changing data dimension. The functions used in this curve fitting process can also be made selectable by the user. Thus, the present invention allows a user to easily identify interesting characteristics of the data being considered.

  FIG. 1 shows a block diagram of an automatic data perspective anomaly detection system 100 in accordance with an aspect of the present invention. The automatic data perspective anomaly detection system 100 comprises an automatic data perspective anomaly detection component 102 that receives a data perspective 104 and automatically determines a data anomaly 106. Data perspectives include but are not limited to spreadsheets, OLAP cubes, and the like. An optional external threshold input 108 can be utilized by the automatic data perspective anomaly detection component 102 to help determine which data is anomalous. The threshold may also be determined as part of the automatic data perspective anomaly detection component 102, such as, for example, a value determined by the system and / or a percentage of deviation determined by the system. A plurality of threshold values specified by the user can be used in the present invention to be used for different types of data anomalies. The automatic data perspective anomaly detection component 102 utilizes a curve fitting process applied to data tubes from the data perspective to determine which data is anomalous. The curve fitting process can also incorporate a user-specified function that helps automatic detection of data anomalies.

  Referring to FIG. 2, another block diagram of an automatic data perspective anomaly detection system 200 is shown in accordance with an aspect of the present invention. The automatic data perspective anomaly detection system 200 includes an automatic data perspective anomaly detection component 202 composed of a data tube component 204 and an anomaly detection component 206. The data tube component 204 receives the data perspective 208 and processes the data perspective 208 into a data tube. A data tube consists of a slice of data from the data perspective 208 with only one changing data dimension. Anomaly detection component 206 receives the data tube and processes the data tube using a curve fitting process to determine if there is a data anomaly. The curve fitting process consists of a process that attempts to generate a function that can estimate the data in the data tube. The estimated data becomes “predicted data” that is used to determine the deviation score of the data in the data tube. A threshold input 212 is used by the anomaly detection component 206 to determine the amount of deviation that can be tolerated. The threshold input 212 may be generated by the system and / or generated by the user. Data determined by the abnormality detection component 206 to exceed the threshold input 212 is output as an abnormality 210.

  Turning to FIG. 3, yet another block diagram of an automatic data perspective anomaly detection component 300 in accordance with an aspect of the present invention is shown. The automatic data perspective abnormality detection component 300 includes a data tube component 310 and an abnormality detection component 302. The abnormality detection component 302 includes a curve fitting function component 304, a data deviation score component 306, and an abnormality determination component 308. The curve fitting function component 304 receives a data tube from the data tube component 310 and determines a suitable function to represent the data tube data. Thereby, the prediction data of the data value of the data tube can be generated. The curve fitting function component 304 may receive an optional function 316 specified by the user and use it as an appropriate function. This allows the user to adjust the detection process. The data deviance score component 306 receives data from the data tube along with the curve fitting function from the curve fitting function component 304. The data deviance score component 306 uses a curve fitting function to predict the value of the data. These values are then compared with actual data values to determine a score based on the amount of deviation from the predicted value. Anomaly detection component 308 receives the departure score and utilizes threshold input 314 to detect data that exceeds the threshold. Data determined to exceed the threshold is considered to be abnormal and is output as data abnormality 312.

  To better understand the above system, it is helpful to understand the context and meaning of the data. Data perspectives such as pivot tables and / or OLAP cubes are important tools for business. They allow users to navigate large data sets quickly and easily, thereby helping business and other decisions. Basically, data perspectives such as pivot tables and OLAP cubes are n-dimensional views of the dataset. For example, Table 2 shows a pivot table corresponding to data partially shown in Table 1.

  In this data perspective, average sales as a function of date and “product category” are shown, and sales are averaged by region (sales region). In this example, “Sales” is the target, while “Date” and “Product Category” are the displayed dimensions, and “Region” is the aggregated dimension. In Table 2, the aggregation is average, but other aggregations are possible, such as total, minimum, maximum. Other data perspectives of the same data set are possible. For example, sales as a function of date and region, averaged by product category, are possible. The number of dimensions displayed may be more than two (see Table 4).

  Each dimension can have a hierarchy. In this example, the date hierarchy is year, quarter, week, the product hierarchy is product category, product, and the location hierarchy is region, state. One important part of a data perspective, such as a pivot table, is the level of the hierarchy that is displayed. In Table 2, the displayed levels are the year in the date dimension, the product category in the product dimension, and the region in the location dimension. The user can drill down to the displayed dimensions. This corresponds to moving to the next lower level in the dimension hierarchy (see Table 5). The user can also drill across a given pivot by expanding the pivot according to a dimension that is not currently in the pivot table. For example, Table 4 shows the result of horizontally expanding the pivot table of Table 2 on a regional basis.

  The pivot table also has a page field that contains the dimensions for selecting the data shown at any level of its hierarchy. In Table 2, the page field contains the location dimension at the regional level. Sales in all regions are selected. Alternatively, the user may select sales for a specific region or state. In general, a dataset pivot table consists of (1) a target, (2) a dimension displayed at any level of the hierarchy, (3) a dimension of a page field at any level of the hierarchy, and (4) an aggregate function. , Corresponding to.

  Typically, there may be an anomaly in one or more cells in a data perspective, such as a pivot table. The present invention automatically detects and displays at least three types of cell abnormalities such as (1) abnormalities in the screen, (2) abnormalities developed in the horizontal direction, and (3) abnormalities developed downward. . A cell becomes abnormal in the screen when there is an abnormality in relation to other data displayed on the screen. A cell becomes an anomaly developed in the lateral direction when the anomaly becomes apparent by expanding the cell in the lateral direction. A cell becomes a downward expansion abnormality when an abnormality becomes apparent by expanding into the cell. These types of abnormalities are shown in Table 3. Table 3 is the same as Table 2 except for the format.

  In Table 3, category 2/1999 cells are anomalies in the screen because they have a higher sales average than any other cell in the same row or column. The cell of category 2/2001 in Table 3 is an anomaly developed in the horizontal direction. This anomaly will not become apparent until the user deploys this data perspective horizontally on a regional basis, as shown in Table 4 below.

  In Table 4, sales of r3 are shown significantly lower than sales of r1 and r2. Moreover, the cell of Table 3, category 3/2002 is an abnormality developed downward. Again, this anomaly will not become apparent until the user expands the product hierarchy down as shown in Table 5 below.

  In Table 5, sales of product 3 are shown much lower than sales of product 1 and product 2. In these examples, abnormalities in the screen are highlighted, and abnormalities developed in the horizontal direction and abnormalities developed downward are indicated by frame lines. However, those skilled in the art will recognize that many other types are possible.

  Continuing with the example of automatic anomaly detection according to the present invention. The term “tube” is used to mean a slice of a given data perspective in which only one dimension changes. In the two-dimensional data perspective, tubes simply correspond to rows and / or columns. Some examples of tubes are shown in the three-dimensional pivot table in Table 4, which shows (1) different product categories with fixed dates and regions, and (2) various fixed product categories and dates. (3) The product category and the region are fixed, and various dates are different.

  A cell will be abnormal for the tube if it deviates significantly from the expected value for that cell as calculated by the curve fitting function. Rather than requiring the values in the data perspective to be contiguous, assume that the data perspective is one-dimensional and has an ordered index. For example, the data perspective can be a data perspective indexed by time, distance, or amount. Thus, the data perspective values may be continuous and / or discrete. In order to detect anomalies, a curve fitting method such as an “auto-regressive” type curve fitting method can be applied to the perspective. In one example of the present invention, anomaly detection can be facilitated by assigning a deviation score to the amount of deviation from the predicted value. The deviation score can then be compared with a given threshold to determine if an anomaly exists. For example, the probability of the observed value in the data perspective is obtained for discrete data. If the probability is significantly low, the data is considered abnormal.

In another example of the present invention, continuous dimensional data in a tube is applied to a piecewise linear function using, for example, a regression tree. And the cell
| Value in cell−Predicted value of cell |> Threshold (Formula 1)
In this case, the left side of this formula is the cell deviation score.

  In yet another case of the present invention, discrete dimensional data in the tube is fitted to the autoregressive model. Then, when the probability of the value in the cell is lower than some threshold value, the cell is abnormal.

  As is apparent from the above, the present invention applies different curve fitting functions to continuous data and discrete data. However, there are multiple methods for determining whether a dimension is discrete or continuous. For example, the user can specify a selection, for example by naming the dimension “number” via a format command or the like. Alternatively, as a further example, this selection can be made automatically by examining the data (named “Determining Whether A Variable Is Numeric Or Non-Numeric” filed by Heckrman et al. On April 23, 1999). System and method as described in US patent application Ser. No. 09 / 298,737).

  The three types of anomalies in this example are then defined for a given data perspective, such as a pivot table. A cell becomes an anomaly in the screen when there is an anomaly for any of the displayed tubes. Other definitions include, but are not limited to: (1) Anomaly in the screen when there is a cell abnormality for all displayed tubes, (2) An average along all tubes on the screen When the degree of deviation made exceeds a threshold value, an abnormality in the screen is possible. When there is an abnormal tube that changes over a dimension that is not displayed, the cell becomes an anomaly that is expanded laterally with the displayed dimension fixed. A cell becomes a downward expansion anomaly with all other displayed dimensions fixed when there is an abnormal tube that has been expanded to a deeper level and changes across the currently displayed dimension.

  Obviously, the abnormality developed in the horizontal direction and the abnormality developed in the downward direction cannot be seen by the user. The mechanism that reveals them tells the user which dimension and / or hierarchy needs to be expanded to see the anomaly. In some data perspective applications, this can be done via right clicking on a pointing device such as a mouse over an abnormal cell. In addition to indicating dimensions and / or hierarchies that include anomalies, the degree of anomalies can also be indicated, for example, by classifying dimensions and hierarchies according to corresponding deviation scores.

  With regard to the abnormality in the screen, the user may ask for an explanation of why the cell is abnormal. This is achieved through the present invention by displaying, for example, highlighting tubes whose deviation score exceeds a threshold. This function can also be used in some applications by using a right click of a pointing device such as a mouse.

  Two cases should be considered for the threshold. The first can expand into the cell in question and / or expand the cell in question horizontally and / or have multiple entries of the same cell in the original data. In this case, a threshold cσ can be used, where c is a constant controlled by the user, and σ is the standard deviation of data resulting from expanding the cell one or more times. Second, if the cell cannot be enlarged, or as an alternative to the above threshold, c <predicted value> or simply c as the threshold can be used, where c is also determined by the user. A constant to be controlled. Alternatively, the top k anomalies can be indicated and k is selected by the user. Alternatively, cells that cannot be expanded can be selected so that they are not named.

  In light of the exemplary system illustrated and described above, methodologies that can be implemented in accordance with the present invention will be better understood with reference to the flowcharts of FIGS. For clarity of explanation, the methodology is illustrated and described as a series of blocks, but the present invention is not limited by this block order, and some blocks are illustrated and described herein according to the present invention. It should be understood that this may be done in a different order than described and / or simultaneously with other blocks. Moreover, not all illustrated blocks may be required to implement a methodology in accordance with the present invention.

  The invention can be described in the general context of computer-executable instructions, such as program modules, being executed by one or more components. Generally, program modules include routines, programs, objects, data structures, etc. that perform particular tasks or implement particular abstract data types. In general, the functions of the program modules may be combined or distributed as necessary in various embodiments.

  FIG. 4 illustrates a flowchart of a method 400 that facilitates automatic data perspective anomaly detection in accordance with an aspect of the present invention. The method 400 begins (402) by receiving (404) tube data representing a data slice from a data perspective in which only one dimension changes. Anomalies in the data are then automatically detected (406) using a curve fitting function applied to the data. The curve fitting function can be derived and / or specified by the user. Anomaly detection can be further facilitated by threshold deviation values supplied by the system and / or user. The threshold departure value may vary depending on the type of data anomaly. The detected anomaly is then output as a data anomaly (408) and the flow ends (410).

  Referring to FIG. 5, another flow diagram of a method 500 that facilitates automatic data perspective anomaly detection in accordance with an aspect of the present invention is shown. The method 500 begins by receiving 504 tube data representing a data slice from a data perspective in which only one dimension changes. The function that best represents the data tube data is then determined (506). This function can be obtained through autoregressive processes such as piecewise linear processes and regression tree processes for continuous and discrete data. This function can also be obtained as a function provided by the user. Next, a deviation score is obtained based on the predicted value of the data obtained from the curve fitting function and the actual data value (508). A threshold is then received (510) that determines the amount of deviation allowed before the data value is considered abnormal. This threshold can be determined via the system and / or provided by the user. This threshold is a static value and / or a dynamic value. The threshold may be different depending on the type of data abnormality. A data anomaly is then detected by determining a data value having a deviation score that exceeds a threshold (512) and the flow ends (514). Typically, data anomalies are communicated to the user via in-screen displays such as highlighting, borders, and / or color coding. However, icons and other graphic indicators can also be used. Such a display allows the user to determine at what level data anomalies can be found. The display can also be used to indicate the type of data anomaly and / or the degree of deviation of the data anomaly. Other instances of the present invention include further operations that automatically display data anomalies to the user without requiring further user input to view the actual anomaly data. This drastically reduces the transmission of data to the user because the user has reached a data anomaly and does not need to know and understand all data level displays for viewing.

  To provide additional context for implementing various aspects of the present invention, a simplified and schematic representation of a suitable computing environment 600 capable of implementing various aspects of the present invention is shown in FIG. 6 and the following description. Provide a description. Although the present invention has been described above in the general context of computer-executable instructions for a computer program executing on a local computer and / or a remote computer, those skilled in the art will implement the invention in combination with other program modules. Recognize that you may. Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks and / or implement particular abstract data types. Further, those skilled in the art will understand that the method of the present invention may be implemented using a single processor or multiprocessor computer system, minicomputer, mainframe computer, personal computer, handheld computing device, microprocessor, and / or programmable home electrification. It will be appreciated that other computer system configurations including products and the like can be implemented. Each device as described above can be in operational communication with one or more associated devices. The aspects of the invention described herein may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of the present invention can be implemented in a stand-alone computer. In a distributed computing environment, program modules can be located in local and / or remote memory storage devices.

  As used herein, the term “component” is intended to refer to an entity associated with a computer, whether hardware, a combination of hardware and software, software, or running software. For example, a component may be, but is not limited to being, a process executed by a processor, a processor, an object, an executable, a thread of execution, a program, a computer, and the like. By way of illustration, an application running on a server and / or the server can be a component. A component can also include one or more subcomponents.

  With reference to FIG. 6, an exemplary system environment 600 for implementing various aspects of the invention includes a conventional computer 602 that includes a processing unit 604, a system memory 606, and various system configurations including system memory. A system bus 608 is included that couples the elements to the processing unit 604. The processing device 604 may be a commercially available processor or a unique processor. The processing device may be implemented as a multiprocessor formed of two or more processors that can be connected in parallel.

  The system bus 608 can be any of several bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using conventional bus architectures such as PCI, VESA, Microchannel, ISA, EISA, to name a few. Good. The system memory 606 includes a read only memory (ROM) 610 and a random access memory (RAM) 612. A basic input / output system (BIOS) 614 including basic routines that help transfer information between elements in the computer 602 at the time of startup or the like is stored in the ROM 601.

  The computer 602 can also include, for example, a hard disk drive 616, such as a magnetic disk drive 618 that reads and writes a removable disk 620, such as an optical disk drive 622 that reads and writes a CD-ROM disk 624 and other optical media. The hard disk drive 616, magnetic disk drive 618, and optical disk drive 622 are connected to the system bus 608 by a hard disk drive interface 626, a magnetic disk drive interface 628, and an optical drive interface 630, respectively. These drives 616-622 and associated computer readable media provide computer 602 with non-volatile storage of data, data structures, computer-executable instructions, and the like. While the above description of computer readable media includes hard disks, removable magnetic disks, and CDs, those skilled in the art will recognize other types of computer readable media such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, etc. It will be appreciated that can also be used in the exemplary operating environment 600, and that such media can hold computer-executable instructions for performing the methods of the present invention.

  A plurality of program modules may be stored in the drives 616-622 and the RAM 612, including an operating system 632, one or more application programs 634, other program modules 636, and program data 638. Operating system 632 may be any suitable operating system or combination of operating systems. By way of example, the application program 634 can include a data perspective analysis method according to an aspect of the present invention.

  A user may enter commands and information into the computer 602 through one or more user input devices such as a keyboard 640 and a pointing device (such as a mouse 642). Other input devices (not shown) include a microphone, joystick, game pad, satellite reception antenna, wireless remote controller, scanner, and the like. These and other input devices are often connected to the processing unit 604 through a serial port interface 644 coupled to the system bus 608, although other interfaces such as parallel ports, game ports, universal serial bus (USB), etc. You may connect with. A monitor 646 or other type of display device is also connected to the system bus 608 via an interface, such as a video adapter 648. In addition to the monitor 646, the computer 602 can include other peripheral output devices (not shown) such as speakers and printers.

  It should be understood that the computer 602 can operate in a network environment that uses logical connections with one or more remote computers 660. The remote computer 660 is a workstation, server computer, router, peer device, or other common network node, and for the sake of brevity, only the memory storage 662 is shown in FIG. Includes many or all of the elements described above in connection with computer 602. The logical connections shown in FIG. 6 may include a local area network (LAN) 664 and a wide area network (WAN) 666. Such networking environments are commonly found in offices, corporate computer networks, intranets, and the Internet.

  For example, when used in a LAN networking environment, the computer 602 is connected to the local network 664 through a network interface or adapter 668. When used in a WAN networking environment, the computer 602 typically includes a modem (telephone modem, DSL modem, cable modem, etc.) 670, is connected to a communication server on a LAN, or establishes communication through a WAN 666 such as the Internet. Have other means to do. The modem 670 may be internal or external to the computer 602 and is connected to the system bus 608 via the serial port interface 644. In a network environment, program modules (including application programs 634) and / or program data 638 can be stored in remote memory storage 662. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers 602 and 660 (wired or wireless) can be used in carrying out aspects of the invention. Like.

  In accordance with the practice of those skilled in the art of computer programming, the invention has been described with reference to acts and symbolic representations of operations performed by a computer, such as the computer 602 and remote computer 660, unless otherwise indicated. Such operations and operations are sometimes referred to as being performed by a computer. The operations represented symbolically as operations include the manipulation of electrical signals representing data bits by the processing unit 604, so that the electrical signal representation is converted or transformed, and the memory system (system memory 606, hard drive) 616, including floppy disk 620, CD-ROM 624, remote memory 662), data bits are retained, thereby reconfiguring the operation of the computer system and other signal processing, It will be understood that other or other changes may be made. The storage location where such data bits are held is a physical location having specific electrical, magnetic, or optical properties corresponding to the data bits.

  FIG. 7 is another block diagram of an example computer environment 700 with which the present invention can interact. System 700 further represents a system that includes one or more clients 702. Client 702 is hardware and / or software (threads, processes, computing devices, etc.). System 700 also includes one or more servers 704. Server 704 is also hardware and / or software (threads, processes, computing devices, etc.). The server 704 can accommodate a thread that performs conversion by using the present invention, for example. One possible communication between client 702 and server 704 is in the form of a data packet adapted to be transmitted between two or more computer processes. System 700 includes a communication framework 708 that can be used to facilitate communication between client 702 and server 704. Client 702 is operatively connected to one or more client data stores 710 that can be used to store information local to client 702. Similarly, server 704 is operatively connected to one or more server data stores 706 that can be used to store information local to server 704.

  In one example of the present invention, a data packet is transmitted between two or more computer components that aids data perspective analysis, the data packet at least partially comprising a curve fitting process applied to data tube data. The data tube consists of data slices that contain at least one data cell of the data perspective where only one data dimension changes.

  It should be understood that the system and / or method of the present invention can be utilized in a data perspective analysis scheme that helps both computer and non-computer related components. Further, those skilled in the art will recognize that the systems and / or methods of the present invention can be used in numerous electronic-related technologies such as, but not limited to, computers, servers, and / or handheld electronics.

  What has been described above includes examples of the present invention. Of course, it is not possible to describe every conceivable combination of components or methodologies to illustrate the invention, and those skilled in the art will appreciate that many further combinations and substitutions of the invention are possible. Let's be done. Accordingly, the present invention is intended to embrace all such alterations, modifications and variations that fall within the spirit and scope of the appended claims. Further, to the extent that the term “comprising” is used in the detailed description or in the claims, this term is similar to the interpretation of the term “comprising” when used as a conjunctive in a claim. Inclusive meaning.

1 is a block diagram of an automatic data perspective anomaly detection system according to an aspect of the present invention. FIG. 3 is another block diagram of an automatic data perspective anomaly detection system according to an aspect of the present invention. FIG. 6 is yet another block diagram of an automatic data perspective anomaly detection system according to an aspect of the present invention. 6 is a flowchart of a method that facilitates automatic data perspective anomaly detection in accordance with an aspect of the present invention. 4 is another flow diagram of a method that facilitates automatic data perspective anomaly detection in accordance with an aspect of the present invention. FIG. 2 is an exemplary operating environment in which the present invention can function. FIG. 4 is a diagram of another exemplary operating environment in which the present invention can function.

Explanation of symbols

104 Data Perspective 102 Automatic Data Perspective Anomaly Detection Component 106 Anomaly 108 Threshold Input 208 Data Perspective 202 Automatic Data Perspective Anomaly Detection Component 204 Data Tube Component 206 Anomaly Detection Component 210 Anomaly 212 Threshold Input 310 Data Tube Component 302 Anomaly Detection Component 304 Curve Fitting Function Component 306 Data deviation score component 308 Anomaly determination component 312 Anomaly 316 User specified function 314 Threshold input 606 System memory 632 Operating system 634 Application program 636 Other program module 638 Program data 604 Processing device 608 Bus 626 Har Disk drive interface 628 Magnetic disk drive interface 630 Optical drive interface 644 Serial port interface 646 Monitor 648 Video adapter 668 Network interface 664 Local area network 666 Wide area network 670 Modem 660 Remote computer 702 Client 704 Server 710 Client data store 708 Communication framework 706 Server data store

Claims (39)

A system that facilitates analysis of data perspectives,
A component that receives at least one data perspective;
An anomaly detection component that automatically analyzes the data perspective to detect at least one data anomaly through a curve fitting process applied to continuous and / or discrete data from a data tube; and The system includes a data slice including at least one data cell of the data perspective, wherein only one data dimension varies.   The system of claim 1, wherein the curve fitting process includes a process that uses a piecewise linear function at least in part.   The system of claim 2, wherein the piecewise linear function includes a function that at least partially uses a regression tree.   The curve fitting process comprises a process that at least partially uses a probabilistic model that predicts a value in the data perspective, the probabilistic model being dependent on the location of the value in the data perspective in a non-obvious manner. The system of claim 1, characterized in that:   The system of claim 4, wherein the probability model comprises an autoregressive model.   The system of claim 1, wherein the data anomalies include anomalies based on significant deviations in data values from other data values found in the data tube.   The system of claim 6, wherein the significant deviation is based on at least one deviation score that exceeds a given threshold.   8. The divergence score is based at least in part on the value of the data cell compared to a predicted value of the data cell obtained from a piecewise linear function representing a data tube that includes the data cell. The described system.   The deviation score is based at least in part on the value of the data cell compared to the predicted value of the data cell obtained from a probability model that predicts discrete values in the data perspective, the probability model being in the data perspective. 8. The system of claim 7, wherein the system depends on the location of the value in a non-obvious manner.   The system of claim 7, wherein the given threshold includes at least one selected from the group consisting of a dynamic threshold and a static threshold.   11. The system of claim 10, wherein the given threshold includes at least one selected from the group consisting of a user defined threshold and a system defined threshold.   The system of claim 11, further comprising a user interface component that provides a plurality of selectable user-defined thresholds for use with different types of data anomalies.   The system of claim 1, wherein the data perspective includes at least one selected from the group consisting of a pivot table and an online analytical processing (OLAP) cube.   The system of claim 1, further comprising a user interface component that indicates the data anomaly to at least one user.   The system of claim 14, wherein the user interface component indicates the data anomaly via at least one selected from the group consisting of visual instructions and audio instructions.   The user interface component via at least one usage selected from the group consisting of highlighting at least one anomaly at the top level and enclosing at least one hidden anomaly with a border; 15. The system of claim 14, which assists in indicating the data abnormality.   The system of claim 14, wherein the user interface component comprises a user interface having user input controls for adjusting the level of indication based on at least one degree of data anomaly.   15. The system of claim 14, wherein the user interface comprises a component that assists in indicating the data anomaly through an automatic screen display of at least one data anomaly. A method that facilitates analysis of data perspectives,
Receiving at least one data perspective;
Establishing a data tube from the data perspective, the data tube comprising a data slice including at least one data cell of the data perspective, wherein only one data dimension varies;
Determining a curve fitting function representing continuous and / or discrete data of the data tube;
Calculating a deviation score based at least in part on a difference between an actual data value and a predicted value provided via the curve fitting function;
Detecting data anomalies through evaluation of the deviation score and detection criteria.   The method of claim 19, wherein the curve fitting function comprises a user-selectable curve fitting function. Classifying the data anomalies according to their accessibility,
20. The method of claim 19, further comprising: displaying the data anomaly to a user utilizing a set of indications of accessibility to the anomaly.   The method further includes the step of limiting the data abnormality displayed to the user by using the number k of data abnormality selectable by the user, limiting the data abnormality to the top k abnormality based on a deviation score. The method according to claim 21. The method of claim 21, further comprising: automatically displaying on the screen at least one data anomaly to the user.   The display of the accessibility to the abnormality includes at least one selected from the group consisting of a display within a screen, a display expanded downward, and a display expanded horizontally. The method described.   The method of claim 19, wherein the data perspective includes at least one selected from the group consisting of a pivot table and an online analytical processing (OLAP) cube.   The method of claim 19, wherein the detection criteria includes a threshold. The evaluation of the deviation score is:
27. The method of claim 26, comprising determining whether a departure score exceeds the threshold.   27. The method of claim 26, wherein the threshold includes at least one selected from the group consisting of a dynamic threshold and a static threshold.   30. The method of claim 28, wherein the threshold includes at least one selected from the group consisting of a user defined threshold and a system defined threshold.   30. The method of claim 29, further comprising adjusting a threshold defined by the user according to a type of data anomaly.   27. The method of claim 26, wherein the curve fitting process includes a process that uses at least partially a piecewise linear function.   32. The method of claim 31, wherein the piecewise linear function includes a function that at least partially utilizes a regression tree.   The curve fitting process includes a process that at least partially uses a probabilistic model that predicts discrete values in the data perspective, wherein the probabilistic model depends in a non-trivial manner on the position of the values in the data perspective. 27. A method according to claim 26.   The method of claim 33, wherein the probabilistic model comprises a function that at least partially utilizes an autoregressive model. A system that facilitates analysis of data perspectives,
Means for receiving at least one data perspective;
Means for automatically analyzing said data perspective to detect at least one data anomaly through a curve fitting process applied to continuous and / or discrete data from a data tube;
The data tube comprises a data slice comprising at least one data cell of the data perspective, wherein only one data dimension changes.   Data packets transmitted between two or more computer components that facilitate analysis of data perspective, at least partially utilizing a curve fitting process applied to continuous and / or discrete data from data tubes A data packet comprising at least a portion of information relating to a data perspective analysis system, wherein the data tube comprises a data slice comprising at least one cell of the data perspective, wherein only one data dimension varies .   A computer readable medium having stored thereon computer executable components of the system of claim 1.   20. A device using the method of claim 19, comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device. The device using the system of claim 1, comprising at least one selected from the group consisting of a computer, a server, and a handheld electronic device.

Images