KR102109854B1 - Data pre processing system and method for clinical data analysis - Google Patents

Abstract

The present invention relates to a system for data preprocessing automation used in clinical data analysis to automate a preprocessing process for clinical data to be used in a statistical analysis system to sequentially and automatically perform the preprocessing process for each variable. Conventionally, a data editor such as Excel was used to manually perform data preprocessing such as missing value processing and typographical error correction work. Such methods are used when the data size is relatively small and are close to an individual data organization concept rather than a data preprocessing concept. The present invention automates a data preprocessing process to be used for clinical data for each variable, divides and sets continuous variables/categorical variables in clinical data, provides a data preprocessing process to automatically perform a preprocessing process to be performed for each variable, and allows a user to easily check and reset properties for a corresponding variable for each variable to automatically perform data preprocessing of clinical data.

Description

Data pre-processing automation system and method used for clinical data analysis {DATA PRE PROCESSING SYSTEM AND METHOD FOR CLINICAL DATA ANALYSIS}

The present invention relates to a data pre-processing automation system used for analyzing clinical data intended to automate so that the pre-processing process for clinical data to be used in a statistical analysis system can be automatically performed sequentially for each variable.

delete

In various technical fields such as science and medicine, a statistical analysis method has been generalized to sample predetermined data from a parameter group or to analyze various parameters from sampled data. For example, clinical data performed in the medical field are analyzed through various statistical methods, and then published as a paper.
Here, data pre-processing is essential for proper statistical analysis.

[Necessity of preprocessing data to be used for statistical analysis]

When creating the data used for statistical analysis, not only errors that occur in human manual work, but also those that need to be excluded from actual analysis for reasons of mishandling by computer, or those that need to be changed are included.

In addition, not only the variables included in the data, but also a combination of them to create a new variable is often used in the analysis.

If this data is used directly for analysis without organizing, it is impossible to obtain a proper analysis result, and as a result, the researcher cannot perform the desired research.

[Conventional data pre-processing method]

When the data size is relatively small (eg, data consisting of 10 variables measured in 100 patients), data processing such as missing value processing and typos correction was performed manually using a data editor such as Excel.

This method is difficult to see as a concept of data pre-processing and is close to the concept of individual data organization.

If the data size is quite large, manual data preprocessing is difficult, so you can directly code a program that can perform data preprocessing using a statistical program language such as SAS or R scrip, or GUI-based statistics such as SPSS Data pre-processing was performed using analysis software.

However, coding the program directly is a difficult approach for non-IT and statistical experts, and it is difficult to obtain the expected effect because existing software does not have a systematic method for preprocessing data.

Republic of Korea Patent Publication No. 10-2018-0132238. Name of invention "Automatic creation and editing method of statistical result table or picture for thesis insertion, computer program implementing it, and information processing device performing the same.

The present invention is to enable the automation of data pre-processing to be used for clinical data by variable, and to set and classify continuous / categorical variables in clinical data, and to perform pre-processing for each variable automatically. By providing a pre-processing process, the user can easily check and reset the properties of the variable for each variable, so that the pre-processing of clinical data can be automatically performed.

The data pre-processing automation system used in the clinical data analysis of the present invention automatically determines the variables and sets the properties of the variables through the processing of missing values, changes in variable properties, etc., and performs data substitution of each variable, outlier processing, and creation of new variables. It is characterized in that data pre-processing for each variable included is provided to the user sequentially, so that the data processing that should be excluded from statistical analysis and the numerical value to be changed for each variable are automatically processed for each variable or easily performed by the user.

Data pre-processing automation system used in the clinical data analysis of the present invention,

The interface means providing an interface for the user's data pre-processing, the variable property automatic setting means for setting variables by checking the characters included in the input data and automatically setting the variable properties for each set variable, and the data pre-processing process are stored. It is managed and provides data preprocessing process management means for providing and managing data preprocessing process for each variable at the request of the data preprocessing control means, and an interface for data preprocessing to users for each variable according to the data preprocessing process, and input through the interface Data pre-processing control means for controlling data pre-processing according to the set value, and a user interface for generating new variables by combining variables after data pre-processing is provided through the interface means, and according to the combination of users, Statistical analysis to provide variable combination variable management means for generating and controlling rule variables and a user interface for checking the combined data distribution between variables through the interface means, and providing distribution confirmation and statistical information according to the user's request And means for extracting and providing data according to conditions input by the user and providing interface data for creating and inputting conditions for the user to extract data.

The data preprocessing control means includes data preprocessing process means of continuous variables that control data preprocessing according to the data preprocessing process for each continuous variable and categorical variable, and data preprocessing process means for categorical variables, and preprocessing data to a user. Provides the information to be provided through the interface required for the data pre-processing process, and controls the data pre-processing according to the user's setting value through the user interface,

The method for preprocessing data of continuous variable is to ask the user to confirm whether to keep the continuous variable or convert it to a categorical variable (new) after the automatic setting process of the variable property of the continuous variable is completed. When determining whether to perform data pre-processing or data pre-processing of categorical variables, or if it is determined to proceed as a continuous variable, the user is provided with an interface to change the variable name and user input information According to the variable name setting process to set the variable name, the outlier definition process to allow the user to select and set the numerical range to be used as the outlier, and replace the data so that the user can change the value of the data Process, missing values that enable users to register and delete new missing values Of course, it includes a presentation impyu (imputation) process, the new process variable generation, variable data distribution process to determine the missing values are replaced with non-missing data number,

The pre-processing method of data for categorical variables, upon completion of the automatic setting process of categorical variables, asks the user to confirm whether to keep the categorical variables or convert (new) to the categorical variables and preprocess the data of categorical variables. The process of determining whether to perform the process, or to preprocess the data of the continuous variable, or if it is determined to proceed by converting to the continuous variable, if the data contains any character, whether to process it as a missing value or not Provides an interface for asking the user whether to replace or not, and processes it according to the user's confirmation, and then creates a continuous variable, provides the user with an interface for changing the variable name and detailed category name, and provides the user with input information. Therefore, variable name setting process to change variable name and detailed category name, Detailed category order change process to change subcategory order, data replacement process to allow users to change the value of data, process to define missing values to allow users to register and delete missing values, numbers that are not missing values, or Characterized in that it includes an imputation process that replaces text, a new variable creation process, and a variable data distribution verification process.

And the system of the present invention, when clinical data is input, a character-containing judgment process for confirming whether a character is included in a variable of the clinical data,

When it is confirmed through the character inclusion determination process, the missing value is checked by referring to the list of missing values, and if all the included characters are missing values, the corresponding variable is deleted from the data, or if any non-numeric characters are included. In the process of setting as a categorical variable, if the letter consists only of numbers, the type of the number is judged. If the number type (N) is less than the set number (C), the variable is set as a categorical variable and the number type (N) is Variable setting process to set a variable as a continuous variable when the set number (C) is large,

If it is determined to be a categorical variable in the variable setting process, the variable property is automatically set, including the process of extracting the name of the variable, setting the title of the variable, and extracting the name of the detailed category and setting the title of the variable. Course,

If it is determined that the variable is continuous in the variable setting process, the process of automatically setting the variable properties of the continuous variable, including the process of extracting the name of the variable, setting the title of the variable, and calculating the numerical distribution,

When the automatic setting process of the variable properties of the continuous variable is completed, the user is asked to confirm whether to keep the continuous variable or convert (new) to a categorical variable to perform a data preprocessing process of the continuous variable, The process of determining whether to perform the pre-processing process of the categorical variable, and if it is determined to proceed as a continuous variable, provides an interface for changing the variable name to the user and sets the variable name according to the user's input information Name setting process, outlier definition process that allows the user to select and set the numerical range to be used as an outlier, data replacement process so that the user can change the value of data, and the user registers missing values And the process of defining missing values that allows deletion, and the missing values as numbers rather than missing values. Capped impyu presentation (imputation) process, data pre-processing of continuous variables including a new generation process variable, the variable distribution data verification process, and,

When the automatic setting process of the categorical variable is completed, the user is asked whether to keep the categorical variable or convert (new) to the categorical variable, or to perform the data preprocessing process of the categorical variable, or the continuous variable The process of deciding whether or not to perform the pre-processing process of data, and if it is determined to proceed by converting to a continuous variable, asks the user to confirm whether to process it as a missing value or replace it with a number when the data contains any characters. The process of creating a continuous variable after providing the interface and processing according to the user's confirmation, providing the user with an interface for changing the variable name and detailed category name, and changing the variable name and detailed category name according to the user's input information The variable name setting process to be done, the detailed category order to change the detailed category order Data substitution process to allow the user to change the value of data, data, definition process of missing values to allow users to register and delete missing values, and imputation process to replace missing values with numbers or letters rather than missing values , Data pre-processing of categorical variables, including new variable creation process and variable data distribution verification process,

When the data pre-processing process of the continuous variable and the data pre-processing process of the categorical variable is completed, the process of deleting the missing value data to delete samples having only missing values from the data;

It is characterized by creating a new variable by combining variables, checking the distribution of variable combination data, and completing the data preprocessing process, including the process of extracting data that meets the conditions.

According to the present invention,

The present invention automatically determines the variables of the clinical data, automatically sets the properties of the variables, and then provides a process so that the pre-processing for each variable can be automatically performed sequentially, thereby pre-processing the clinical data to be used for statistical analysis effectively. It can be done.

That is, while the pre-processing of clinical data is automatically performed, the user can perform substitution and setting of specific data for each variable during the pre-processing of each data for each variable, and also create new variables by combining each variable, making it more accurate. Data pre-processing can be done.

In addition, the user can adaptively receive statistics and analysis information of clinical data during each data pre-processing process, thereby enabling data pre-processing for more accurate statistical analysis.

1 is a flowchart showing an execution process of a data preprocessing automation method used in analyzing clinical data of the present invention.
2A is a diagram showing the definition of a missing value code, and FIG. 2B is a diagram showing a list of missing value characters.
Figure 3 is a table defining the basic properties of the variable.
4A and 4B are diagrams showing an example of a data file in which basic attributes of variables are defined.
5 is a flow chart showing a continuous variable data pre-processing process.
6 is a flow chart showing a categorical variable data pre-processing process.
7 is a block diagram showing the configuration of a data pre-processing automation system used in clinical data analysis of the present invention.
8 to 15 is a view showing a continuous variable data pre-processing control interface provided to the user through the interface means,
8 shows a case where the variable name / attribute change is selected when the user selects continuous variable data preprocessing.
9 shows a user interface for outlier definition.
10 and 11 are diagrams showing a user interface for performing a data substitution process, and FIG. 10 is an interface for changing individual values for directly selecting and inputting data from a list of data values, and FIG. 11 is a data value that satisfies the conditions After selecting and changing the conditions, the substitution interface is displayed.
12 shows a user interface for performing a process of defining missing values.
13 shows an interface for performing an imputation process.
14 and 15 show an interface for performing a new variable generation process, FIG. 14 shows a new variable generation interface to which a function transformation is applied, and FIG. 15 shows a new variable generation interface using an interval categorization setting.
16 to 22 is a view showing a categorical variable data pre-processing control interface provided to the user through the interface means,
16 shows a case in which the variable / detail category name and attribute change are selected when the user selects preprocessing of categorical variable data.
17 and 18 are views showing a user interface for performing a data substitution process, and FIG. 17 shows an interface for changing individual values by directly selecting and changing from a data value list, and FIG. 18 is a data value that meets the conditions After selecting and changing the conditions, the substitution interface is displayed.
19 shows a user interface for performing a process of defining missing values.
20 shows an interface for performing an imputation process.
21 and 22 show an interface for performing a new variable generation process, FIG. 21 shows a new variable generation interface by sub-category integration, and FIG. 22 shows a new variable generation interface by sub-category separation.
23 shows the data distribution confirmation interface of the continuous variable.
FIG. 24 shows a box float visualization parameter modification interface that appears when selecting the picture correction means of the box float in the interface of FIG. 23, and FIG. 25 shows a histogram visualization parameter modification interface that appears when selecting the picture modification means of the histogram in the interface of FIG. Indicates.
26 shows an interface for confirming the distribution of categorical variable data.
FIG. 27 shows a bar float visualization parameter modification interface that appears when a picture correction means for bar floats is selected in the interface of FIG. 26.
FIG. 28 shows an interface provided through text setting means to be inserted into a picture provided by the interfaces of FIGS. 25 and 27.
FIG. 29 shows the color adjustment interface provided through the color adjustment and picture file format setting means provided by the interfaces for the continuous and categorical variables of FIGS. 25 and 27.
30 to 32 show an interface for generating new variables through variable combination, FIG. 30 shows an interface for generating new variables by combining continuous variables, and FIG. 31 shows new variables by combining categorical variables. Represents an interface for generating, and FIG. 32 shows an interface for generating a new variable by combining a continuous variable and a categorical variable.
33 to 35 show an interface for confirming a variable combination data distribution, FIG. 33 is an interface for confirming a combined data distribution between continuous variables, and FIG. 34 is a combination between continuous variables and categorical variables. An interface for checking data distribution, and FIG. 35 is an interface for checking the combined data distribution between categorical variables.
36 shows an interface for extracting data meeting conditions.

The data preprocessing automation system used in the clinical data analysis of the present invention will be described with reference to FIG. 7 as follows.

Interface means 10 for providing an interface for the user's data pre-processing,

Variable property automatic setting means (20) for setting variables by checking the characters included in the input data and automatically setting variable properties for each set variable,

The data preprocessing process is stored and managed, and the data preprocessing process management means 30 for providing and managing the data preprocessing process for each variable at the request of the data preprocessing control means 40,

Data pre-processing control means (40) that provides an interface for data pre-processing to users for each variable according to the data pre-processing process and controls data pre-processing according to a set value input through the interface.

A variable combination variable management means (50) for providing a user interface for generating new variables by combining variables after data preprocessing is completed, and generating and controlling new variables according to a combination of users,

A statistical analysis means (60) for providing a user interface for confirming the combined data distribution between variables through the interface means and providing distribution confirmation and statistical information according to a user's request,

It provides interface data for creating and inputting conditions for the user to extract data, and includes data extraction means (70) for extracting and providing data according to the conditions input by the user.

The data pre-processing automation system used for the clinical data analysis of the present invention can determine whether it is a continuous variable or a categorical variable by checking the clinical data, and the user can easily determine the process items that require data pre-processing for each variable sequentially. By allowing the user to perform operations such as setting (change) and data substitution, data pre-processing of clinical data can be automatically performed.

The interface means 10 is a means for providing a user interface for inputting and checking information necessary for data preprocessing.

The variable property automatic setting means 20 is a means for setting a variable by checking a character included in the input data and automatically setting a variable property for each set variable,

Variable setting means for checking the characters included in the input data and processing the missing values in the characters by referring to the missing value list information, and variable setting means for setting continuous variables or categorical variables according to numeric information contained in the characters (22), variable attribute setting means (23) for automatically setting variable attributes according to a continuous variable or categorical variable set by the variable setting means (22), and missing value management means (24) for storing and managing missing value lists do.

The variable property automatic setting means 20 is set to a continuous variable when all numbers are excluding the missing value included in the letter, but is set to a continuous variable when the number of types is more than a predetermined number. If all are not numbers, if the number type is less than the specified number, set as a categorical variable.

The variable property setting means 22 is a means for automatically setting variable properties for each continuous variable and categorical variable. For the basic property information that is automatically set, the continuous variable is a variable title and a variable name. , Numeric distribution information, and categorical variables are variable title, variable name, and title of variable subtype and name of variable subtype.

The data preprocessing process management means 30 is a means for storing and managing the data preprocessing process, and includes a data preprocessing process for each continuous variable and categorical variable.

The data preprocessing process management means 30 may further include data preprocessing editing means for adding and deleting data preprocessing processes.

The data pre-processing control means 40 is a means for controlling data pre-processing according to the data pre-processing for each continuous variable and categorical variable, and provides information to be provided to the user through an interface required for data pre-processing according to the data pre-processing process. In addition, data preprocessing is performed and controlled according to a set value for data preprocessing input by a user through a user interface.

Data pre-processing for continuous variables is pre-processed for data of continuous variables by asking the user to confirm whether to keep the continuous variables or convert them into categorical variables (new) after the automatic setting process of the variable properties of the continuous variables is completed. If it is determined that the process is to be performed, whether to perform the pre-processing process of the categorical variable, or if it is determined to proceed as a continuous variable, the user is provided with an interface for changing the variable name and according to the user's input information. The variable name setting process to set the variable name, the outlier definition process to allow the user to select and set the numerical range to be used as the outlier, the data substitution process to allow the user to change the value of the data, The process of defining missing values, allowing users to register and delete new missing values It includes an imputation process that replaces values with numbers rather than missing values, a new variable creation process, and a variable data distribution verification process.

Data pre-processing of categorical variables, when the automatic setting process of categorical variables is completed, asks the user to confirm whether to keep categorical variables or convert (new) to categorical variables, and then process the data pre-processing of categorical variables. The process of determining whether to perform the pre-processing of data of the continuous variable, or converting it to the continuous variable, and if it is determined to proceed, replace it with a number to be processed as a missing value when the data contains any character. Provides an interface for asking the user to confirm whether it is processed and processes it according to the user's confirmation, and then creates a continuous variable, provides the user with an interface for changing the variable name and detailed category name, and provides the variable according to the user's input information. Variable name setting process to change name and detailed category name, detailed category order Kyung-ha is a process to change the detailed category order, the data substitution process to allow the user to change the value of the data, the process of defining the missing value to enable the user to register and delete the missing value, and to replace the missing value with a number or character instead of the missing value It includes the imputation process, the new variable creation process, and the variable data distribution verification process.

The variable combination variable management means (50), statistical analysis means (60), and data extraction means (70) are means for controlling a process that can be additionally performed by the user after the data preprocessing is completed.

The variable combination variable management means 50 is a means for generating new variables by combining data-preprocessed variables, and provides data pre-processed variables information, and uses the information to create new variables according to the user-combined information. Generate control.

The statistical analysis means 60 is a means for confirming the distribution of combined data between variables that have been preprocessed and providing statistical information.

The data extracting means 70 is a means for extracting and providing data according to a condition input by the user, and provides interface information for creating and inputting conditions for the user to extract data, the conditions input by the user, Data is extracted and provided according to the value.

Meanwhile, the method for automating data preprocessing used in the analysis of clinical data in the system of the present invention is as follows.

When the clinical data is input, the character-inclusive judgment process to check whether the variables in the clinical data contains characters,

When it is confirmed through the character inclusion determination process, the missing value is checked by referring to the list of missing values, and if all the included characters are missing values, the corresponding variable is deleted from the data, or if any non-numeric characters are included. In the process of setting as a categorical variable, if the characters in the data consist only of numbers, the type of the number is judged, and if the type of the number (N) is less than the set number (C), the variable is set as the categorical variable and the type of the number (N ) If the set number (C) is the same or more, the variable is set as a continuous variable and the variable setting process,

If it is determined to be a categorical variable in the variable setting process, the variable property is automatically set, including the process of extracting the name of the variable, setting the title of the variable, and extracting the name of the detailed category and setting the title of the variable. Course,

If it is determined that the variable is continuous in the variable setting process, the process of automatically setting the variable properties of the continuous variable, including the process of extracting the name of the variable, setting the title of the variable, and calculating the numerical distribution,

When the automatic setting process of the variable properties of the continuous variable is completed, the user is asked to confirm whether to keep the continuous variable or convert (new) to a categorical variable to perform a data preprocessing process of the continuous variable, The process of determining whether to perform the pre-processing process of the categorical variable, and if it is determined to proceed as a continuous variable, provides an interface for changing the variable name to the user and sets the variable name according to the user's input information Data pre-processing of continuous variables, including name setting process, outlier definition process, data substitution process, missing value definition process, imputation process, new variable creation process, and variable data distribution verification process,

When the automatic setting process of the categorical variable is completed, the user is asked whether to keep the categorical variable or convert (new) to the categorical variable, or to perform the data preprocessing process of the categorical variable, or the continuous variable The process of deciding whether or not to perform the pre-processing process of data, and if it is determined to proceed by converting to a continuous variable, asks the user to confirm whether to process it as a missing value or replace it with a number when the data contains any characters. The process of creating a continuous variable after providing the interface and processing according to the user's confirmation, providing the user with an interface for changing the variable name and detailed category name, and changing the variable name and detailed category name according to the user's input information The variable name setting process to be done, the detailed category order to change the detailed category order Information, data replacement process, missing values defined process, impyu presentation process, a data pre-processing of the categorical variable with the newly generated process variable, the variable distribution data verification process, and,

When the data pre-processing process of the continuous variable and the data pre-processing process of the categorical variable is completed, the process of deleting the missing value data to delete samples having only missing values from the data;

It consists of completing the data pre-processing, including creating new variables by combining variables, checking the distribution of variable combination data, and extracting data that meets the conditions.

The present invention relates to a pre-processing method for data used for statistical analysis of clinical data, and categorizes data into categorical variables and continuous variables to preprocess data for each categorical variable and continuous variable. It was made possible.

1 is a flowchart showing an execution process of a data preprocessing automation method used in analyzing clinical data of the present invention.

The character-containing judgment process and the variable setting process are processes for determining whether data variables are continuous variables or categorical variables.

The character inclusion determination process is for determining whether a character is included in the data, where the character includes a number.

The variable setting process,

Check the missing value by referring to the list of missing values.

(a). If all characters are missing, the corresponding variable is deleted from the data.

(b). If all characters except numbers are missing, the type (N) of the number is extracted, and if the type (N) of the number is more than a certain number (C), it is set as a continuous variable, otherwise set as a categorical variable. do.

2A is a diagram showing the definition of a missing value code, and FIG. 2B shows a list of missing value characters.

(c). If not all are set as categorical variables.

In other words, if all of them are numbers, it is set as a continuous variable, and when non-numeric characters are included, it is set as a categorical variable, but even when all of them are numbers, variables can be set according to the type (N) of numbers.

The automatic setting of the variable properties of the categorical variable and the automatic setting of the variable properties of the continuous variable are processes for automatically setting the basic properties of each variable.

The categorical variable is automatically set by extracting the variable title, variable name, and title of variable subtype and name of variable subtype.

Continuous variables are automatically set by extracting variable titles and variable names, and the numerical distribution is calculated.

3 is a table defining basic attributes of variables.

4A and 4B are diagrams showing an example of a file of data in which basic attributes of a variable are defined.

The data preprocessing process of the continuous variable and the data preprocessing process of the categorical variable represent a data preprocessing process that allows a user to easily check and modify (reset) the properties of the variable.

First, the data pre-processing process of the continuous variable,

(a). The process of confirming whether to perform preprocessing of data of continuous variables or whether to newly create or convert variables,

(b). Variable name setting process, (c). The outlier definition process, (d). Data substitution process, (e). The process of defining missing values, (f). Imputation process, (g). New variable creation process, (h). It includes the process of confirming the distribution of variable data.

5 shows a continuous variable data preprocessing process.

First, the user is asked whether to keep the continuous variable or convert to a categorical variable, and performs a data preprocessing process according to the user's selection.

That is, when the automatic setting process of the variable properties is completed, the user is asked whether to keep the continuous variable, convert to a categorical variable, or create a new one as a categorical variable, thereby performing the data preprocessing process of the continuous variable accordingly. It is determined whether or not to perform a data preprocessing process for categorical variables.

For example, when converting the continuous variable 'conVar1' consisting of numbers of (1,2,3, ..., 10) into a categorical variable, 'conVar1' was previously continuous, but after conversion It is a variable, and accordingly, a data pre-processing process is performed.

At this time, when the 'conVar1' variable is newly created as a categorical variable, the 'conVar1' variable remains, and a new variable in which the 'conVar1' variable is converted to a categorical type is created.

The variable name setting process is a process for providing a user with an interface for changing a variable name and setting a variable name according to user input information.

The outlier definition process is a process that allows a user to select and set a numerical range to be used as an outlier.

It provides one or more numeric ranges to choose from, and also provides an interface to allow users to directly enter numbers to specify the range, and sets outliers according to their selection and input.

The outlier process is only applicable to continuous variables, and means a number outside the minimum to maximum range to be used for the actual analysis.

The outlier-related figures that are automatically calculated as initial values are as follows.

Calculation of Q1 (1 ^st quartile), Q3 (3 ^rd quartile), and IQR (inter quartile range) → These numbers are all calculated statistically and cannot be modified by the user.

Outlier lower limit = (minimum value, Q1-1.5 x IQR or Q1-3 x IQR) → automatically calculated using Q1, Q3 or IQR automatically calculated above, the user can select one of them or directly modify the numerical value It is possible (initial value = minimum value)

Outlier upper limit = (maximum value, Q3 + 1.5 x IQR or Q3 + 3 x IQR) → auto-calculated using Q1, Q3 or IQR automatically calculated above, the user can select one of them or directly modify the numerical value Possible (initial value = maximum value)

The data replacement process is a process that allows a user to change the value of data, and provides an interface for a user to input a value along with the current value and sets the data value according to the user's input.

The value substituted by the user can be separately managed by registering in the substitution list.

Data substitution means replacing non-missing values with numbers or letters that are not missing.

The method of substituting data is (a). Method of directly inputting individual data, (b). A method that automatically replaces all data that meets the conditions.

Example: Replace all letters or numbers that match, with another letter or number, or a part of a letter or number with a specified letter or part of a number.

The process of defining a missing value is a process that allows a user to register and delete a missing value, allowing a user to input a missing value and register it in the missing value list, and also modify a missing value list that can be deleted by reading the missing value from the missing value list. Interface.

The imputation process is a process of replacing missing values with numbers rather than missing values.

The interface is provided to the user to replace the missing value, and the missing value is substituted according to the value input by the user.

Imputation means to replace missing values with numbers or letters that are not missing, and the specific method of implantation is as follows.

(a). In case of imputation by variable; Select all missing values of the variable from mean, median, and min / max (default = median)

(b). Repeatedly measured data with a time difference; Apply LOCF (Last Observation Carried Forward) method: Replace the missing value with the most recently observed value rather than the missing value → (used as the initial value), mean of the repeated measured data (mean), median (median), min / max Selection of values.

The new variable creation process is a process for creating a new variable, and provides a function item selection interface to apply to the variable selected by the user and an interface for directly inputting a function, and a newly generated variable selected or input by the user It creates a list and provides an interface for users to modify and manage the list.

The data distribution confirmation process is a process in which the user checks the data distribution of the registered continuous variable, and the user can check the numerical distribution in units of each continuous variable, and provides an interface for providing a numerical distribution as a figure (block) or graph. Provides

When it is a continuous variable, the data distribution (data distribution of variable),

(a). Minimum, maximum, mean, standard deviation, median

(b). Quartile 1 (1 ^st quartile), 3 quartile ^{(3 rd quartile), IQR (} inter quartile range)

(c). 95% confidence interval

(d). Numerical distribution

-Skewness (skewness), Kurtosis (Kurtosis)

-Normal distribution

Kolmogorov-Smirnov normality test result p value

Lilliefors normality test result p value

Shapiro-Wilk normality test result p value

(e). Effective sample count and percentage (%)

(f). Number and percentage of missing samples (%)

(g) cannot be modified

(h). Automatically created results

-Data distribution table by variable, histogram with normal distribution line, box plot

-Table of results incorporating variable data distribution

The data pre-processing process of the categorical variable,

(a). The process of confirming whether to perform pre-processing of data for categorical variables or whether to create or convert new variables,

(b). Variable name setting process to change variable name and detailed category name, (c). Detailed category order change process, (d). Data replacement process, (e). The process of defining missing values, (f). Imputation process, (g). New variable creation process, (h). It includes the process of confirming the distribution of variable data.

In addition, if a variable needs to be newly created or converted, if there is only a number, it is created as a continuous variable to perform the automatic setting process of the continuous variable, and if there is any character except a number in the data, the character is defined as a missing value or a number. It further includes a continuous conversion / new generation process to allow substitution.

6 shows a categorical variable data pre-processing process.

First, the user is asked whether to keep the categorical variable or convert it to a continuous variable, and performs a data preprocessing process according to the user's selection.

When the automatic setting process of the variable properties is completed, the user is asked whether to keep the categorical variable, convert to the continuous variable, or create a new one as the continuous variable, thereby performing the data preprocessing process of the categorical variable accordingly. It is determined whether or not to perform the data preprocessing process of the continuous variable.

As described in the continuous variable, when the categorical variable is converted into a continuous variable, the existing categorical variable is forcibly converted into a continuous variable, and the continuous variable preprocessing process is performed.

Also, when a new variable is created as a continuous variable, the variable is maintained as a categorical variable, and a new continuous variable is created.

The variable name setting process is a process for providing a user with an interface for changing variable names and detailed category names and setting variable names and detailed category names according to user input information.

The detailed category order changing process is a process of changing the order of detailed categories, providing a list of detailed category names, providing an interface for a user to change the order of detailed categories in the list, and providing detailed categories in the order the user selects. This is the process of setting the order.

The data substitution process is a process that allows a user to change the value of data, and provides an interface for a user to input a value along with the current value and sets the data value according to the user's input.

The value substituted by the user can be separately managed by registering in the substitution list.

Data substitution means replacing non-missing values with numbers or letters that are not missing.

As described in the data pre-processing method of the continuous variable, the data substitution method is (a). Method of directly inputting individual data, (b). It can be classified by automatically replacing all data that meets the conditions.

The process of defining a missing value is a process that allows a user to register and delete a missing value, allowing a user to input a missing value and register it in the missing value list, and also modify a missing value list that can be deleted by reading the missing value from the missing value list. Interface.

The imputation process is a process of replacing missing values with numbers or letters rather than missing values.

The interface is provided to the user to replace the missing value, and the missing value is substituted according to the value input by the user.

As also revealed in the data preprocessing process of the continuous variable, imputation means to replace missing values with numbers or letters rather than missing values, and the specific imputation method is as follows.

(a). In case of imputation by variable; Select all missing values of the variable from mean, median, and min / max (default = median)

(b). Repeatedly measured data with a time difference; Apply LOCF (Last Observation Carried Forward) method: Replace the missing value with the most recently observed value rather than the missing value → (used as the initial value), mean of the repeated measured data (mean), median (median), min / max Selection of values.

The new variable creation process is a process for creating a new variable, and provides a function item selection interface to apply to the variable selected by the user and an interface for directly inputting a function, and a newly generated variable selected or input by the user It creates a list and provides an interface for users to modify and manage the list.

The data distribution confirmation process is a process in which the user checks the data distribution of the registered continuous variable, and the user can check the numerical distribution in units of each continuous variable, and provides an interface for providing a numerical distribution as a figure (block) or graph. Provides

For categorical variables,

(a). The number of samples included in the subcategory and the percentage of the number of samples (%), at a 95% confidence interval of the percentage of samples,

-Calculation is based on only valid samples excluding missing samples

-Calculation is performed on all samples except for missing samples

(b). Effective sample count and percentage (%)

(c). Number and percentage of missing samples (%)

(d). Cannot be modified

(e). Automatically created results

-Data distribution table for each star, bar plot

-Table of results incorporating variable data distribution

On the other hand, if a variable needs to be newly created or converted, if there is only a number, it is created as a continuous variable to perform the automatic setting process of the continuous variable, and if there is any character except a number in the data, the character is defined as a missing value or a number. It further includes a continuous conversion / new generation process to allow substitution.

The continuous conversion / new generation process

When converting to a continuous variable or creating a new one, if the data contains at least one character, decide whether to treat it as a missing value or replace it with a number, and then attempt to convert or create the continuous variable. It is to provide and provide a continuous variable after processing according to the user's confirmation, to proceed with the continuous variable data preprocessing process.

The deletion data deletion process is a process for deleting data having only missing values after completing the data pre-processing process for the continuous variable and the data pre-processing process for the categorical variable.

The variable combination new variable creation process is a process for creating a new variable with a combination of variables.

(a). When using one variable,

When it is a continuous variable, (1). New generation of continuous variables by processing mathematical functions on variables (e.g., log conversion), new generation of categorical variables by segmenting the numerical range of variables (e.g., under 40, 41-59, over 60)

For categorical variables, (1). Create a new categorical variable by integrating the subcategories of variables (eg, no smoking, old smoking or current smoking, current smoking → smoking cessation, smoking), (2). Create a new two-way category variable with the number of categories in the same way as variables (specific detailed categories vs. others) (e.g. DM, HTN, HL → DM vs. others, HTN. Vs others, HL vs. others)

(b). When using two or more continuous variables in combination

Create new continuous variables using four or more continuous variables (eg body mass index = weight / key ²⁾

(c). When two or more categorical variables are used in combination

Combining detailed categories of each categorical variable to create a new categorical variable, e.g. (under 40, over 40) × (male, female) → (male under 40, female under 40, female over 40, 40 years old or older)

The variable combination data distribution confirmation process is a process of checking the distribution of variable combination data consisting of variable combinations as described above.

(a). Continuous variable * When the relationship between continuous variables,

[Automatic statistical analysis] Pearson correlation, Spearman correlation analysis

[Automatic result] Correlation index, p value, Scatter plot with regression line

(b). Continuous variables * When there is a relationship between categorical variables

[Statistic analysis applied automatically]

(One). When there are two subcategories included in a categorical variable (two sample T test)

Parametric method; Student T test, Welch T test result p value, non-parametric method: Mann-Whitney U test result p value

(2). When there are 3 or more subcategories included in a categorical variable

Parametric method: 1-way ANOVA result p value, non-parametric method: Kruskal-Wallis H test result p value

[Results created automatically]

(One). Table showing the number of samples, percentage (%), mean ± standard deviation, and p value in all samples and subcategories

(2). Bar plot

(c). Categorical variable * When there is a relationship between categorical variables

[Statistic analysis applied automatically]

(One). chi-squared test (with / without Yates ’continuity correction) result p value

(2). Fisher ’s exact test result p value

[Results created automatically]

(One). Table that shows the number of samples, percentage (%), and p values in the subcategory combination

(2). Bar plot

The data extraction process that satisfies the above conditions extracts only data that satisfies the condition range by combining the numerical range of the continuous variable and the detailed category of the categorical variable.

At this time, data extraction is performed by combining one or more of the AND, OR, and NOT operators.

The method for automating the preprocessing of the data preprocessing automation system used in the clinical data analysis of the present invention made as described above will be described with reference to the embodiments shown in the accompanying drawings.

The present invention determines the clinical data and automatically sets continuous variables and categorical variables, and provides an interface to the user so that each continuous variable and categorical variables can be automatically made according to a pre-processing process.

When data is first input, the data pre-processing control means 40 checks the characters included in the data through the variable attribute automatic setting means 20 to set the variables.

The variable attribute automatic setting means 20 determines whether a missing value is included in the input data in the missing value processing means 21, and then determines whether a missing value is made by referring to the missing value list shown in FIG. 2B in the missing value management means 24. Is done.

At this time, if all characters are missing values, the corresponding variable is deleted.

On the other hand, if all of the characters are not missing values, the variable setting means 22 determines the number of types of numbers to determine continuous variables and categorical variables.

That is, if all of them are numbers, they are set as categorical variables, and if the number of types of numbers (N) is greater than or equal to the set number (C), they are set as continuous variables. Set to

Thereafter, the toilet property setting means 23 sets the automatic property information of the continuous variable and the categorical variable.

The categorical variable is automatically set by extracting the variable title, variable name, and title of variable subtype and name of variable subtype.

Continuous variables are automatically set by extracting variable titles and variable names, and the numerical distribution is calculated.

In this way, when the automatic setting information for the continuous variable and the categorical variable is set through the variable property automatic setting means 20, the data preprocessing control means 40 enables the preprocessing process of the continuous variable and the categorical variable to be performed. A data pre-processing interface is provided to the user through the interface means 10.

The interface provided for data pre-processing for each variable includes variable name display means, variable data pre-processing selecting means for selecting variables to execute data pre-processing, and includes a data pre-processing sub-menu that appears according to the selection of the variable data pre-processing selecting means. do.

8 to 15 show a continuous variable data pre-processing control interface provided to the user through the interface means 10.

When the user selects the continuous variable data pre-processing, the variable name is displayed on the variable name display means at the top, and the user selects the lower sub-menu of the continuous variable data pre-processing selection means in order to check and reset the properties of the variable. It will perform.

8 shows a case where the variable name / attribute change is selected when the user selects continuous variable data preprocessing.

If variable name / property change is selected, the variable name change means is configured at the bottom, and the set variable name appears on the variable name change means, and at the bottom, it is used as a continuous variable or converted to a categorical variable. Variable setting means for resetting the variable for selecting whether or not is configured.

It also includes a data preprocessing step selection means (proceeding to the next step, returning to the previous step) and a data attribute composition checking means to check data attributes and distribution so that the data can be processed to the next step or to the previous step.

The user may select whether to reset the continuous variable to the categorical variable or keep the continuous variable as the continuous variable through the variable setting means.

When the user sets it as a categorical variable, pre-processing of categorical variable data is executed, and if it is maintained as a continuous variable, the current data pre-processing is maintained.

Thereafter, the user can select the outlier definition in the sub-menu or proceed to the next step to proceed with the outlier definition process.

9 shows a user interface for outlier definition.

The outlier definition interface provided to the user includes a range selection means to select a numerical range to be used as an outlier.

The range selection means includes a selection item to select a value to be used as an outlier, and an input means to directly input the value.

10 and 11 are diagrams showing a user interface for performing a data substitution process, and FIG. 10 is an interface for changing individual values for directly selecting and inputting data from a list of data values, and FIG. 11 is a data value that satisfies the conditions After selecting and changing the conditions, the substitution interface is displayed.

As shown in FIG. 10, the individual value change interface includes a data substitution list in which data old data values (old values) and new values (new values) are displayed, and includes data value input means for changing data values. do.

The user can check the previous data value and change the new data value through the data value input means, and the changed value is recorded in the 'New Value' of the data substitution list.

After setting the conditions, as shown in FIG. 11, the substitution interface includes a condition value input means to be changed and a value input means to be changed, and includes a data substitution list.

The user may select a data value to be substituted by directly inputting a data value to be changed or selecting from a data substitution list, and input the value after the change to add it to the substitution list.

In addition, the user can delete values by selecting (double clicking or deleting all) values of the data substitution list.

12 is a view showing a user interface for performing a process of defining missing values.

As shown in FIG. 12, the missing value definition interface includes a missing value list and missing value input means for adding to the missing value list, and the missing value list includes means for deleting missing values.

The user can add a value to be defined as a missing value to the missing value list by entering a new value in the missing value input section, and select the missing value list individually (double-click) or select Delete All to delete all missing values in the missing value list.

13 shows an interface for performing an imputation process.

13, the interface includes an item for selecting a missing value replacement method, as shown in FIG.

The user can perform the imputation (replacement of missing values) process by selecting the following items.

(a). Not used,

(b). Select values to replace missing values,

(c). When using the repeated measurement variable value over time

14 and 15 show an interface for performing a new variable generation process, FIG. 14 shows a new variable generation interface to which a function transformation is applied, and FIG. 15 shows a new variable generation interface using an interval categorization setting.

As illustrated in FIG. 14, the new variable generation interface using the function conversion provides a function selection item to be applied to the function conversion and a direct input means, and includes a list of new generation variables by applying the function conversion.

The newly created variable list provides newly created variable information, and includes means for selecting a newly created variable and deleting or deleting all of them.

The user can create a new variable by entering a function through one of the function selection items to be applied to the function conversion or through a direct input method, and select (double-click) the new variable created from the list of newly created variables to delete or delete all. You can delete all newly created lists by selecting it.

As shown in FIG. 15, the new variable generation interface using the section categorization setting includes a section setting means to be categorized, and includes a new variable generation list to which the categorization section is applied.

The section setting means to be categorized includes section number selecting means and decimal point number selecting means, and a section range list selected by section number selecting means and decimal point number selecting means is provided so that the section range can be modified.

The user can set the section to be categorized, and apply the categorization section to create new variables, and select and delete all the new variables created from the new variable creation list.

As described above, FIGS. 8 to 15 show a data preprocessing process of a continuous variable, and each interface commonly includes data attribute / distribution checking means, and includes data preprocessing finishing selection means to complete the data preprocessing process during the data preprocessing process. It includes a pre-processing data selection means that is activated to apply data pre-processing when all of the sub-menus of continuous variable data pre-processing are performed.

16 to 22 show a categorical variable data pre-processing control interface provided to a user through interface means.

When the user selects pre-processing of categorical variable data, the variable name is displayed on the variable name display means at the top, and the user selects the lower sub-menu of the continuous variable data pre-processing selection method in order and checks and resets the properties of the variable. It will perform.

16 shows a case in which the variable / detail category name and attribute change are selected when the user selects preprocessing of categorical variable data.

If you select to change the variable / detail category name and attribute, the variable name changing means is configured at the bottom, and the variable name setting means appears on the variable name changing means, or at the bottom, whether to use it as a categorical variable or a continuous variable. Variable setting means for resetting the variable for selecting whether to convert to is configured.

It also includes a detailed category name and order change means, and a detailed category name and order change means includes detailed category order initialization selection means.

It also includes a data preprocessing step selection means (proceeding to the next step, returning to the previous step) and a data attribute composition checking means to check data attributes and distribution so that the data preprocessing can proceed to the next step or proceed to the previous step.

The user can select whether to reset the categorical variable to a continuous variable or keep the categorical variable as it is through the variable setting means.

When the user sets the continuous variable, the preprocessing of the continuous variable data as described above is executed, and if the user maintains the categorical variable, the preprocessing of the current data is maintained.

In addition, the user can change each detailed category name or change the order through the detailed category name and order change means.

Thereafter, the user can select the individual value change from the sub-menu or proceed to the next step to perform the individual value change process in the data substitution process.

17 and 18 are views showing a user interface for performing a data substitution process, and FIG. 17 shows an interface for changing an individual value to directly select and input from a data value list, and FIG. 18 is a data value that satisfies a condition After selecting and changing the conditions, the substitution interface is displayed.

The individual value change interface includes a data substitution list in which the data old data value (old Value) and the new value (new Value) are displayed, as shown in FIG. 17, and the data substitution list changes data to change the data value. Means.

The user can check the previous data value and directly change the data value through the data value input unit.

After the condition is set, the substitution interface includes a condition value input means to be changed and a value input means to be changed, as shown in FIG. 18, and includes a data substitution list.

The user may select a data value to be substituted by directly inputting a data value to be changed or selecting from a data substitution list, and input the value after the change to add it to the substitution list.

In addition, the user can delete values by selecting (double clicking or deleting all) values of the data substitution list.

19 is a diagram showing a user interface for performing a process of defining missing values.

The missing value definition interface includes a missing value list and a missing value input means for adding to the missing value list, and the missing value list includes means for deleting missing values.

The user can add a value to be defined as a missing value to the missing value list by entering a new value in the missing value input means, and select the missing value list individually (double-click) or select Delete All to delete all missing values in the missing value list.

  20 shows an interface for performing an imputation process.

The implantation (imputation value replacement) interface includes a missing value replacement method selection means, as shown in FIG. 20.

The user may perform the imputation (missing value replacement) process by selecting the following items provided by the missing value replacement method selection means.

(a). Not used,

(b). Select values to replace missing values,

(c). When using the repeated measurement variable value over time

21 and 22 show an interface for performing a new variable generation process, FIG. 21 shows a new variable generation interface by sub-category integration, and FIG. 22 shows a new variable generation interface by sub-category separation.

As shown in FIG. 21, a new variable generation interface by subcategory integration is provided with detailed category display means included in the variable, and the display means includes setting means for setting detailed categories.

In addition, it includes a detailed category integration application means for generating a new variable by integrating and applying the selected detailed category to the setting means, and a list of newly generated variables applied with the detailed category integration.

The newly created variable list provides information on newly created variables, and includes means for selecting the newly created variables and deleting or deleting all of them.

As shown in FIG. 22, the interface for generating new variables by sub-category separation is selected for selecting whether to create new variables or to create new variables after separation without separating the sub-categories in creating new variables. Means and a list of newly created variables.

The user selects whether or not to separate the sub-category, or to create it as an individual variable after separating the sub-category, and creates a new variable by applying it, and the created variable is recorded in the new generated variable list.

The user can delete all of the new creation list by selecting (double clicking) the new variable created from the new creation variable list or selecting delete or delete all.

As described above, FIGS. 16 to 22 show a data preprocessing process for categorical variables, and each interface commonly includes data attribute / distribution checking means, and includes data preprocessing finish selecting means to complete the data preprocessing process during the data preprocessing process. It includes a pre-processing data selection means that is activated to apply data pre-processing when all of the sub-menus of continuous variable data pre-processing are performed.

And Figure 23 shows the data distribution confirmation interface of the continuous variable.

Executed through data attribute / distribution checking means, variable name display means, numerical display means for displaying numerical distribution as text figures, box plot display means for numerical distribution in block form, numerical distribution in graph form A histogram indicating means (including histogram display means), and the box float display means and histogram display means include picture correction means so that each picture can be corrected, and picture viewing means as a pop-up window for viewing pictures in a pop-up window.

FIG. 24 shows a box float visualization parameter modification interface that appears when selecting the picture correction means of the box float in the interface of FIG. 23, and FIG. 25 shows a histogram visualization parameter modification interface that appears when the picture modification means of the histogram is selected in the interface of FIG. Indicates.

The user can modify the box float visualization parameters through the box float box width, horizontal tail width, vertical tail width display range, axis display range, axis display method, picture visualization direction, picture width and height, resolution, and picture file format. Etc. can be set.

In addition, you can set the number of bars, axis display range, axis display method, horizontal and vertical size, resolution, and picture file format through the histogram visualization parameter modification interface.

26 shows an interface for confirming the distribution of categorical variable data.

And variable name display means for categorical variable data, numerical display means for displaying numerical distributions as text figures, and bar float display means for displaying numerical distributions as bar plots, wherein the bar float display means are It includes picture modification means so that you can modify the picture, and picture viewing means as a pop-up window that allows you to view the picture in a pop-up window.

FIG. 27 shows a bar float visualization parameter modification interface that appears when a picture correction means for bar floats is selected in the interface of FIG. 26.

The user can modify the bar float visualization parameter through the interface to display the bar width, bar spacing Y-axis display (number of samples, sample rate), Y-axis display range, picture visualization direction (vertical, horizontal), and text on the bar through the interface ( You can set the number of samples, the sample rate), the aspect ratio of the picture, the picture resolution, and the picture file format.

FIG. 28 shows an interface provided through text setting means to be inserted into a picture provided by the interfaces of FIGS. 25 and 27.

In the text setting means to be inserted into the picture, the text setting to be inserted into the picture includes text input, text size, text horizontal position, text vertical position, font, color, and a text list to be inserted. And means for deleting the registered text.

FIG. 29 shows the color adjustment interface provided through the color adjustment and picture file format setting means provided by the interfaces for the continuous and categorical variables of FIGS. 25 and 27.

The color adjustment interface allows you to adjust the inside color of the figure, the color of the outside line of the figure, and the background color, and includes a name and numerical (R, G, B) adjustment means for each color.

Meanwhile, FIGS. 30 to 32 show an interface for generating new variables through variable combination, FIG. 30 shows an interface for generating new variables by combining continuous variables, and FIG. 31 shows new by combining categorical variables. An interface for generating a variable is shown, and FIG. 32 shows an interface for generating a new variable by combining a continuous variable and a categorical variable.

When the user selects to create a new variable by combining variables, as shown in FIGS. 30 to 32, the menu of variable generation by continuous variable combination, variable generation by combination of categorical variable, and variable generation by combination of continuous variable and categorical variable An interface having a configuration is configured, and a new variable creation interface through each combination of variables as shown in FIGS. 30 to 32 is provided according to selection of each menu.

As shown in FIG. 30, the continuous variable combination new variable generation interface represents a registered continuous variable and allows a user to select a variable, a new variable for creating a new variable using variables selected from variable selection means and variable selection means. It comprises a variable generating means and a new variable list providing means providing a newly created variable list, and the new variable list providing means includes a new variable deleting means to delete the newly created registered variable according to a user's selection. .

As shown in FIG. 31, the categorical variable combination interface provides a variable selection means to select a variable to be combined, a detailed category reset means to reset a detailed category, a new variable generation means, and a newly created variable list. Means, and the newly created variable list providing means includes deletion means for deleting the newly created variable.

Continuous variable, categorical variable combination The new variable generation interface, as shown in FIG. 32, is a continuous variable selection and section setting means to be categorized, a categorical variable selection means to be combined with a continuous variable, and a detailed category adjustment means to be combined. , Means for generating new variables, and means for providing a list of newly created variables, and means for providing a list of newly created variables includes means for deleting a newly created variable.

33 to 36 show an interface for confirming a variable combination data distribution, FIG. 33 is an interface for confirming a combined data distribution between continuous variables, and FIG. 34 is a combination between continuous variables and categorical variables. An interface for checking data distribution, and FIG. 35 is an interface for checking the combined data distribution between categorical variables.

As shown in FIG. 33, the interface for confirming the combined data distribution between continuous variables includes means for selecting the variable to be combined, means for confirming the data distribution / statistic of the combined variable, and data distribution / statistic verification. It includes picture means representing the combined distribution of variable data through means, and means for providing a result of correlation analysis between variables.

The drawing means shows the result in a scatter plot considering the properties of the continuous variable.

As shown in FIG. 34, the interface for checking the combined data distribution between continuous variables and categorical variables includes means for selecting variables to be combined, means for checking data distribution / statistics of the combined variables, data Includes plotting means for displaying the combined variable data distribution through distribution / statistics checking means, and means for providing a result of mean difference test between variables.

The above-described drawing means is represented by a graph representing the difference in average values taking into account the properties of continuous variables and categorical variables.

As shown in FIG. 35, the interface for checking the combined data distribution between categorical variables includes means for selecting variables to be combined, means for checking data distribution / statistics of the combined variables, and data distribution / statistics checking. Includes a means of plotting the distribution of the variable data combined through the means, and a means of providing a result of contingency table analysis between the variables.

The above picture means is graphically considered in consideration of categorical variable properties.

36 shows an interface for extracting data meeting conditions.

Variable list providing means for variable selection, means for selecting sub-levels of categorical variables, means for specifying continuous variable ranges, setting condition setting means for setting conditions, registration means for registering data extraction conditions , Means for providing a list of conditions registered through the registration means, and data extraction and storage means for extracting and storing data according to the set conditions.

As described above, the present invention checks clinical data to determine continuous variables and categorical variables, and automatically sets each continuous variable, and then provides a preprocessing process required for each variable to the user to check and change data values (reset) ) To automatically pre-process clinical data.

Claims (26)

Interface means 10 for providing an interface for the user's data pre-processing,
Variable attribute automatic setting means (20) for determining whether a character is included in the input data, setting a variable according to the included character, and automatically setting a variable attribute for each set variable,
The data preprocessing process is stored and managed, and the data preprocessing process management means 30 for providing and managing the data preprocessing process for each variable at the request of the data preprocessing control means 40,
Data pre-processing control means (40) which provides an interface for data pre-processing to users for each variable according to the data pre-processing process and controls data pre-processing according to a set value for data pre-processing input through the interface.
A variable combination variable management means 50 that provides a user interface for creating new variables by combining variables after data preprocessing is completed, and generates and controls new variables according to a combination of users,
A statistical analysis means (60) for providing a user interface for confirming the combined data distribution between variables through the interface means (10), and providing distribution confirmation and statistical information according to a user's request,
It is used for clinical data analysis, characterized in that it comprises data extraction means (70) that provides interface data for creating and inputting conditions for extracting data and extracting and providing data according to conditions input by the user. Data preprocessing automation system. The method according to claim 1, wherein the variable attribute automatic setting means (20) comprises: a missing value processing means (21) for identifying missing characters in the input data and processing missing values in the characters by referring to missing value list information. Variable setting means 22 for setting a continuous variable or categorical variable according to numeric information, variable property setting means for automatically setting variable properties according to the continuous variable or categorical variable set by the variable setting means 22 23), a data preprocessing automation system used for clinical data analysis, characterized in that it comprises a missing value management means (24) in which the missing value list is stored and managed. The method of claim 2, wherein the variable attribute setting means (23) is a means for automatically setting variable attributes for each continuous variable and categorical variable, and the basic attribute information that is automatically set is a variable title for a continuous variable. , Variable name, numeric distribution information, and categorical variables are variable title, variable name, and title of variable subtype, name of variable subtype. ) Data pre-processing automation system used for clinical data analysis characterized by including information. The method of claim 1, wherein the data preprocessing process management means (30) includes a data preprocessing process for each continuous variable and categorical variable, and further includes data preprocessing editing means for adding and deleting data preprocessing processes. Data pre-processing automation system used in clinical data analysis, characterized in that. The method according to claim 1, wherein the data preprocessing control means (40) is a continuous variable, a data preprocessing process means of a continuous variable that controls data preprocessing according to the data preprocessing process for each categorical variable, and a data preprocessing process of a categorical variable. It includes a means, and provides the user with information to be provided through an interface required for data preprocessing according to the data preprocessing process, and performs and controls data preprocessing according to a user's setting value through a user interface.
After the automatic setting process of the variable properties of the continuous variable is completed, the method for preprocessing the data of the continuous variable asks the user to confirm whether to keep the continuous variable or convert (new) to a categorical variable, and input by the user In order to change the variable name to the user if it is determined to proceed as a continuous variable, the process of determining whether to perform the data preprocessing process of the continuous variable or the data preprocessing process of the categorical variable according to one information. A variable name setting process that provides an interface and sets variable names according to user input information, an outlier definition process that allows the user to select and set a numerical range to be used as an outlier, and the value of data Data substitution process so that the user can change, the user registers the missing value newly Definition process of missing values to be deleted, imputation process to replace missing values with numbers other than missing values, new variable creation process, and variable data distribution confirmation process to perform data preprocessing of continuous variables, and the categories If it is judged to be performed as a data preprocessing process of a type variable, the process includes switching to a categorical variable so that the data preprocessing process of the categorical variable is performed by the data preprocessing process means of the categorical variable,
The categorical variable data pre-processing process means, upon completion of the automatic setting of the categorical variable, asks the user to confirm whether to keep the categorical variable or convert (new) to a continuous variable, and to enter the user's input information. Therefore, the process of determining whether to perform the preprocessing process of the data of the categorical variable or the preprocessing process of the data of the continuous variable. Provides an interface for changing the name and detailed category name, the variable name setting process to change the variable name and detailed category name according to the user's input information, the detailed category order change process to change the detailed category order, and the value of the data (value ) So that the user can change the data substitution process, the user registers and deletes missing values Categorical variable data pre-processing including continuous process of defining variable data, including the process of defining missing values, imputation that replaces missing values with numbers or letters, missing variables, and checking the distribution of variable data If it is judged to proceed by converting to, it provides an interface for asking the user to check whether to treat it as a missing value or replace it with a number when the data contains any character, process it according to the user's input information, and then process the continuous variable A data pre-processing automation system used in clinical data analysis, comprising the step of generating a data pre-processing process of a continuous variable by means of a data pre-processing process of the continuous variable. The method according to claim 1, wherein the interface for data pre-processing for each variable provided from the data pre-processing control means (40),
As the user selects preprocessing of continuous variable data and preprocessing of categorical variable data, the variable name is displayed on the variable name display means at the top, and the process of checking and resetting the properties of the variable is selected while preprocessing the data of each variable selected by the user. The sub-menu is configured to perform, and variable setting means for resetting the variable whether to proceed with the selected variable (continuous variable or categorical variable) as it is or to convert to another variable (categorical variable or continuous variable) Data pre-processing for each configured variable and set data pre-processing step selection means, data attribute composition checking means for checking data attribute and distribution so as to proceed to the next step or proceed to the previous step. Data preprocessing automation system used. The method of claim 5, wherein the outlier definition interface provided to the user according to the continuous variable data preprocessing process from the data preprocessing control means (40) comprises:
The range selection means includes a range selection means to select a range of numbers to be used as an outlier, and the range selection means includes a selection item to select a number to be used as an outlier, and an input means to directly input a number. Data pre-processing automation system used in clinical data analysis, characterized in that. The method of claim 5, wherein the user interface for performing the data substitution process according to the variable data pre-processing process from the data pre-processing control means (40),
An individual value change interface including a data substitution list in which data old data values (old values) and new values (new values) are displayed, and including data value input means for changing data values,
A condition value input means to be changed, a value input means to be changed are configured, and after setting a condition including a data substitution list, a substitution interface is included,
The user can select the data value to replace by directly inputting the data value to be changed or selecting from the data substitution list, and add the value after the change to the substitution list, and the user selects the values of the data substitution list ( Automated data pre-processing system used in clinical data analysis, characterized in that it comprises a means to be deleted by double-clicking or deleting all). The method according to claim 5, wherein the user interface for performing the process of defining missing values according to the variable data preprocessing process from the data preprocessing control means (40) comprises:
A data preprocessing automation system used for clinical data analysis, comprising a missing value list and a missing value input means for adding to the missing value list, wherein the missing value list includes a deleting means. The method of claim 5, wherein the user interface for performing an imputation process according to the variable data pre-processing process from the data pre-processing control means (40),
Contains items for selecting the missing value substitution method,
The item for the user to select the method for replacing the missing value,
(a). Not used,
(b). Select values to replace missing values,
(c). In case of using the repeated measurement variable over time,
Data pre-processing automation system used in clinical data analysis, characterized in that it comprises a. The method according to claim 5, wherein the user interface for performing a new variable generation process according to the continuous variable data preprocessing process from the data preprocessing control means (40) includes a new variable creation interface to which a function transformation is applied, and a new variable using a section categorization setting. Includes a creation interface,
The new variable generation interface using the function conversion provides a function selection item and a direct input means to apply the function conversion, and includes a list of new generation variables by applying a function conversion, and the new generation variable list provides new generated variable information And includes means for selecting a newly created variable and deleting it or deleting all,
The new variable generation interface using the section categorization setting includes a section setting means to be categorized, and includes a new variable generation list to which the categorization section is applied,
The section setting means to be categorized includes a section number selection means and a decimal place number selection means, and a section range list selected by the section number selection means and the decimal place number selection means is provided so that the section range can be modified.
Includes means for selecting and deleting all new variables created in the new variable creation list, detailed category name and order changing means, and detailed category name and order changing means includes detailed category order initialization selecting means,
It includes data pre-processing step selection means (proceeding to the next step, returning to the previous step) and data attribute composition checking means to check data attributes and distribution so that data pre-processing can proceed to the next step or proceed to the previous step. Data pre-processing automation system used for clinical data analysis. 10. The method of claim 8, wherein the individual value change interface includes a data substitution list in which data old data values (old values) and new values (new values) are displayed, and the data substitution lists change data to change data values. Means,
After the condition is set, the substitution interface comprises a condition value input means to be changed and a value input means to be changed, and includes a data substitution list,
Data pre-processing automation system used in clinical data analysis, characterized in that it comprises a deletion means to select and delete the values of the data substitution list. The method according to claim 5, wherein the user interface for performing a new variable generation process according to the categorical variable data preprocessing process from the data preprocessing control means (40) includes: a new variable generation interface by subcategory integration, and a new one by subcategory separation. Contains a variable creation interface,
The new variable creation interface by sub-category integration is provided with a sub-category display means included in the variable, and the display means includes setting means for setting sub-categories, and the selected sub-category is integrated into the setting means to apply new A detailed category integration application means for creating a variable, and a list of newly created variables applied with a detailed category integration, and the newly created variable list provides new created variable information, and selects or deletes all newly created variables. Means that include,
The new variable generation interface by the sub-category separation includes selection means for selecting whether to create a new variable or to create a new variable after separation, without separating the sub-category in creating the new variable. Data pre-processing automation system used in clinical data analysis characterized in that it comprises a list. The method according to any one of claims 6 to 13, further comprising a data attribute / distribution checking means, a data preprocessing finishing selection means to complete the data preprocessing process during the data preprocessing process, and a variable data preprocessing submenu. It includes a data pre-processing execution selection means that is activated to apply the data pre-processing when all of the,
The interface to check the data distribution of continuous variables is executed through the data attribute / distribution verification means, the variable name display means, the numerical display means to display the numerical distribution as text figures, and the box plot to display the numerical distribution in block form. ) Display means, including a histogram display means indicating a numerical distribution in the form of a graph, wherein the box float display means and histogram display means are picture correction means so that each picture can be modified, and a pop-up window that allows you to view the picture with a pop-up window. Including means for viewing pictures, and
The interface for checking the distribution of categorical variable data is:
And variable name display means for categorical variable data, numerical display means for displaying numerical distributions as text figures, and bar float display means for displaying numerical distributions as bar plots, wherein the bar float display means are A data preprocessing automation system used for clinical data analysis, comprising a picture modification means so that a picture can be modified, a picture viewing means as a pop-up window allowing a picture to be viewed in a pop-up window. The method of claim 5, Variable combination The interface for creating a new variable through the variable combination provided by the variable management means (50), the interface to create a new variable by combining the continuous variable, new by combining the categorical variable It includes an interface for creating variables, and an interface for creating new variables by combining continuous and categorical variables.
The continuous variable combination new variable creation interface represents a registered continuous variable and allows a user to select a variable, a variable selection means, a new variable generation means for generating a new variable using variables selected from the variable selection means, and a new generation It comprises a new variable list providing means for providing a variable list, the new variable list providing means includes a new variable deletion means to delete the newly created registered variable according to the user's selection,
The categorical variable combination interface includes a variable selection means for selecting variables to be combined, a detailed category resetting means for resetting detailed categories, a new variable generation means, and a new variable list providing means. The newly created variable list providing means includes deletion means for deleting the newly created variable,
Continuous variable, categorical variable combination The new variable creation interface includes continuous variable selection and section setting means to be categorized, categorical variable selection means to be combined with continuous variables, detailed category adjustment means to be combined, and new variable generation means. A data preprocessing automation system used in clinical data analysis, comprising means for providing a newly created variable list, and the means for providing a newly created variable list comprises a deletion means for deleting the newly created variable. The method of claim 5, wherein the interface for checking the variable combination data distribution provided by the statistical analysis means (60) is an interface for checking the combined data distribution between continuous variables, continuous variables, combined between categorical variables. It includes an interface to check the data distribution, and an interface to check the combined data distribution between categorical variables.
The interface for checking the combined data distribution between continuous variables includes: means for selecting variables to be combined, means for checking data distribution / statistics of the combined variables, and variable data combined through data distribution / statistics checking means Includes a means for providing a picture representing the distribution and a result of correlation analysis between variables.
The interface for checking the combined data distribution between continuous variables and categorical variables includes means for selecting variables to be combined, means for checking the data distribution / statistics of the combined variables, and data distribution / statistics checking means. Includes plotting means representing the combined variable data distribution, and means for providing a result of mean difference test between the variables.
The interface for checking the combined data distribution between categorical variables includes means for selecting variables to be combined, means for checking the data distribution / statistics of the combined variables, and variable data combined through the data distribution / statistics checking means. Data pre-processing automation system used for clinical data analysis, characterized by comprising a means for providing a picture means for representing the distribution, a cross table analysis results between variables (Result of contingency table analysis). 17. The method according to claim 16, wherein the plotting means of the interface for confirming the combined data distribution between continuous variables shows the result as a scatter plot considering the properties of the continuous variables,
The plotting means of the interface to check the combined data distribution between continuous variables and categorical variables is represented by a graph representing the difference in average values taking into account the properties of continuous and categorical variables.
Data pre-processing automation system used for clinical data analysis, characterized in that the graphical means of the interface for confirming the combined data distribution between categorical variables is graphed considering categorical variable properties. The method of claim 6, wherein the interface for extracting data meeting the conditions provided by the data extraction means (70),
Variable list providing means for variable selection, means for selecting sub-levels of categorical variables, means for specifying continuous variable ranges, setting condition setting means for setting conditions, registration means for registering data extraction conditions , A means for providing a list of conditions registered through the registration means, and a data pre-processing automation system used for clinical data analysis, comprising data extraction and storage means for extracting and storing data according to the set conditions. When clinical data is inputted, the entered clinical data is checked to determine whether or not a character is included, the variable is automatically set according to the included character, and the variable attribute of the variable attribute automatic setting means 20 is used to automatically set the attribute for each set variable. It consists of an automatic setting process and a data preprocessing process of the data preprocessing control means 40 that performs data preprocessing according to the variables set in the automatic setting process and the properties of the variables,
The variable attribute automatic setting process of the variable attribute automatic setting means 20,
When clinical data is input, the character-inclusive judgment process to determine whether the variable of the clinical data contains a character, and when it is confirmed that the character is included through the character-inclusive judgment process, refer to the missing value list to confirm the missing value and include the If all the letters are missing, the corresponding variable is deleted from the data, if any non-numeric characters are included, the process is set as a categorical variable, and if the letters are composed only of numbers, the type of the number is judged and the type of the number (N ) Is less than the set number (C), the variable is set as a categorical variable, and when the number (C) of the number type (N) is set, the variable is set as a continuous variable, and the variable setting process and the variable setting process. If it is judged as a categorical variable in the process of extracting the name of the variable and setting the title of the variable, extracting the name of the detailed category and setting the title of the variable Auto-configuration process variables and properties of the categorical variable that contains the process,
In the variable setting process, if it is determined to be a continuous variable, it consists of the process of automatically setting the variable properties of the continuous variable, including the process of extracting the name of the variable, setting the title of the variable, and calculating the numerical distribution.
The data pre-processing process of the data pre-processing control means 40,
When the automatic setting process of the variable properties of the continuous variable is completed, the user is asked whether to keep the continuous variable or convert (new) to the categorical variable, and the data of the continuous variable according to the information input by the user. If it is determined to proceed with preprocessing, whether to perform preprocessing of data of categorical variables, or if it is determined to proceed as a continuous variable, the user is provided with an interface to change the variable name and the user's input information Therefore, the variable name setting process to set the variable name, the outlier definition process to allow the user to select and set the numerical range to be used as the outlier, and the data substitution process to allow the user to change the value of the data. , The process of defining missing values to enable users to register and delete missing values, The process of performing data pre-processing of continuous variables, including the process of imputation, new variable generation, and variable data distribution verification, replacing the measurement with numbers rather than missing values, and performing the pre-processing of data for the categorical variable. If it is determined that the pre-processing of the data of the continuous variable, including the process of switching to the categorical variable so that the data pre-processing of the categorical variable can be performed,
When the automatic setting process of the categorical variable is completed, the user is asked whether to keep the categorical variable or convert (new) to a continuous variable, and preprocess the data of the categorical variable according to the user's input information. The process of determining whether to perform or not to perform the preprocessing of data of continuous variables, provides the user with an interface for changing the variable name and detailed category name, and changes the variable name and detailed category name according to the user's input information. The variable name setting process to be performed, the detailed category order change process to change the detailed category order, the data substitution process to allow the user to change the value of data, the process to define the missing value to allow the user to register and delete the missing value, the missing value Immutation process, replacing new variables with numbers or letters rather than missing values, new variables If it is judged to perform the process of determining performance, variable data distribution, and preprocessing the data of the continuous variable by converting it to the continuous variable, if there is any character in the data, replace it with a number or treat it as a missing value. A data preprocessing process of a categorical variable that provides an interface for asking a user to confirm whether or not to process the data according to the user's input information, and then generates a continuous variable to perform the data preprocessing process of the continuous variable;
When the data pre-processing process of the continuous variable and the data pre-processing process of the categorical variable is completed, the process of deleting the missing value data to delete samples having only missing values from the data;
Create a new variable by combining variables, check the distribution of variable combination data, and complete the data preprocessing process, including the process of extracting data that meets the conditions.
Automated data pre-processing method used in clinical data analysis, characterized in that comprises. The method of claim 19, wherein in the data pre-processing of the continuous variable and the data pre-processing of the categorical variable, the process of substituting the data is (a). Method of directly inputting individual data, (b). Automated data pre-processing method used in clinical data analysis, characterized in that it comprises a method that automatically replaces all of the data that meets the conditions. 20. The method of claim 19, In the data pre-processing process of the continuous variable and the data pre-processing process of the categorical variable, the process of defining the missing value provides an interface for modifying the missing value list so that the user can newly register and delete the missing value in the missing value list. Automated data pre-processing method used in clinical data analysis, characterized in that it comprises a provided process. 20. The method of claim 19, In the data pre-processing process of the continuous variable and the data pre-processing process of the categorical variable, the imputation process is that the imputation is to replace the missing value with a number or a character rather than a missing value,
(a). In case of imputation by variable; Select all missing values of the variable from mean, median, and min / max (default = median)
(b). Repeatedly measured data with a time difference; Apply LOCF (Last Observation Carried Forward) method: Replace the missing value with the most recently observed value rather than the missing value → (used as the initial value), mean of the repeated measured data (mean), median (median), min / max Data pre-processing automation method used for clinical data analysis, characterized in that it is selected from among values. 20. The method of claim 19, In the process of completing the data pre-processing, creating a new variable by combining variables provides a function item selection interface to be applied to a variable and a user interface to directly input a function to be selected by the user, and the user selects it. Alternatively, a method of automating data pre-processing used in clinical data analysis, characterized in that it consists of a process of generating a newly created variable list to be input and providing an interface for a user to modify and manage the list. 20. The method according to claim 19, In the process of completing the data pre-processing, the variable combination data distribution check allows the user to check the data distribution of the registered continuous variables, and the user distributes the numerical distribution in units of each continuous variable. Data preprocessing automation method used in clinical data analysis, characterized in that it consists of a process of providing an interface that provides a graphical distribution of figures (blocks) and graphs. delete 20. The method of claim 19, In the process of completing the data pre-processing, the process of generating new variables by combining variables,
(a). When using one variable,
When it is a continuous variable, (1). Create new continuous variables by processing mathematical functions on variables, and create categorical variables by segmenting the numerical range of variables,
For categorical variables, (1). New creation of categorical variables by incorporating detailed categories of variables, (2). Create a new categorical variable of 2 minutes as many as the number of categories in the same way as variables (specific subcategory vs. specific subcategory; others)
(b). When using two or more continuous variables in combination
Creation of new continuous variables using four arithmetic operations on two or more continuous variables,
(c). When two or more categorical variables are used in combination
Combining detailed categories of each categorical variable to create a new categorical variable,
Automated data preprocessing method used in clinical data analysis, characterized in that.

Images