WO2025257932A1 - Method for optimizing data processing, data processing apparatus and program - Google Patents

Abstract

There is provided a technique that contributes to the optimization of data cleaning prior to data analysis. A method for optimizing data processing of an input table includes performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting a column having one or more missing values, the null bitmap being a metadata of the first target table; and performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein, the limitation process is performed with the input table as the first target table.

Description

METHOD FOR OPTIMIZING DATA PROCESSING, DATA PROCESSING APPARATUS AND PROGRAM

The present invention relates to a method for optimizing data processing, a data processing apparatus and a program.

In today’s world, we have a huge amount of raw data which can be used to address different kinds of business problems. Data preparation and cleaning is one of the very important steps before performing any kind of data analysis related to the business problem. As a part of data cleaning, it’s important to know the number of missing values present in a given tabular data and treat them before performing any meaningful analysis.

As a method of treatment, we often remove (drop) the rows/columns of tabular data that contain a significant number of missing values. The time for calculating the number of missing values in a given tabular data takes longer with the growth of input data size.

Often, a Data Scientist calculates the number of missing values present row-wise or column-wise in the given tabular data (input table) to make a necessary decision on how to impute the missing values, as illustrated in Fig. 21(a).

According to a disclosure in Non-Patent Literature 1, for calculating the number of missing values in pandas-like dataframe libraries, insull() method is used. This means that during calculation, an intermediate table of M x N dimension (where M is the number of rows and N is the number of columns of the input table) having Boolean values (True/False) representing whether the respective value in the Mth row and Nth column is missing (NA) is created, as illustrated in Fig. 21(b). Then, row-wise summation of the intermediate table considering True as 1 and False as 0 is calculated to find number of missing values in each row. If necessary, rows/columns of data that include a significant number of missing values are dropped according to predetermined drop options.

[NPLT 1] "pandas: Detect and count NaN (missing values) with isnull(), isna()", Modified: 2023-08-02 Internet <URL: https://note.nkmk.me/en/python-pandas-nan-judge-count/#count-nan-in-each-row-and-column>

The following analysis is made by the inventor of the present invention.

Both calculating the number of missing values and dropping rows/columns with a predetermined number of missing values involve steps of creating the intermediate table, which has the same dimension as the input table. Therefore, a computational memory to be allocated increases. Also, even if there are no missing values in a given column, it will be scanned, and a column of False values will be created. The column having False values will not impact the result of the count, so this is a waste of computation time. Accordingly, data processing apparatus are required to optimize data cleaning by reducing the amount of computational memory to be allocated during computation and reducing computation time.

In view of the foregoing, the present invention has been made. It is an object of the present invention to provide a technique that contributes to an optimization of data cleaning prior to data analysis.

According to a first aspect of the present invention, there is provided a method for optimizing data processing of an input table, comprising:
-- performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
-- performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
-- the limitation process is performed with the input table as the first target table.

According to a second aspect of the present invention, there is provided a data processing apparatus for processing data in an input table, comprising:
-- a limitation part that performs a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
-- a count part that performs a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
-- the limitation part performs the limitation process with the input table as the first target table.

According to a third aspect of the present invention, there is provided a program causing a computer to execute:
-- performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
-- performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap.

The above program may be stored in a computer-readable storage medium. The storage medium may be a non-transitory one such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, and the like. The present invention can be realized as a computer program product.

According to the present invention, it is possible to contribute to an optimization of data cleaning prior to data analysis.

[Fig. 1] Fig. 1 is a diagram illustrating an outline of an example of a present disclosure.
[Fig. 2] Fig. 2(a) is a block diagram illustrating an example of a functional configuration of a data processing apparatus of the present disclosure. Fig. 2(b) is a flowchart illustrating an example of a missing value count operation of the present disclosure.
[Fig. 3] Fig. 3(a) and Fig. 3(b) are diagrams illustrating an example of a null bitmap of the present disclosure.
[Fig. 4] Fig. 4 is a diagram illustrating an outline of an example of the present disclosure.
[Fig. 5] Fig. 5 is a diagram illustrating an outline of an example of the present disclosure.
[Fig. 6] Fig. 6(a) and Fig. 6(b) are block diagrams illustrating examples of functional configurations of data processing apparatuses of the present disclosure.
[Fig. 7] Fig. 7(a) is a flowchart illustrating an example of a missing value count operation of the present disclosure and Fig. 7(b) is a flowchart illustrating an example of a chunk operation of the present invention.
[Fig. 8] Fig. 8 is a diagram illustrating an outline of an example of the present disclosure.
[Fig. 9] Fig. 9 is a diagram illustrating an outline of an example of the present disclosure.
[Fig. 10] Fig. 10 is a diagram illustrating examples of drop options of the present disclosure.
[Fig. 11] Fig. 11 is a block diagram illustrating an example of a functional configuration of a data processing apparatus of the present disclosure.
[Fig. 12] Fig. 12(a) to Fig. 12(c) are diagrams illustrating an example of threshold-approach of the present disclosure.
[Fig. 13] Fig. 13 is a diagram illustrating an outline of an example of threshold-approach of the present disclosure.
[Fig. 14] Fig. 14 is a diagram illustrating an outline of a variation example of threshold-approach of the present disclosure.
[Fig. 15] Fig. 15(a) to Fig. 15(c) are diagrams illustrating an example of all-approach of the present disclosure.
[Fig. 16] Fig. 16 is a diagram illustrating an outline of a variation example of all-approach of the present disclosure.
[Fig. 17] Fig. 17(a) to Fig. 17(c) are diagrams illustrating an example of all-approach of the present disclosure.
[Fig. 18] Fig. 18 is a diagram illustrating an outline of a variation example of all-approach of the present disclosure.
[Fig. 19] Fig. 19(a) is a flowchart illustrating an example of a missing value drop operation of the present disclosure and Fig. 19(b) is a flowchart illustrating an example of a chunk operation of the present invention.
[Fig. 20] Fig. 20 is a hardware configuration diagram of a data processing apparatus according to the present disclosure.
[Fig. 21] Fig. 21(a) and Fig. 21(b) are diagrams illustrating related art.

<<First Example Embodiment>>
Fig. 1 is a drawing illustrating an outline of the first example embodiment. Hereinafter, in the description of the example embodiments, an input tabular data to be processed is referred to as an input table 310.

In the present example embodiment, the number of missing values (NA, or null) in the input table 310 is counted row-wise. As illustrated in Fig. 1, prior to counting the missing values, an extraction table 320 is generated, in which only a column(s) having the missing value(s) is extracted from the input table 310. Additionally, bits of a null bitmap 330, which are added to the input table 310 as metadata in advance, are used to count the missing value(s).

<Functional Configuration>
Fig. 2 (a) is a block diagram for illustrating an example of a functional configuration of a data processing apparatus 100 according to the present example embodiment. As illustrated in Fig. 2(a), the data processing apparatus 100 is provided with a limitation part 110 and a count part 120.

The limitation part 110 performs a limitation process in which an extraction table 320 is generated from a first target table using bits of the null bitmap 330 by extracting a column having one or more missing values. The null bitmap 330 is a metadata of the first target table (310). The extraction table 320 is a table having only a column(s) with one or more missing values. Here, the limitation part 110 uses the input table 310 as the first target table.

The count part 120 performs count process. In the count process, the count part 120 obtains the number of missing values per row in the extraction table 320 by performing a calculation with the bits of the null bitmap 330.

Fig. 3(a) and Fig. 3(b) are diagrams illustrating an example of the null bitmap 330. The null bitmap 330 is a well-known lightweight data structure (consumes 1 bit for each value) that represents each valid value (non-missing value/non-null) as “1” and each invalid value (missing value/null) as “0”, as illustrated in Fig. 3(a).

Using bits of the null bitmap 330, the nullity of 8 (or more) elements can be checked at once, instead of checking them one by one. This means that the number of missing values (null-count) of each column can be calculated (counted) quickly. The calculated null count is maintained as metadata, as illustrated in Fig. 3(b), to decide whether to extract the column in limitation process for further processing in count part.

The count part 120 may perform a calculation that, for example, subtracting a sum of the bits of the null bitmap 330.b for each row from the number of columns in the extraction table 320. Here, the null bitmap 330.b is a table extracting columns having nulls with respect to table 320.

<Missing values count operation>
The flow of a missing value count operation performed by the data processing apparatus 100 according to the present example embodiment is described. Fig. 2(b) is a flowchart illustrating an example of the missing value count operation performed by the data processing apparatus 100. This operation is initiated when an input table 310 is given and/or an instruction to start the process is received from a user.

The limitation part 110 performs a limitation process (Step S1101). In the limitation process, the limitation part 110 generates the extraction table 320 from the input table 310 using bits of the null bitmap 330. The extraction table 320 is a table generated by extracting a column(s) having a missing value(s). The null bitmap 330 is a metadata of the input table 310.

The count part 120 performs a count process (Step S1102) and terminates the missing value count operation. In the count process the count part 120 performs a calculation with the bits of the null bitmap 330.b and obtains the number of the missing values per row of the extraction table 320.

The data processing apparatus 100 may refer to the number of missing values in each row, determine whether a row is to be dropped (drop row) according to a predetermined drop option, and drop the row if it is determined to be dropped. For example, the data processing apparatus 100 may create a mask vector that masks the drop row(s) and perform a filter process on the extraction table 320 using the created mask vector. The mask vector is a vector having True for each row determined to be kept and having False for each row determined to be dropped.

As explained above, the data processing apparatus 100 limits the check only to columns having missing values when counting the number of missing values row-wise. Therefore, according to the data processing apparatus 100, unnecessary calculations can be avoided, resulting in reduction in total computation time.

Further, the data processing apparatus 100 uses bits of the null bitmap 330 when counting the number of missing values, thereby, avoiding creation of an intermediate table having the same size of the input table 310. As a result, an increase in computational memory is avoided. That is, the amount of computational memory to be allocated during the computation is reduced.

According to the data processing apparatus 100, it is possible to contribute to an optimization of data cleaning prior to data analysis.

<<Second Example Embodiment>>
A data processing apparatus, according to the present example embodiment, can further reduce the amount of memory allocated during calculations and unnecessary scans when counting missing values.

For example, as illustrated in Fig. 4, in an extraction table 320, which is generated from an input table 310 in a manner described in the first example embodiment, rows without a single missing value do not need to be calculated.

The data processing apparatus according to the present example embodiment reduces scan for rows that do not require calculations as much as possible. That is, it reduces scan of rows of a chunk, if there are no null values in the entire chunk (given the tables can be split in k number of chunks).

For example, as illustrated in Fig. 5, the data processing apparatus of the present example embodiment generates the extraction table 320 from the input table 310 as described in the first example embodiment. Then, it splits the extraction table 320 into K chunk tables 410 of a suitable size. In Fig. 5, an example of a case where the extraction table 320 is split into three chunk tables 411, 412, and 413.

The data processing apparatus of the present example embodiment performs the same processes as described in the first example embodiment on each chunk table 410 to count the number of missing values row-wise. The processes can be performed in parallel.

As illustrated in Fig. 5, if no columns have missing values (N) in any of the chunk tables (411, 412, 413), the data processing apparatus of the present example embodiment returns an array 332 of length L (length of the chunk table 412 currently processed) filled with zeros. Further, if there are columns with missing values, it extracts only those columns to generate extraction chunk tables 420. Then, it inspects their bits of a null bitmap 330 to calculate the row-wise sum 331 of non-missing values of each chunk table 410 and subtracts the result from the number of extracted columns to get the row-wise sum of missing values.

To achieve this, the data processing apparatus 100a of the present example embodiment is provided with a limitation part 110, a count part 120, and a split part 130, as illustrated in Fig. 6(a). The data processing apparatus 100a is also provided with a storage part 180. The storage part 180 stores various data necessary for performing processes. Configurations with the same name as the first example embodiment have basically the same functions as the first example embodiment.

The limitation part 110 performs a limitation process. In the limitation process, the limitation part 110 generates an extraction table 320 from a target table using bits of a null bitmap 330 by extracting a column having one or more missing values from the target table. The null bitmap 330 is a metadata of the target table. In the present example embodiment, the limitation part 110 generates the extraction table 320 from the input table 310, as in the first example embodiment. In addition, as described below, the limitation part 110 generates an extraction chunk table 420 from each of the chunk tables 410 generated by the split part 130.

The limitation part 110 of the present example embodiment counts the number of columns having one or more missing values (null values) of the given input table 310, using bits of the null bitmap 330. When the number of columns having the missing value(s) is one or more, it specifies the column(s) having the missing value(s). Then, it extracts the column(s) specified and thereby generates the extraction table 320.

The limitation part 110 sums the bits in the column direction for each column of the input table 310 and determines a column that the sum result is equal to or greater than 1 as the column having the missing value(s). Hereinafter, the column having the missing value(s) is referred to as null-col (null-column).

The count part 120 of the present example embodiment performs a count process in the extraction table 320. In the count process, the count part 120 obtains the number of missing values per row of the target table by performing calculation with the bits of the null bitmap 330. In the present example embodiment, the count part 120 counts the number of missing values in each extraction chunk table 420.

The calculation performed by the count part 120 is to subtract a sum of the bits of the extracted null bitmaps 330.b for each row from the number of columns in the extraction chunk table 420.

The split part 130 performs a split process. In the split process, the split part 130 splits (slices) the extraction table (320) or in the row direction to generate K chunk tables 410. In the present example embodiment, the split part 130 performs the split process on the extraction table 320, as illustrated in Fig. 5. The number of splits is stored in the storage part 180 in advance. The number of splits is predetermined in consideration of the performance of the data processing apparatus 100a. Instead of the number of splits, the number of rows in each chunk table 410 may be specified and stored therein.

<Outline of the number of missing values count process>
An outline of how to count the number of missing values row-wise in the present example embodiment with a concrete example using Fig. 5.

First, the limitation part 110 extracts only a null-col(s) in the input table 310 to generate the extraction table 320. If there are no null-col(s), it returns an array of size M (M is a natural number greater than or equal to 1 and the number of rows in the input table 310) filled with zeros and the processes terminates.

The split part 130 splits the extraction table 320 into K chunk tables 410 (411,412, and 413).

Then, the limitation part 110 performs again a limitation process and extracts a null-col(s) from chunk tables 410 to generate extraction chunk tables 420, respectively. If no columns have the missing values in the specific chunk table 410, such as the chunk table 412, the limitation part 110 returns an array of length L (L is a natural number greater than or equal to 1 and length of the chunk table 412) filled with zeros for that chunk.

If there are null-cols, such as the chunk tables 411 and 412, the limitation part 110 extracts only those columns from the chunk tables 410 to generate extraction chunk tables 420, respectively. Then, the count part 120 inspects its respective null bitmap columns to calculate the row-wise sum of the number of non-missing (non-null) values (sum:non-null) 331 of each extraction chunk table 420, subtracts the result from the number of extracted columns to obtain the row-wise sum of the number of missing (null) values (sum:null) 332. These processes can be performed in parallel.

<Missing values count operation>
The flow of a missing value count operation performed by the data processing apparatus 100a according to the present example embodiment is described. Fig. 7(a) is a flowchart illustrating an example of the missing value count operation performed by the data processing apparatus 100a. This operation is initiated when the input table 310 is given and/or an instruction to start the operation is received from a user as the first example embodiment. Here, the input table 310 has M rows and N columns, and K chunk tables 410 are generated therefrom, where M, N, and K are natural numbers greater than or equal to 1.

The limitation part 110 specifies a null-col(s) in the input table 310 in a manner described above and counts the number of null-col(s) (#null-col) (Step S2101). The number of missing values of each column obtained during calculation can be, for example, stored in the storage part 180 corresponding to the column.

The limitation part 110 determines whether the #null-col is greater than 0 (Step S2102). If the #null-col is 0 (Step S2102; No), the limitation part 110 returns an array of zeros of size M (Step S2103) and terminates the missing value count operation.

If the #null-col is greater than 0 (Step S2102; Yes), the limitation part 110 performs a limitation process (Step S2104). In the limitation process, the limitation part 110 extracts the column(s) specified as the null-col to generate the extraction table 320.

Then, the split part 130 performs a split process (Step S2105). In the split process, the split part 130 slices the extraction table 320 in the row direction to generate K chunk tables 410.

Then, the limitation part 110 and the count part 120 perform a chunk operation on each chunk table 410 (Step S2106). A detail of the chunk operation is described below.

The count part 120 creates an array combining count results from each chunk table 410 in the order before splicing thereof (Step S2107) and terminates the missing value count operation.

<Chunk operation>
The flow of the chunk operation of Step S2106 of the missing value count operation is illustrated in Fig. 7(b). As illustrated in Fig. 7(b), in the chunk operation the limitation part 110 and the count part 120 perform the following processes (Steps S2202 to S2206) for each of the K chunk tables 410 (Step S2201, S2207, and S2208). Note that k is a counter.

The limitation part 110 specifies a null-col(s) in the current chunk table 410 and counts the #null-col (Step S2202).

The limitation part 110 determines whether the #null-col is greater than 0 (Step S2203). If the #null-col is 0 (Step S2203; No), the limitation part 110 returns an array of zeros with the size of the number of rows in the current chunk table 410 (Step S2204) and proceeds to Step S2207 below.

For example, in the example illustrated in Fig. 5, since the chunk table 412 (number of rows L) has no missing values in each column, the #null-col thereof is 0. Therefore, the limitation part 110 returns an array of zeros of size L as a count result for the chunk table 410.

If the #null-col is greater than 0 (Step S2203; Yes), the limitation part 110 performs a limitation process (Step S2205). In the limitation process, the limitation part 110 extracts the column(s) specified as the null-col(s) to generate the extraction chunk table 420.

For example, in the example illustrated in Fig. 5, the #null-col of each of the chunk tables 411 and 413 is 2 (greater than 0). The limitation part 110 specifies columns c_3 and c_4 in the chunk table 411 as the null-cols and extracts them to generate the extraction chunk table 421. The limitation part 110 also specifies columns c_4 and c_5 in the chunk table 413 as the null-cols and extracts them to generate the extraction chunk table 423.

The count part 120 performs a count process (Step S2206). In the count process the count part 120 performs a calculation with the bits of the null bitmap 330 and obtains the number of the missing values per row of each of the extraction chunk tables 420. The count part 120 returns results obtained as the count result for each of the corresponding chunk table 410.

For example, in the example illustrated in Fig. 5, the count part 120 calculates the sum of the bits of the null bitmap 330 in the extraction chunk table 421 in the row direction as the sum:non-null, and subtracts it from the number of columns in the extraction chunk table 421 to obtain the result. The result is then output as the count result of the chunk table 411. The same is performed for the extraction chunk table 423.

The data processing apparatus 100a performs the above processes for all chunk tables 410 (Steps S2207 and S 2208) and terminates the chunk operation.

The chunk operation may be performed in parallel for each of the chunk tables 410.

As explained above, the data processing apparatus 100a, according to the present example embodiment, includes the same configuration as the data processing apparatus 100, according to the first example embodiment. Therefore, the same effects as the first example embodiment are obtained.

In addition, the data processing apparatus 100a, according to the present example embodiment, slices the input table 310 into a plurality of chunk tables 410 and performs processes on each chunk table 410. This allows, for example, the calculation to be omitted for a chunk table 410 that has no missing values in any rows, such as the chunk table 412 illustrated in Fig. 5. Thus, it further increases the possibility of reducing wasteful calculations. This further reduces computation time.

In addition, according to the data processing apparatus 100a, the computation time can be reduced by performing processes for each chunk table 410 in parallel.

Therefore, according to the data processing apparatus 100a of the present example embodiment, it is possible to contribute to optimizing data cleaning prior to data analysis.

<Variation Example 1>
In the second example embodiment, the limitation process is first performed on the input table 310, followed by the split process. However, the order of performing these processes is not limited thereto.

For example, as illustrated in Fig. 8, the split part 130 may first slice the input table 310 in the row direction to generate K (three) chunk tables 410. Then, the limitation part 110 may generate extraction chunk tables 420 from K chunk tables 410, respectively.

In the variation example 1, the split part 130 performs the split process on the input table 310 as a target table. The limitation part 110 performs the limitation process on the chunk tables 410 as target tables to generate the extraction chunk tables 420, respectively.

<Variation Example 2>
In the above example embodiment and the variation example, there are no restrictions on the number of chunk tables 410 generated and the number of rows in each chunk table 410. However, it is preferable to make the number of rows in each chunk table 410 approximately equal. This allows, for example, to roughly equalize each processing time when performing the chunk operation in parallel, thereby reducing the waiting time.

<Variation Example 3>
As a part of data cleaning, rows/columns of a data that contain a significant number of missing values are often dropped. To achieve this, as illustrated in Fig. 6(b), the data processing apparatus 100b may further be provided with a decision part 140 and a drop part 150 in addition to the configuration of the data processing apparatus 100a.

The decision part 140 performs a decision process. In the decision process, the decision part 140 determines a row to be dropped as a drop row according to the number of missing values per row and a predetermined drop option. The number of missing values is counted by the count part 120. Concretely, the decision part 140 generates a mask vector 501 that masks the determined drop row(s) 502, as illustrated in Fig. 9. Drop options are described below. The drop option to be used is predetermined and stored in the storage part 180.

The drop part 150 drops the drop row(s) 502 which is determined by the decision part 140 to generate a processed table 510. For example, the drop part 150 performs a filter process that applies the mask vector 501 to the input table 310 to obtain the processed table 510.

The drop options are described hereinafter. Usually there are three drop options for dropping a row having the missing value(s). They are a threshold-approach, an all-approach and an any-approach.

The threshold-approach: when the number of non-missing (valid) values of a given row is less than a given threshold value, the row is determined as the drop row 502. The all-approach: when all values of a given row are missing, the row is determined as the drop row 502. The any-approach: when any of the values of a given row is missing, the row is determined as the drop row 502.

Examples of processed tables 510 for each of the drop options obtained from the input table 310 are illustrated in Fig. 10. Here, "#non-null" indicates the number of non-missing values, and "#null" indicates the number of missing values.

The processed table 510_TH is a result of determining the drop rows 502 according to the threshold-approach. Here, the threshold value (TH) used in the threshold-approach is 4. The processed table 510_ALL is a result of determining the drop row 502 according to the all-approach. The processed table 510_ANY is a result of determining the drop rows 502 according to the any-approach.

According to the variation example 3, the data processing apparatus 100 can clean the input table 310 without a large amount of memory and with a low computational cost.

<<Third Example Embodiment>>
A data processing apparatus, according to the present example embodiment, can further speed up when dropping a row(s) having one or more missing values.

When dropping the row(s) with one or more missing values to clean the input table 310, the data processing apparatus of the present example embodiment basically counts the number of missing values in each row and decide whether or not to drop it according to the result thereof as in the variation example 3.

In the present example embodiment, quick cases are prepared for each drop option. In each quick case, a quick requirement and a quick process are defined. When the input table 310 satisfies the quick requirement, the processed table 510 is generated by the quick process. This allows for further reduction of wasteful calculations.

The functional block of the data processing apparatus 100c according to the present example embodiment is illustrated in Fig. 11. As illustrated in Fig. 11, the data processing apparatus 100c is further provided with a quick part 160 to the configuration of the data processing apparatus 100b illustrated in Fig. 6(b).

The quick part 160 determines whether a target table satisfies the predetermined quick requirement. If the target table satisfies the quick requirement, it performs a quick process predetermined for each quick requirement to generate the processed table 510, instead of performing the limitation process and the count process described above.

In the present example embodiment, when performing a drop operation, the quick part 160 determines whether the target table satisfies the quick requirement before performing the processes in each of the above process steps. The quick part 160 determines it using bits of the null bitmap 330 of the target table.

The quick requirement and the quick process for each drop option are stored in the storage part 180 in advance. The drop option to be used is specified by the user or set in advance.

Other functions and other processes are the same as in each of the above example embodiments and/or the variation examples. Therefore, the following explanation will focus on the differences.

<Quick Cases>
For each drop option, a quick case provided therefore is described below.

<Threshold-approach>
A quick case for the input table 310 when the drop option is the threshold-approach is described. As illustrated in Fig. 12(a), the quick requirement thereof is that there exists a sufficient number (as per the given threshold) of non-null columns (columns having no null values). The number of non-null columns (#non-null_col) can be obtained by subtracting the number of null columns (#null_col) from the total number of columns (#col) in the input table 310. #null_col can be obtained by checking how many columns in input table 310 have null_count > 0. The null_count of each column of input table 310 can be obtained by summing up the null bitmap of that column as explained previously.
When the quick requirement is satisfied, the quick process is to return the input table 310 as it is. That is, the quick part 160 makes the input table 310 the processed table 510.

For example, in the example illustrated in Fig. 12(b), the number of columns (#col) of the input table 310 is 10. The number of columns having only non-missing values (#non-null_col) thereof is 5. These are the columns of c_1, c_2, c_3, c_6, and c_7, which are not shaded in the drawing. Therefore, for all the threshold values (TH) greater than or equal to the value of (#col) - (#non-null_col), the quick part 160 makes the input table 310, as it is, the processed table 510.

When the input table 310 is given, the quick part 160 obtains information on the number of columns (#col) and the number of rows (#row) thereof. These are obtained, for example, from an attribute information of the input table 310.

The quick part 160 counts the number of columns having only non-missing values (#non-null_col) using a column-wise sum of bits of the null bitmap 330 of the input table 310. As illustrated in Fig. 12(c), the quick part 160 calculates the number of non-null columns (#non-null_col) in an input table by checking the total number of columns having null_count > 0. The null_count of a column can be counted by summing the bits of the null bitmap of that column, which is kept as metadata for each column, as explained above.
The quick part 160 performs this process for all columns.

The quick part 160 may, for example, multiply the bits of a given column to determine whether the column is a column having only non-missing values. That is, if the result of the multiplication is 1, the quick part 160 determines that the column has only non-missing values.

The same applies when the target table is another table such as the chunk tables 410. The quick part 160 subtracts the number of columns (#ext_col) extracted when generating each of the extraction chunk tables 420 from the number of columns (#col) of the input table 310. If the value obtained by the subtraction is greater than or equal to the threshold value (TH), then the quick part 160 returns the chunk table 410 as it is.

A concrete example is illustrated in Fig. 13. Here, the limitation part 110 generates an extraction table 320 with 4 columns from the input table 310 with 10 columns in the manner described above, as illustrated in Fig. 13. Then the limitation part 110 and the split part 130 generate four extraction chunk tables 421, 422, 423, and 424 by slicing the extraction table 320 into four and performing the limitation process.

For the extraction chunk table 421, (#col) - (#ext_col) is 7, since the number of extracted columns (#ext_col) is 3. For the extraction chunk table 422, (#col) - (#ext_col) is 9, since the number of extracted columns (#ext_col) is 1. For the extraction chunk table 423, (#col) - (#ext_col) is 8, since the number of extracted columns (#ext_col) is 2. For the extraction chunk table 424, (#col) - (#ext_col) is 7, since the number of extracted columns (#ext_col) is 3.

When the threshold value TH is 8, for example, the quick part 160 determines there are no target rows to be dropped and return as they are for the extraction chunk tables 422 and 423 because their (#col) - (#ext_col) are greater than or equal to TH.

For the extraction chunk tables 421 and 424, of which their (#col) - (#ext_col) are less than TH, the count part 120 counts the number of row-wise missing values in the manner described above. The decision part 140 may calculate an adjusted threshold value (THa), which is obtained by subtracting the number of columns not extracted ((#col) - (#ext_col)) from the threshold value TH. That is, the count part 120 counts the number of missing values for each row of the extraction chunk table 420. If the result of subtracting the count result from the number of columns in the extraction chunk table 420 is less than the adjusted threshold value (THa), the decision part 140 determines the row to be a drop row.

In the input table 310 or the extraction table 320, the decision part 140 may determine the drop row(s) 502 by computing the bits of the null bitmap 330.

As illustrated in Fig. 14, in the extraction table 320 (or in the input table 310), the row-wise number of non-missing values (#non-null) is the sum of the bits of the null bitmap 330 for each row. Therefore, in the case of the threshold-approach, the decision part 140 may compare the result of summing row-wise the bits of the null bitmap 330 in the extraction table 320, with the threshold value TH to determine if it is a drop row or not, instead of counting in the manner described in each of the above example embodiments or variation examples.

<All-approach>
A quick case for the input table 310 when the drop option is the all-approach is described. As illustrated in Fig. 15(a), the quick requirement thereof is that there exists at least one column without any missing values, i.e., #non-null_col > 0. When there exists at least one non-null column, all-approach is failed to drop any row and the quick process thereof is to return the input table 310 as it is. That is, the quick part 160 makes the input table 310 the processed table 510.

When the input table 310 is given, the quick part 160 obtains information on the number of columns (#col) and the number of rows (#row) thereof in the manner above. The quick part 160 counts the number of columns having only non-missing values (#non-null_col) using a column-wise sum of bits of the null bitmap 330 of the input table 310. The quick part 160 calculates the number of non-null columns (#non-null_col) in an input table by checking the total number of columns having null_count > 0. The null_count of a column can be counted by summing the bits of the null bitmap of that column, which is kept as metadata for each column, as explained above. The quick part 160 performs this process for all columns.

For example, in the example illustrated in Fig. 15(b), the number of columns (#col) of the input table 310 is 10. The number of columns having the missing value(s) (#null_col) thereof is 5. These are the columns of c_4, c_5, c_8, c_9, and c_10, which are shaded in the drawing. In the input table 310, values of #col and #null_col are not equal, therefore, the quick part 160 determines that the input table 310 satisfies the quick requirement and makes the input table 310, as it is, the processed table 510.

For example, in the example illustrated in Fig. 15(c), the number of columns (#col) of the input table 310 is 10. The number of columns having the missing value(s) (#null_col) thereof is 10. These are shaded in the drawing. In the input table 310, values of #col and #null_col are equal, therefore, the quick part 160 determines that the input table 310 does not satisfy the quick requirement and instructs other functions to proceed usual process.

In the input table 310 or the extraction table 320, the decision part 140 may determine the drop row(s) 502 by computing the bits of the null bitmap 330, even in the all-approach.

As illustrated in Fig. 16, in the extraction table 320, a row, of which the result of the row-wise OR operation of the bits of the null bitmap 330 is 0, is a row in which all values are missing. The OR operation is an operation in which the result thereof for a row is 0 if all bits in the row are 0, and 1 otherwise.

In the case of the all-approach, the quick part 160 may perform a row-wise OR operation of bits of the null bitmap 330 in the input table 310 or the extraction table 320, and the decision part 140 may determine that a row, whose result thereof is 0, is the drop row 502, instead of counting in the manner described in each of the above example embodiments or variation examples.

<Any-approach>
A quick case for the input table 310 when the drop option is the any-approach is described. As illustrated in Fig. 17(a), the quick requirement thereof is that the input table 310 has a column in which all values are missing values. The quick process thereof is to return an empty table. That is, the quick part 160 drops all rows in the input table 310 and makes the table the processed table 510.

For columns where all values are missing values (i.e., all values are missing), the column-wise sum of the bits of the null bitmap 330 is also 0. When the input table 310 is given, the quick part 160 calculates the sum of bits of the null bitmap 330 for each column. As a result, if there is a column in which a result of the sum is 0, it determines that the column is a column where all values are missing values. It then determines that the input table 310 satisfies the quick requirement.

For example, in the example illustrated in Fig. 17(b), the sum of the bits of the null bitmap 330 in column c_10 of the input table 310 is 0. Therefore, the quick part 160 determines that the input table 310 satisfies the quick requirement and proceeds the quick process. On the other hand, in the example illustrated in Fig. 17(c), there is no column in the input table 310 where the sum of bits of the null bitmap 330 is 0. Therefore, the quick part 160 determines that the input table 310 does not satisfy the quick requirement and instructs other functions to proceed usual process.

In the input table 310 or the extraction table 320, the decision part 140 may determine the drop row(s) 502 by computing the bits of the null bitmap 330, even in the any-approach.

As illustrated in Fig. 18, in the extraction table 320, a row, where the result of the row-wise AND operation of the bits of the null bitmap 330 is 0, is a row that has at least one missing value therein. The AND operation is an operation in which the result thereof for a row is 1 if all bits in the row are 1, and 0 otherwise.

In the case of the any-approach, the quick part 160 may perform a row-wise AND operation of bits of the null bitmap 330 in the input table 310 or the extraction table 320, and the decision part 140 may determine that a row, whose result thereof is 0, is the drop row 502, instead of counting in the manner described in each of the above example embodiments or variation examples.

<Missing values drop operation>
The flow of a missing value drop process performed by the data processing apparatus 100c according to the present example embodiment is described. Fig. 19(a) is a flowchart illustrating an example of the missing value drop process performed by the data processing apparatus 100c. This process is initiated when the input table 310 is given and/or an instruction to start the process is received from a user as the first or second example embodiment. Here, the input table 310 has M rows and N columns, and K chunk tables 410 are generated therefrom, where M, N, and K are natural numbers greater than or equal to 1. The drop option is determined in advance.

The quick part 160 determines whether the quick case is triggered on the input table 310 (Step S3101). The quick part 160 determines whether the input table 310 satisfies the quick requirement for the predetermined drop option. If it satisfies the quick requirement, the quick part 160 generates the mask vector 501 according to the quick process associated with the quick requirement, return thereof (Step S3102), and terminates the missing value drop operation.

If it does not satisfy the quick requirement, the limitation part 110 performs a limitation process (Step S 3103). In the limitation process, the limitation part 110 extracts a null-col(s) from the input table 310 to generate an extraction table 320.

The split part 130 performs a split process (Step S 3104). In the split process, the split part generates K chunk tables 410 by slicing the input table 310 in the row direction.

The limitation part 110, the count part 120, the decision part 140, and the quick part 160 perform a chunk operation on each chunk table 410 (Step S3105). A detail of the chunk process is described below.

The decision part 140 combines chunk mask vectors 503 obtained by the chunk process, which mask the drop row(s) 502 of each chunk table 410, to obtain the mask vector 501 (Step 3106).

The drop part 150 performs a filter process on the input table 310 using the mask vector 501 (step S3107) to obtain the processed table 510 with the drop row(s) 502 dropped and terminates the missing value drop operation.

<Chunk operation>
The flow of the chunk operation of Step S3105 in the missing value drop operation is illustrated in Fig. 19(b). As illustrated in Fig. 19(b), in the chunk operation, the data processing apparatus 100c performs following processes (Steps S3202 to S3205) for each of the K chunk tables 410 (Step S 3201, S3206 and S3207). Note that k is a counter.

The quick part 160 determines whether the quick case is triggered on the current chunk table 410 (Step S3202). The quick part 160 determines whether the current chunk table 410 satisfies the quick requirement for the drop option. If it satisfies the quick requirement, the quick part 160 generates the chunk mask vectors 503 according to the quick process associated with the quick requirement, return thereof (Step S3203) and proceeds to Step S3206 below.

If it does not satisfy the quick requirement, the limitation part 110 performs a limitation process (Step S3204). In the limitation process, the limitation part 110 extracts a null-col(s) from the current chunk table 410 to generate the extraction chunk table 420.

The decision part 140 performs a decision process (Step S3205). In the decision process, the decision part 140 determines the drop row(s) 502 in the extraction chunk table 420 according to the predetermined drop option, to generate the chunk mask vectors 503. For example, if the predetermined drop option is the all-approach or the any-approach, the drop row(s) 502 may be determined by the bits calculation by the quick part 160 above. If it is the threshold-approach, the drop row(s) 502 may be determined by determining the adjusted threshold value in the manner described above.

The data processing apparatus 100c performs the above processes for all chunk tables 410 (Steps S3206 and S3207) and terminates the chunk process.

As explained above, the data processing apparatus 100c, according to the present example embodiment, includes the same configuration as the data processing apparatuses 100, 100a, and 100b. Therefore, the same effects as the above example embodiments and variation examples are obtained.

In addition, according to the data processing apparatus 100c, when a target table falls under one of the predetermined quick cases, the count process, etc., that is normally processed is avoided. Thus, higher speed can be achieved. As a result, it reduces the amount of computational memory to be allocated during computation and reduces the computation time. Then, it contributes to the optimization of data cleaning prior to data analysis.

<<Hardware Configuration>>
The data processing apparatuses 100, 100a, 100b, and 100c are realized by a so-called general-purpose information processing apparatus (computer) and each have, for example, a configuration as exemplified in Fig. 20. Hereinafter, the data processing apparatuses 100, 100a, 100b, and 100c are represented by the data processing apparatus 100.

The data processing apparatus 100 is, for example, provided with a CPU (Central Processing Unit) 591, a main storage device (memory) 592, an auxiliary storage device 593, a communication I/F (interface) 594, and an extension I/F 595 which are interconnected by an internal bus as illustrated in Fig. 20.

The CPU 591, for example, loads the program stored in the auxiliary storage device 593 into the main storage device 592 and executes it to realize each of the above functions and to control the entire data processing apparatus 100. One or more processors such as a micro processing unit (MPU) may be used instead of the CPU 591.

The main storage device 592 is a memory such as RAM (Random Access Memory). The main storage device 592 is a work area for the CPU 591 to process the programs, etc. executed by the data processing apparatus 100.

The auxiliary storage device 593 stores various programs executed by the data processing apparatus 100. The auxiliary storage device 593 also stores various data. The storage part 180 can be configured on the auxiliary storage device 593.

The above functions of the data processing apparatus 100 are realized by the CPU 591 loading and executing the program stored in the auxiliary storage device 593 into the main storage device 592. Data generated during processing is stored in the main storage device 592 or the auxiliary storage device 593.

The auxiliary storage device 593 has, for example, a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc. Auxiliary storage device 593 may have a storage medium such as a flexible disk, hard disk, optical disk, CD-ROM, CD-R, magnetic tape, non-volatile memory card, DVD, etc.

The program stored in the auxiliary storage device 593 can be provided as a program product recorded on a non-transitory computer-readable storage medium. The auxiliary storage device 593 can be used to store various programs recorded in the non-transitory computer readable storage medium in mid and long term.

The communication I/F 594 transmits and receives signals and data by wired or wireless means. The data processing apparatus 100 may be provided with, for example, a NIC (Network Interface Card) as the communication I/F 594.

The extension I/F 595 is an interface that connects an output device such as a display device, and an input device. The display device is, for example, an LCD monitor. The input device is a device that accepts user operations, such as a keyboard or mouse, for example. In the above example embodiments and/or variation examples, for example, the input table 310 and the drop option selected may be accepted via the input device, and the processed table 510 may be displayed on the display device.

It should be noted that the hardware configuration illustrated in Fig. 20 is not intended to limit the hardware configuration of the data processing apparatus 100. The data processing apparatus 100 may include a hardware(s) not illustrated in Fig. 20.

It is to be noted that programs which realize each function of the data processing apparatus 100 can be recorded on a computer-readable storage medium. The storage medium can be a non-transitory one, such as a semiconductor memory, a hard disk, a magnetic recording medium, an optical recording medium, and so on. The present invention can be implemented as a computer program product.

Although, in a plurality of flowcharts used in the above description, a plurality of processes is described in sequence, an execution order of processes performed in each example embodiment and variation examples is not limited by an order of description thereof. In each example embodiment and variation examples, it is possible to change an order of processes illustrated in the drawings within an extent that will not offer any problem to content, for example, executing each processing in parallel and so on.

A part or a whole of the above-mentioned example embodiments and/or variation examples may be described as, but not limited to, the following supplementary notes.
<Supplementary Note 1>
A method for optimizing data processing of an input table includes:
-- performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
-- performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
-- the limitation process is performed with the input table as the first target table.
<Supplementary Note 2>
The method for optimizing data processing of the input table described in supplementary note 1, preferably further includes:
-- performing a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction, wherein
-- the split process is performed with the input table as the second target table,
-- the limitation process is performed with each chunk table, instead of the input table, as the first target table, and
-- the count process is performed for each extraction table generated.
<Supplementary Note 3>
The method for optimizing data processing of the input table described in supplementary note 1, preferably further includes:
-- performing a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction; and
-- performing a second limitation process in which an extraction chunk table is generated from each chunk table using the bits of the null bitmap by extracting a column(s) having at least one missing values, wherein
-- the count process is performed for each extraction chunk table, instead of the extraction table.
<Supplementary Note 4>
The method for optimizing data processing of the input table described in any one of supplementary notes 1 to 3, preferably further includes:
-- performing a decision process in which a row to be dropped is determined as a drop row according to the number of missing values per row and a drop option predetermined, and
-- generating a processed table by dropping the drop row.
<Supplementary Note 5>
The method for optimizing data processing of the input table described in supplementary note 4, preferably further includes:
-- determining whether the input table satisfies quick requirements predetermined,
-- performing a quick process predetermined for each quick requirement if the input table satisfies one of the quick requirements, instead of performing the limitation process and the count process, to generate a processed table.
<Supplementary Note 6>
In the method for optimizing data processing of the input table described in supplementary note 5,
-- in a case that the drop option is a Threshold-approach that a row, for which the number of non-missing values is less than a threshold value predetermined, is determined as the drop row,
-- the quick requirement may be that a value subtracting the number of columns having missing values from a total number of columns of the input table is greater than or equal to the threshold value, and
-- the quick process may be to return the input table as the processed table.
<Supplementary Note 7>
In the method for optimizing data processing of the input table described in supplementary note 5,
-- in a case that the drop option is an All-approach that a row in which all values are missing values is determined as the drop row,
-- the quick requirement may be that there exists at least one column without any missing values in the input table, and
-- the quick process may be to return the input table as the processed table, and
-- the method preferably further includes:
-- if the input data does not satisfy the quick requirement, performing an OR operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the OR operation is 0 as the drop row, in the decision process.
<Supplementary Note 8>
In the method for optimizing data processing of the input table described in supplementary note 5,
-- in a case that the drop option is an ANY approach that a row having a single missing value is determined as the drop row,
-- the quick requirement may be that the input table has a column that are all missing values, and
-- the quick process may be to return an empty table as the processed table, the empty table having zero rows and N columns (N is the number of columns in the input table), and
-- the method preferably further includes:
-- if the input data does not satisfy the quick requirement, performing an AND operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the AND operation is 0 as the drop row, in the decision process.
<Supplementary Note 9>
A data processing apparatus for processing data in an input table, includes:
-- a limitation part that performs a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
-- a count part that performs a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
-- the limitation part performs the limitation process with the input table as the first target table.
<Supplementary Note 10>
A program causing a computer to execute:
-- performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values the null bitmap being a metadata of each column in the first target table; and
--performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap.
<Supplementary Note 11>
In the method for optimizing data processing of the input table described in any one of supplementary notes 1 to 8,
-- the calculation may be to subtract a sum of the bits of the null bitmap for each row from the number of columns in the extraction table.
<Supplementary Note 12>
The data processing apparatus described in supplementary note 9, preferably further includes:
-- a split part that performs a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction, wherein
-- the split part may perform the split process with the input table as the second target table,
-- the limitation part may perform the limitation process with each chunk table, instead of the input table, as the first target table, and
-- the count part may perform the count process for each extraction table generated.
<Supplementary Note 13>
The data processing apparatus described in supplementary note 9, preferably further includes:
-- a split part that performs a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction, wherein
-- the split part may perform the split process with each chunk table as the second target table,
-- the limitation part may further perform with each chunk table as the first table, and
-- the count part may perform the count process for each chunk table generated.
<Supplementary Note 14>
The data processing apparatus described in any one of supplementary notes 9, 12 and 13, preferably further includes:
-- a decision part that performs a decision process in which a row to be dropped is determined as a drop row according to the number of missing values per row and a drop option predetermined, and
-- a dropping part that generates a processed table by dropping the drop row.
<Supplementary Note 15>
The data processing apparatus described in supplementary note 14, preferably further includes:
-- a quick part that determines whether the input table satisfies quick requirements predetermined and performs a quick process predetermined for each quick requirement if the input table satisfies one of the quick requirements, instead of performing the limitation process and the count process, to generate the processed table.
<Supplementary Note 16>
In the data processing apparatus described in supplementary note 15,
-- in a case that the drop option is a Threshold-approach that a row, for which the number of non-missing values is less than a threshold value predetermined, is determined as the drop row,
-- the quick requirement may be that a value subtracting the number of columns having missing values from a total number of columns of the input table is greater than or equal to the threshold value, and
-- the quick process may be to return the input table as the processed table.
<Supplementary Note 17>
In the data processing apparatus described in supplementary note 15,
-- in a case that the drop option is an All-approach that a row in which all values are missing values is determined as the drop row,
-- the quick requirement may be that there exists at least one column without any missing values in the input table, and
-- the quick process may be to return the input table as the processed table, and
-- the decision part may perform an OR operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the OR operation is 0 as the drop row, if the input data does not satisfy the quick requirement.
<Supplementary Note 18>
In the data processing apparatus described in supplementary note 15,
-- in a case that the drop option is an ANY approach that a row having a single missing value is determined as the drop row,
-- the quick requirement may be that the input table has a column that are all missing values, and
-- the quick process may be to return an empty table as the processed table, the empty table having zero rows and N columns (N is the number of columns in the input table), and
-- the decision part may perform an AND operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the AND operation is 0 as the drop row, if the input data does not satisfy the quick requirement.
- The above supplementary note 10 can be expanded in the same way as supplementary note 1 is expanded to supplementary notes 2 to 8.

It should be noted that, each disclosure of the NPL cited above is incorporated herein by reference thereto. It is to be noted that it is possible to modify or adjust the example embodiments or examples within the whole disclosure of the present invention (including the Claims) and based on the basic technical concept thereof. Further, it is possible to variously combine or select (or partially delete) a wide variety of the disclosed elements (including the individual elements of the individual claims, the individual elements of the individual example embodiments or examples, and the individual elements of the individual figures) within the scope of the whole disclosure of the present invention. That is, it is self-explanatory that the present invention includes any types of variations and modifications to be done by a skilled person according to the whole disclosure including the Claims, and the technical concept of the present invention. Particularly, any numerical ranges disclosed herein should be interpreted that any intermediate values or subranges falling within the disclosed ranges are also concretely disclosed even without specific recital thereof. For example, the configurations illustrated in the drawings have been used only as examples to facilitate understanding of the present invention. That is, the present invention is not limited to the configurations illustrated in the drawings.

reference numerals

100：data processing apparatuses
100a：data processing apparatus
100b：data processing apparatus
100c：data processing apparatus
110：limitation part
120：count part
130：split part
140：decision part
150：drop part
160：quick part
180：storage part
310：input table
320：extraction table
330：null bitmap
331：row-wise sum
332：array
410：chunk table
411：chunk table
412：chunk table
413：chunk table
420：extraction chunk table
421：extraction chunk table
422：extraction chunk table
423：extraction chunk table
424：extraction chunk table
501：mask vector
502：drop row
503：chunk mask vectors
510：processed table
510_TH：processed table
510_ALL：processed table
510_ANY：processed table
591：CPU
592：main storage device
593：auxiliary storage device
594：communication I/F
595：extension I/F

Claims (10)

1. A method for optimizing data processing of an input table, comprising:
   performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
   performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
   the limitation process is performed with the input table as the first target table.
2. The method for optimizing data processing of the input table according to claim 1, further comprising:
   performing a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction, wherein
   the split process is performed with the input table as the second target table,
   the limitation process is performed with each chunk table, instead of the input table, as the first target table, and
   the count process is performed for each extraction table generated.
3. The method for optimizing data processing of the input table according to claim 1, further comprising:
   performing a split process in which a predetermined number of chunk tables are generated by splitting a second target table in a row direction; and
   performing a second limitation process in which an extraction chunk table is generated from each chunk table using the bits of the null bitmap by extracting a column(s) having at least one missing values, wherein
   the count process is performed for each extraction chunk table, instead of the extraction table.
4. The method for optimizing data processing of the input table according to claim 1, further comprising:
   performing a decision process in which a row to be dropped is determined as a drop row according to the number of missing values per row and a drop option predetermined, and
   generating a processed table by dropping the drop row.
5. The method for optimizing data processing of the input table according to claim 4, further comprising:
   determining whether the input table satisfies quick requirements predetermined,
   performing a quick process predetermined for each quick requirement if the input table satisfies one of the quick requirements, instead of performing the limitation process and the count process, to generate a processed table.
6. The method for optimizing data processing of the input table according to claim 5, wherein,
   in a case that the drop option is a Threshold-approach that a row, for which the number of non-missing values is less than a threshold value predetermined, is determined as the drop row,
   the quick requirement is that a value subtracting the number of columns having missing values from a total number of columns of the input table is greater than or equal to the threshold value, and
   the quick process is to return the input table as the processed table.
7. The method for optimizing data processing of the input table according to claim 5, wherein,
   in a case that the drop option is an ALL approach that a row in which all values are missing values is determined as the drop row,
   the quick requirement is that there exists at least one column without any missing values in the input table, and
   the quick process is to return the input table as the processed table, and
   the method further comprising:
   if the input data does not satisfy the quick requirement, performing an OR operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the OR operation is 0 as the drop row, in the decision process.
8. The method for optimizing data processing of the input table according to claim 5, wherein,
   in a case that the drop option is an ANY approach that a row having a single missing value is determined as the drop row,
   the quick requirement is that the input table has a column that are all missing values, and
   the quick process is to return an empty table as the processed table, the empty table having zero rows and N columns (N is the number of columns in the input table), and
   the method further comprising:
   if the input data does not satisfy the quick requirement, performing an AND operation in a row direction on bits of the null bitmap in the input data to decide the row of which a result of the AND operation is 0 as the drop row, in the decision process.
9. A data processing apparatus for processing data in an input table, comprising:
   a limitation part that performs a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
   a count part that performs a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap, wherein
   the limitation part performs the limitation process with the input table as the first target table.
10. A program causing a computer to execute:
    performing a limitation process in which an extraction table is generated from a first target table using bits of a null bitmap by extracting columns having one or more missing values, the null bitmap being a metadata of each column in the first target table; and
    performing a count process in the extraction table, in the count process the number of missing values per row of the extraction table being obtained by performing a calculation with the bits of the null bitmap.