KR102177792B1 - A system that displays large amounts of data on a chart without memory capacity limitation by using a binary file storage structure per column - Google Patents

Abstract

In the present invention, the data retrieved from the database is created as a column-based binary file and stored in a disk rather than a memory. When reading data stored on the disk, the entire data is not loaded into the memory at once, but is read in record units and charted. Because it is displayed in (Chart), a large amount of data is charted without memory capacity limitation by using a binary file storage structure for each column that does not cause a memory overflow error no matter how large the size of the data retrieved from the DBMS (Database Management System) is. It relates to the system represented by.
According to the present invention, a plurality of record index information corresponding to each coordinate on a chart expressed using a large amount of data retrieved from a DBMS is created as a binary file in column units and stored on a disk, and a specific location expressed on the chart When there is a request for record information indicated by the coordinates of the image, by applying a hash function structure that uses the position coordinates on the chart as the key value and the record index information as the value, it is possible to quickly find the record index information corresponding to the coordinates. In addition, since the information corresponding to the record index is retrieved and loaded from the column-level binary file stored on the disk after the record index information is retrieved, there is no memory overflow error, so that the chart can be displayed without limiting the memory capacity. There is this.

Description

A SYSTEM THAT DISPLAYS LARGE AMOUNTS OF DATA ON A CHART WITHOUT MEMORY CAPACITY LIMITATION BY USING A BINARY FILE STORAGE STRUCTURE PER COLUMN}

The present invention uses a binary file storage structure for each column to process large-capacity data received as a result of data inquiry in environments such as big data and cloud computing, without limitation of memory capacity, to a system that displays large-capacity data as a chart without memory capacity limitation. About.

When data is retrieved from the existing server database, the XML-type string is converted into a DataSet using API, saved in the web server, and then transmitted to the user's computer, or PHP or JSP files are converted to JSON using the API and transferred to the web server. After saving, transfer to user's computer (3 tier method). Alternatively, the data retrieved from the server database is directly transmitted to the user's computer, and the XML string in the user's computer is converted into a DataSet using API and saved, or a PHP or JSP file is changed to JSON using API and saved ( 2 tier method).

When data is retrieved from the database using such an existing method, the retrieved data is created in the form of a TEXT file stream in XML, JSON, or CSV format, and thus becomes larger than the byte size required to express the original value.

Eventually, when all of the data retrieved in this way is loaded into memory, more memory than the original size of the actual value is used.

Accordingly, a memory overflow error occurs due to a limit on the size of a memory area that can be accessed by a process according to the type of operating system (32bit, 64bit).

For reference, the total memory available in the 32bit operating system is 4G, and the total memory available in the 64-bit operating system is 8G ~ 2TB. In addition, the available memory for 32bit application programs is 2~3G, and the memory available for 64bit applications is 8TB, and up to 2G can be used as a single entity.

Referring to FIG. 1, a general data search method using an XML DataSet is shown. ① First, execute the query in the DBMS. ② Load the search result data in the middle tier (Web server) into the memory in the form of XML DataSet. In this process, if the memory capacity per process of the middle tier (Web server) is exceeded, a memory overflow error occurs. ③ Loads the query result set data received from the client into the memory in the form of XML DataSet. During this process, if the memory capacity per process of the client is exceeded, a memory overflow error occurs. ④ Finally, bind the XML DataSet in memory to the Chart control and display it as a chart. In the process of binding to the chart control, an additional memory overflow error occurs depending on the data binding method of the chart control.

As described above, when a large amount of data is searched in a conventional way in a big data and cloud computing environment, not only a memory overflow error occurs due to exceeding the memory capacity per process, but also the time it takes to search the data takes a long time.

In addition, in order to express a large amount of data in a chart, and after the chart is drawn, in order for the user to view detailed information about a specific position on the chart, the record corresponding to a specific position in the memory must be searched for. In case of managing in the system, there is a risk of memory overflow, and when sequentially searching for information having the corresponding location coordinates in the memory, it consumes a lot of CPU resources and takes a long time.

Registered Patent 10-1530441 (Column-based data processing method and device)

The present invention was invented to improve the above-described problems, and the data retrieved from the database is created as a column-based binary file and stored in a disk rather than a memory. When reading data stored on the disk, the entire data is saved. Because the data is not loaded into memory at once, it is read in units of records and displayed on the chart, so no matter how large the size of the data retrieved from the DBMS (Database Management System) is, a binary file for each column that does not cause a memory overflow error is saved. It is to provide a system for displaying large-capacity data in a chart without limitation of memory capacity using a structure.

In addition, without loading the large amount of original data retrieved from the DBMS (Database Management System) into memory, multiple record index information corresponding to each coordinate on the chart is created as a binary file in column units and stored on disk. When there is a request for record information indicated by the coordinates on a specific location expressed in, by applying a hash function structure that uses the location coordinates on the chart as the key value and record index information as the value, record index information corresponding to the coordinates It is to provide a system that displays large-capacity data as a chart without memory capacity limitation by using a binary file storage structure for each column that can quickly find.

In order to achieve the above object, the present invention is a column binary file storage unit that generates a table (Result Set) data received as a result of a data inquiry from a database management system (DBMS) as a column-based binary file and stores it on a disk. ; A column binary file reader configured to read cell values one by one from a binary file for each column stored on a disk by the column binary file storage unit, construct a record, and then load a memory in a record unit; And visually display in the form of pixels at positions corresponding to X and Y coordinates on the chart using cell values corresponding to X and Y coordinates in the record loaded into the memory by the column binary file reader. It characterized in that it comprises a chart display unit.

According to the present invention having the above configuration, the following effects can be achieved.

Memory overflow caused by managing the large amount of data retrieved from the DBMS in memory because the large amount of original data retrieved from the DBMS (Database Management System) is not loaded into memory, but is written as a column-unit binary file and stored on disk. It can block the risk of

In addition, a plurality of record index information corresponding to each coordinate on the chart expressed using the large amount of data retrieved from the DBMS is created as a binary file in column units and stored on disk, and the coordinates on a specific location expressed on the chart are indicated. When there is a request for record information, it is possible to quickly find the record index information corresponding to the coordinate by applying a hash function structure that uses the position coordinate on the chart as the key value and the record index information as the value. Since the index information is retrieved and the information corresponding to the record index is retrieved and loaded from the column-unit binary file stored on the disk, memory overflow errors do not occur, so that the chart can be displayed without limiting the memory capacity.

1 shows a general data search method using an XML DataSet according to the prior art.
FIG. 2 shows a method in which a binary file for each column is stored in a disk and then loaded into a memory in a record unit according to the present invention, thereby using little memory.
3 is a block diagram illustrating a function unit for storing large-capacity data searched in a database as a binary file for each column according to an embodiment of the present invention (first embodiment).
4 is a block diagram illustrating a function unit for storing large-capacity data searched from a database as a binary file for each column according to another embodiment of the present invention (second embodiment).
5 shows a description of a basic type that does not reduce a capacity and a first to fourth capacity reduction policy that reduces the capacity when generating a binary file for each column according to the present invention.
6 is a conceptual diagram of a binary file structure for each column that maintains the size of an original format without reducing capacity when a table (Result Set) is created as a column-based binary file and stored on disk according to an embodiment of the present invention to be.
7 illustrates the format of an original column according to a capacity reduction policy suitable for the characteristics of cell data values belonging to each column when a table (Result Set) is created as a column-based binary file and stored on disk according to another embodiment of the present invention. This is a conceptual diagram of the binary file structure for each column to be converted.
8 is a conceptual diagram illustrating a structure of a header generated when a binary file in column units is generated according to an embodiment of the present invention and another embodiment.
9 shows an example in which the size of a column after conversion is compared with the size of an original column before conversion when a capacity reduction policy is applied according to another embodiment of the present invention.
10 is a block diagram illustrating a function of a system for displaying a large amount of data in a chart without a memory capacity limitation using a binary file storage structure for each column according to the present invention.
11 is a conceptual diagram of a structure in which record index information corresponding to each coordinate displayed on a chart is generated as a column-based binary file and stored in a disk according to the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to embodiments to be described later in detail together with the accompanying drawings.

However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms.

In the present specification, the present embodiment is provided to complete the disclosure of the present invention, and to completely inform the scope of the invention to those of ordinary skill in the art to which the present invention pertains.

And the invention is only defined by the scope of the claims.

Accordingly, in some embodiments, well-known components, well-known operations, and well-known techniques have not been described in detail in order to avoid obscuring interpretation of the present invention.

In addition, throughout the specification, the same reference numerals refer to the same constituent elements, and terms used in the present specification (referred to) are for describing exemplary embodiments and not limiting the present invention.

In this specification, the singular form also includes the plural form unless specifically stated in the phrase, and the components and actions referred to as'include (or, have)' do not exclude the presence or addition of one or more other components and actions. .

Unless otherwise defined, all terms (including technical and scientific terms) used in the present specification may be used as meanings that can be commonly understood by those of ordinary skill in the art to which the present invention belongs.

In addition, terms defined in a commonly used dictionary are not interpreted ideally or excessively unless defined.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

In the present invention, a record has the same meaning as a row of a table, and a column has the same meaning as a column of a table. One column consists of one or more cells. Result Set refers to the result data (table) obtained by executing a query in the DBMS.

In the case of passing through an intermediate server between the upper server and the client (3 tier method) The structure for storing large-capacity data searched in the database according to the present invention as a binary file for each column is installed and operated not only in the intermediate server but also in the client. In this case, when the intermediate server receives the data search result from the DBMS, it creates a binary file for each column and stores it on the disk, and the client receives the binary file storage structure for each column received from the intermediate server and stores it on the disk.

In the case of consisting of only a server and a client (2-tier method), a structure for storing large-capacity data searched in a database according to the present invention as a binary file for each column is installed and operated in the client. In this case, when the client receives the data search result from the DBMS, it creates a binary file for each column and stores it on the disk.

10 and 11, the system 300 for displaying large-capacity data as a chart without memory capacity limitation using the binary file storage structure for each column according to the present invention is a column binary file storage unit 110 and column It includes binary file reader units 120 and 220 and a chart display unit 330.

The column binary file storage unit 110 and the column binary file reader unit 120 are described in detail in the following first embodiment, and the column binary file storage unit 210 and the column binary file reader unit 220 are described below. This is described in detail in the second embodiment.

The column binary file storage units 110 and 210 generate table data received as a result of a data search from a database management system (DBMS) as a binary file for each column and store it in a disk.

The column binary file reader units 120 and 220 read cell values one by one from the binary file for each column stored on the disk by the column binary file storage units 110 and 210, form a record, and then load the memory in units of records. do.

The chart display unit 330 uses each cell value corresponding to the X coordinate and the Y coordinate in the record loaded into the memory by the column binary file reader units 120 and 220 to position the X coordinate and the Y coordinate on the chart. Visually displayed in the form of pixels. Records are displayed in pixel form at positions corresponding to X and Y coordinates on the chart using values having the characteristics of X and Y coordinates among the columns of the record.

The chart display unit 330 reads all the inquired records or all records corresponding to the range designated by the user and displays them on the chart. Each pixel displayed on the chart represents a record.

Specifically, the chart display unit 330 includes a two-dimensional array index number management unit 331, an array structure conversion unit 332, a record index creation unit 333, and a record index reader unit 334.

For example, when the column binary file readers 120 and 220 read cell values one by one from the binary file for each column stored on the disk, compose records and load them into the memory in units of records, the chart display unit 330 will display 3 million records. When is sequentially displayed on the chart, each record is displayed at a different location on the two-dimensional coordinates, and a plurality of records may be overlapped and displayed in the same X and Y coordinates.

11 shows an example in which a total of 14 records are superimposed on coordinates 75 and 50. Previously, in order to check the records stamped at coordinates 75 and 50, it was necessary to sequentially search for values with an X coordinate of 75 and Y coordinate of 50 in a table. Not only is this a very time-consuming operation, but there has always been a risk of memory overflow because a large amount of data must be managed in memory for exploration.

Referring to FIG. 11, the two-dimensional array index number management unit 331 assigns a row index number to a location corresponding to the X coordinate and the Y coordinate on the chart in the order of visual display in the form of pixels. One or more record index numbers are stored in correspondence with each position of a pixel in a two-dimensional array corresponding to the X coordinate and the Y coordinate.

In the present invention, when the column binary file reader units 120 and 220 read from the disk in column units to compose records and then load them into the memory in record units, the two-dimensional array index number management unit 331 visually displays the order in pixel form. By assigning a record index number as described above, one or more record index numbers corresponding to each pixel position are managed.

The array structure conversion unit 332 converts a two-dimensional array into a one-dimensional array, converts the X coordinates and Y coordinates in the two-dimensional array to the index number of the one-dimensional array, Save the above record index number.

In FIG. 11, coordinates of 75 and 50 corresponding to a two-dimensional array are converted into a one-dimensional array, and a one-dimensional index number is given again. Record indexes for 14 records are stored in the index of the one-dimensional array corresponding to coordinates 75 and 50.

The record index creation unit 333 has the same address value indicating the start position in the record index data file ([imifilename_hash].imi.data) of the record index number corresponding to the index number of the one-dimensional array and the number of record index numbers. It is organized sequentially by belonging to one cell of the size, and after converting all cell data to binary values in turn, a record index indicator file ([imifilename_hash].imi.index) is created and stored on the disk, and is stored in one dimension. After converting the record index number corresponding to the index number of the array into a binary value sequentially in byte units corresponding to the specified size, a record index data file ([imifilename_hash].imi.data) is created and stored on the disk.

Referring to FIG. 11, for example, a record index indicator file ([imifilename_hash].imi.index) has a 4 byte header, and a record index data file ([imifilename_hash].imi.data) has a 2 byte header. heard. The 4 byte headers (Type(2), Size(2)) of the record index indicator file ([imifilename_hash].imi.index) indicate the start position in the record index data file ([imifilename_hash].imi.data). The format of the indicated address value and the format of the number of record index numbers are allocated and stored by 1 byte, respectively, and the size and record of the address value format indicating the start position in the record index data file ([imifilename_hash].imi.data). The size of the number format of the index number is allocated and stored by 1 byte each. The format of the record index number is stored in the 2-byte header (Type(2)) of the record index data file ([imifilename_hash].imi.data).

When designating a specific pixel on the chart, the record index reader 334 uses the index number of a one-dimensional array corresponding to the X coordinate and Y coordinate of the pixel, and the record index indicator file ([imifilename_hash].imi.index) stored in the disk. ) From the record index data file ([imifilename_hash].imi.data) of the record index number ([imifilename_hash].imi.data), calculate the address value indicating the starting position and the number of record index numbers in the record index data file ([imifilename_hash].imi.data). Gets the index number of the corresponding number of records.

In FIG. 11, when a user requests a record search by clicking 75 or 50 coordinates, a record index indicator file ([imifilename_hash].imi.index) using an index number of a one-dimensional array corresponding to the 75 or 50 coordinates ([imifilename_hash].imi.index) You can find the location of the corresponding cell in. In the record index indicator file ([imifilename_hash].imi.index) of FIG. 11, the address value 2 indicating the start position in the record index data file and 14 corresponding to the number of record index numbers are calculated in the first cell, and then record index data Get 14 record index numbers from the file ([imifilename_hash].imi.data).

11 shows an example in which 14 records are displayed at coordinates 75 and 50, 10 records are at coordinates 76 and 50, 5 records are at coordinates 77 and 50, and 11 records are displayed at coordinates 78 and 50.

The column binary file reader units 120 and 220 receive one or more record index numbers obtained from the record index reader unit 334 and read cell values corresponding to the record index numbers one by one from the column-unit binary file stored on the disk. Rows) are sequentially configured for all the received record index numbers and transferred to the chart display unit 330, and the chart display unit 330 displays record information corresponding to the record index number.

The column binary file storage units 110 and 210 and the column binary file reader units 120 and 220 will be described in detail in the first and second embodiments below.

[First embodiment]

The first embodiment is a structure in which the size of the original format is maintained when a table (Result Set) received as a result of data inquiry from a DBMS is stored as a binary file for each column.

2, 3, 5, 6, and 8, a structure 100 for storing large-capacity data searched in a database as a binary file for each column according to an embodiment of the present invention includes a column binary file storage unit 110 and It includes a column binary file reader 120.

The column binary file storage unit 110 is for a table (Result Set) received as a result of data search from a database management system (DBMS), and a numeric column is a corresponding column at the cell size interval of the original column type included in the table (Result Set). Converts all cell data of in order to binary values, creates a binary file for each column and stores it on disk, and the character column is the size corresponding to the number of characters in each cell included in the table (Result Set) After converting as many as binary values, it is created as one binary file for each column and saved on disk.

Numeric columns include big int, small int, int, and float, and character columns include Nvarchar, varchar, and char. In addition, various types of numeric and character columns are applicable.

The column binary file reader 120 reads cell values one by one from the binary file for each column stored on the disk by the column binary file storage unit 110, composes a row of a specific order, and then loads it into the memory in units of records. do. Users can make various requests and inquiries for large-capacity data stored on the disk, and when a user's request occurs, the memory overflow problem does not occur as the minimum memory resources can be used by loading into the memory in units of records. .

Specifically, the column binary file storage unit includes a fixed-length column binary storage unit 111 and a variable-length column binary storage unit 112.

The fixed-length column binary storage unit 111 converts all cell data in the corresponding column sequentially into binary values at the cell size interval of the original column type included in the numeric column table (Result Set), and then creates a single binary file. Save to disk.

Referring to FIG. 6, Col1 included in the table (Result Set) converts the data of each cell into binary values sequentially at 8 byte intervals corresponding to the big int type starting with 8 bytes of the header, and then creates the ColumnFile0001.conbin binary file. And save it to disk. Col2 converts the data of each cell into a binary value sequentially at 2-byte intervals corresponding to the small int type, starting with 8 bytes of the header, and then creates the ColumnFile0002.conbin binary file and stores it on the disk. Col3 converts to binary values sequentially at 4-byte intervals corresponding to int type starting with 8 bytes of header, and then creates ColumnFile0003.conbin binary file and stores them on disk.Col4 starts with 8 bytes of header and stores 8 corresponding to float type. After converting to binary values sequentially at byte intervals, it is created as a ColumnFile0004.conbin binary file and saved to disk. The binary file name of each column is provided as an example, and the column format of the original table is maintained.

The variable-length column binary storage unit 112 stores a size in bytes corresponding to the number of characters included in each cell of a character column and an address value indicating the start position of each cell in the cell data file. Consecutively configure each cell to belong to one cell, convert all cell data into binary values in turn, create a cell indicator file and store it on the disk, in byte units corresponding to the number of characters contained in each cell After converting to binary values sequentially as much as the size, cell data files are created and stored on disk.

Referring to FIG. 6, Col5 corresponding to a character column is composed of two binary files, ColumnFile0005.conbin corresponding to the cell indicator file and ColumnFile0005.conbin.data corresponding to the cell data file.

For example,'A…' for the second cell of Col5. In the case of C', the cell indicator file contains 15 characters and'A… The address value 28 indicating the position where C'starts is included in the second cell position. In the cell data file, the character string included in the original cell of the character column is continuously stored in bytes.

When'A..F' corresponding to the third cell position of a character column is converted to a binary value and saved, the third cell position of the cell indicator file is an address indicating the position where'A..F' starts in the cell data file. The value 43 and 5 corresponding to the number of characters are stored, and'A..F' is stored in the cell data file following the string at the second cell position.

In the same way as above, the ColumnFile0005.conbin binary file corresponding to the cell indicator file and the ColumnFile0005.conbin.data binary file corresponding to the cell data file are created and saved on the disk.

The fixed-length column binary storage unit 111 includes a fixed-length header configuration unit 111a.

Referring to FIG. 6, the fixed-length header configuration unit 111a generates a header for each column when generating a binary file for each column, and in the header, the original format of the numeric column defined in the table (Result Set), the binary file for each column Saves the file generation format and the size per cell when generating binary files for each column. In the example of FIG. 6, a case in which all the headers of the numeric column are configured identically with 8 bytes is shown.

At this time, the original format of the numeric column and the generation format of the binary file for each column have the same value, and when the binary file for each column is created, the size per cell is the same as the size of the original format.

In FIG. 6, the fixed-length header configuration unit 111a generates an 8-byte header for each column when generating binary files ColumnFile0001.conbin, ColumnFile0002.conbin, ColumnFile0003.conbin, and ColumnFile0004.conbin for each column.

Referring to FIG. 8, in the case of fixed length basic, each of Fetch Type is 2 bytes, P Type1, P Type2 is 1 byte, P Size1, P Size2 is 1 byte, and Reserved1 and Reserved2 are each 1 byte.

Here, Fetch Type represents the original format of the searched table (Result Set), P Type1 and P Type2 represent the column binary file creation format according to the present invention, and P Size1 and P Size2 represent the cell of the column binary file according to the present invention. It represents the size per unit (byte). Reserved1 and Reserved2 are reserved for further use.

The fixed-length column binary storage unit 111 converts the header stored by the fixed-length header construction unit 111a into a binary value, and converts cell data of the original column included in the table (Result Set) into a fixed-length header construction unit 111a. When creating a binary file for each column saved by ), it is converted into a binary value sequentially at intervals of size per cell, and it is created as a single binary file that combines the header and data for each column and stores it on disk.

In FIG. 6, Col1 to Col4 corresponding to the numeric type is composed of a fixed-length binary file, but Col5 corresponding to the character type is composed of a variable-length binary file as follows. In addition, one binary file is created for each original column in the numeric column, but two binary files consisting of a cell indicator file and a cell data file are created for each original column in the character column.

The variable-length column binary storage unit 112 includes a cell indicator header construction unit 112a and a cell data header construction unit 112b.

Referring to FIG. 6, ColumnFile0005.conbin represents a cell indicator file, and ColumnFile0005.conbin.data represents a cell data file.

The cell indicator header configuration unit 112a generates a header when generating a cell indicator file, and in the header, the original format of the character column defined in the table (Result Set), the cell indicator file generation format, and the cell indicator file are generated. Save the sugar size.

Referring to FIG. 8, an 8-byte First File Header generated by the variable-length basic cell indicator header configuration unit 112a is shown.

Fetch Type represents the original format of the retrieved Result Set, P Type1 and P Type2 represent the cell indicator file generation format, and P Size1 and P Size2 represent the size per cell generated by the cell indicator file.

Specifically, P Type1 represents an integer format representing the location of the data file, P Type2 represents the integer format of the number of characters to be read, P Size1 represents the size of the data file location integer format, and P Size2 represents the size of the integer format of the number of characters to read ( byte). Reserved1 and Reserved2 are reserved for further use.

6 shows an example in which each cell constituting the cell indicator file is composed of 6 bytes, and an integer indicating the location of the data file and the number of characters to be read of the data file are entered in the 6 byte space. Of course, the location of the data file contains the location value calculated from the 2-byte header of the data file.

The variable-length column binary storage unit 112 converts the header stored by the cell indicator header construction unit 112a into a binary value, and a size in bytes corresponding to the number of characters included in each cell of a character column and a cell data file When the cell indicator file stored by the cell indicator header configuration unit 112a is generated, the cell indicator header and the cell indicator data are combined in order by converting the address value indicating the start position of each cell in the image into a binary value in order of size per cell. It is created as a single binary file and saved to disk.

In the example of FIG. 6, the size in bytes corresponding to the number of characters included in each cell and the address value indicating the start position of the character in each cell in the cell data file are sequentially binary values at 6 byte intervals corresponding to the size per cell. To create a binary file in which the cell indicator header and the cell indicator data are combined and stored in the disk.

Referring to FIG. 6, the cell data header configuration unit 112b generates a header when generating a cell data file, and stores the original format of a character column defined in a table (Result Set) in the header.

Referring to FIG. 8, a 2 byte header structure generated by the cell data header configuration unit 112b is shown in which the variable length basic of the Second File Header. The header of the cell data file may be composed of 2 bytes or 4 bytes, and FIG. 6 shows an example composed of 2 bytes. The header generated by the cell data header configuration unit 112b includes the original format of the result set, and when a character column is returned from the cell data binary file stored on the disk, the original format of the result set included in the header is returned. .

The variable-length column binary storage unit 112 converts the header stored by the cell data header configuration unit 112b into a binary value, and stores each cell at a byte unit size interval corresponding to the number of characters included in each cell of the character column. The character is converted into a binary value, and the cell data header and cell data are combined into a single binary file and stored on the disk.

In the cell data file, the original string data contained in the data (Result Set) is sequentially converted into binary values, and the cell data after checking the starting position of each cell's character and the number of characters to be read in the cell indicator file discussed above. It reads as much as the number of characters in the file, converts it to the original format, and returns it.

The column binary file reader unit 120 includes a fixed-length column binary reader unit 121 and a variable-length column binary reader unit 122.

The fixed-length column binary reader 121 reads the Nth record value from the binary file of the numeric column stored on the disk (header size of the binary file + N × size per cell of the original column format defined in the header of the binary file) ), move to the Nth record position, and read as much as the cell size of the original column format.

Referring to FIG. 6, when reading the 3rd record value from ColumnFile0001.conbin corresponding to a numeric column (header size of binary file 8 byte + 2 × size per cell of original column format defined in header of binary file) ), after moving to the third record position, you can read as much as the cell size of the original column format.

For reference, the record index starts from 0, and the third record index value becomes 2.

When reading the Nth record value from the binary file of the character column stored on the disk, the variable-length column binary reader 122 reads (header size of the cell indicator file + N × (cell data file defined in the header of the binary file). Position Integer format size + number of characters to read Integer format size)), move to the Nth record position, read as much as that corresponding to the cell data file location integer format size, and find the starting address of the Nth record in the cell data file, and the character to read After reading as many as the size of the number integer format and obtaining the byte size corresponding to the number of characters to be read from the cell data file, the byte size corresponding to the number of characters to be read is read from the start address of the Nth record of the cell data file.

Referring to FIG. 6, when reading the third record value from a binary file of a character column (header size of the cell indicator file ColumnFile0005.conbin 8 byte + 2 × (Cell data file position constant defined in the header of the binary file) Calculate format size + number of characters to read integer format size)). In the example of FIG. 6, when the integer format size of the cell data file location defined in the header of the binary file and the integer format size of the number of characters to be read are added, it will be 6 bytes. 8 + 2 × 6 = 20 bytes.

From that location, read as much as the size of the cell data file location integer format, find '43', the starting address of the Nth record in the cell data file ColumnFile0005.conbin.data, and read as much as the number of characters to be read corresponding to the integer format size. After calculating the byte size of '5' corresponding to the number of characters to be read from the cell data file ColumnFile0005.conbin.data, the size of 5 bytes corresponding to the number of characters to be read is read from the start address of the Nth record of the cell data file.

[Second Embodiment]

In the second embodiment, when a table (Result Set) received as a result of a data inquiry from a DBMS is stored as a binary file for each column, a capacity reduction policy is applied to store it in a way that the capacity is reduced compared to the original column format. Applying a capacity reduction policy can significantly reduce the capacity stored on the disk. Detailed descriptions of parts that overlap with the case of the first embodiment will be omitted.

4, 5, 7, 8, and 9, a structure 200 for storing large-capacity data searched in a database as a binary file for each column according to another embodiment of the present invention includes a column binary file storage unit 210 and It includes a column binary file reader unit 220, and may further include a capacity reduction policy information unit 230.

The column binary file storage unit 210 provides each numeric column for the original column according to the characteristics of the cell data values belonging to each numeric column for a table (Result Set) received as a result of data search from a database management system (DBMS). By applying different capacity reduction conversion formats, all cell data in the corresponding column is sequentially converted into binary values at the cell size interval of the converted format, and then created as a single binary file for each column and stored on disk.

The column binary file reader 220 obtains a cell value one by one from the binary file for each column stored on the disk by the column binary file storage unit, constructs a record of a specific order, and loads it into the memory in a record unit.

The capacity reduction policy information unit 230 converts one or more numeric columns of a table (Result Set) received as a result of data inquiry into a binary file, and generates the characteristics of cell data values belonging to each column to reduce the size of the binary file. The appropriate capacity reduction policy is matched for each column and stored. The details of each capacity reduction policy are described in detail below.

The capacity reduction policy does not apply to character columns, but only to numeric columns. Here, the characteristics of cell data values belonging to a column refer to the column types such as big int, float, int, small int, etc., the degree of overlap of each cell value belonging to the column, and the relationship with the cell values belonging to other columns. A capacity reduction policy that fits the characteristics of the cell data value belonging to may be matched by the administrator.

The column binary file storage unit 210 extracts the cell data value for each column from the received table (Result Set) and refers to the capacity reduction policy information unit 230 to determine the original column of the table according to the capacity reduction policy matched to the column. Creates a binary file for each column whose capacity is smaller than that of the format and saves it on disk.

The capacity reduction policy information unit 230 is smaller than the original column type of the table (Result Set), but converts the type of the corresponding column into a column type of a designated size that can accommodate the maximum actual data value of the cell belonging to the corresponding column. And a first capacity reduction policy 231 to reduce capacity.

5, 7, 8, and 9, the format conversion corresponds to the first capacity reduction policy 231. The Col1 column format of the table (Result Set) is bigint (8 byte), but the actual maximum value is int (4 byte), so a type conversion is applied to reduce the capacity of a cell of 8 byte to 4 byte. It is to let. You can cut the original size in half and save it to disk.

The column binary file storage unit 210 includes a column format of the size converted by the first capacity reduction policy 231 corresponding to the original format of the corresponding column defined in the table (Result Set) and the generation format of binary files for each column, and And a first capacity reduction header configuration unit 211 that generates a header storing the size of the converted format.

The first capacity reduction header configuration unit 211 configures a header of 8 bytes, and stores the original type of the column defined in the Result Set in Fetch Type, stores the format conversion type in P Type, and converts it in P Size. The size of the format is saved, and reserved for future use. Fetch Type, P Type, P Size, and Reserved are each composed of 2 bytes.

The column binary file storage unit 210 converts the header generated by the first capacity reduction header configuration unit 211 into a binary value, and converts the cell data of the original column included in the table (Result Set) into a first capacity reduction policy. Depending on the column format and size of the size converted by (231), the size of the table is converted to binary values in order per cell size interval, which is smaller than the original column format of the table, and the header and data are combined into a single binary file to be created. Save it to.

When the original column type of the table (Result Set) is a floating-point number, the capacity reduction policy information unit 230 divides the data value of a cell belonging to the corresponding column into an integer part and a decimal part, and has a size that can accommodate the integer part. By designating the first type and designating a second type with a size that can accommodate a fractional part, the original column format is converted into a column type corresponding to the combined size of the first type and the second type. And a second capacity reduction policy 232 to reduce the capacity of the column.

5, 7, 8 and 9, the format complex conversion corresponds to the second capacity reduction policy 232. Although the Col2 column of the table (Result Set) is a float (8 bytes), the maximum value of the integer part and the decimal part (when the decimal part is converted to an integer) can be expressed in 2 bytes, so a total of 4 bytes is applied and saved to disk. It is possible to reduce the amount of data to be used. An integer is stored in a first type having a size that can accommodate an integer portion by dividing one cell space, and a decimal number is stored in a second type having a size that can accommodate the decimal portion.

The column binary file storage unit 210 is capable of accommodating an integer portion according to the second capacity reduction policy 232 corresponding to the original format of the corresponding column defined in the table (Result Set) and the generation format of binary files for each column. A second type that generates a header storing the size of the first type format, the size of the second type format that can accommodate a fractional part according to the second capacity reduction policy, the size of the first type format, and the size of the second type format. It includes a capacity reduction header configuration unit 212.

Referring to FIG. 8, an original format of a table (Result Set) is stored in a 2-byte Fetch Type, and a first type format having a size that can accommodate an integer part is stored in a 1-byte P Type1, and P Type2 stores the second type format with a size that can accommodate the fractional part, 1 byte P Size1 stores the size of the first type type, and 1 byte P Size2 stores the size of the second type format. And the precision of the decimal part is stored in Reserved1.

The column binary file storage unit 210 converts the header generated by the second capacity reduction header configuration unit 212 into a binary value, and converts the cell data of the original column included in the table (Result Set) into a second capacity reduction policy. Header and data are converted into binary values by converting the integer part and the decimal part of each cell of the column into binary values at the size interval per cell whose capacity is reduced compared to the original column format of the table according to the column format and size of the size converted by (232). It is created as a single binary file that is combined and saved to disk.

The capacity reduction policy information unit 230 forms a relationship that can be obtained by a calculation formula in which each cell data value belonging to an original column of a table (Result Set) is a cell data value at the same record (Row) position in another column as a variable. In this case, a third capacity reduction policy 233 for reducing the capacity of the original column by converting all of the cell data belonging to the original column into a single cell having a predetermined size and storing the calculation formula is included.

5, 7, 8, and 9, the conversion of the calculation formula corresponds to the third capacity reduction policy 232. The Col3 column format in the table (Result Set) is int (4 bytes), but the formula operation result of the Col5 column, Col3 = 2 * Col5 + 5, is established, so the {Col5} * 2 + 5 formula is converted to postfix notation to convert the calculation formula. Apply. Using such a calculation formula, the value of Col3 can be easily calculated by the calculation formula with Col5, so only the calculation formula is needed instead of the original column data of Col3.

In the example of FIG. 7, the size of an original column corresponding to 400 million bytes can be reduced to 10 bytes by storing only the expression {Col5} * 2 + 5 in a 10-byte space in addition to the 8-byte header by applying the calculation formula conversion.

The column binary file storage unit 210 is a third capacity that generates a header that stores the original format of a corresponding column defined in a table (Result Set), a binary file generation format for each column, and a size per cell when a binary file is generated for each column. It includes a reduced header component 213.

The column binary file storage unit 210 converts the header generated by the third capacity reduction header configuration unit 213 into a binary value, and a calculation formula defined in the third capacity reduction policy 233 (in the example of FIG. 7, { Col5} * 2 + 5) is converted into a binary value, and the header and data are combined into a single binary file and stored on the disk. The other columns and calculation formulas above are specified by the user.

The capacity reduction policy information unit 230 removes duplicates for values in which each cell data value belonging to the original column of a table (Result Set) is the same, so that the columnFile0005.conbin corresponds to a dictionary column composed of only different values (Distinct). Create a .Dictionary binary file, create a ColumnFile0005.conbin binary file that corresponds to the index column of the designated size that is smaller than the original column type of the table (Result Set) and stores the index on the dictionary column of the original cell data value. And a fourth capacity reduction policy 234 to reduce the capacity of the unit.

5, 7, 8, and 9, the column dictionary corresponds to the fourth capacity reduction policy 234. The Col4 column format of the table (Result Set) is int (4 bytes), but the number of values is less than 200 when distincted, so a dictionary file is created by removing duplicates of cell data, and the hash value (Index Key) of the dictionary is Apply Column Dictionary transformation that is saved as a colbin file.

Referring to FIG. 7, the first column 0 of ColumnFile0005.conbin indicates 36 of ColumnFile0005.conbin.Dictionary, the second column 1 indicates 42, and the third column 0 indicates 36 removed because it is a duplicate value. Fourth column 2 refers to 366 pulled forward one space after deduplication.

The column binary file storage unit 210 is an index column format and an index column designated by the fourth capacity reduction policy 234 corresponding to the original format of the corresponding column defined in the table (Result Set) and the generation format of binary files for each column. And a fourth capacity reduction index header configuration unit 2140a that generates a header storing the format size.

Referring to FIG. 8, a First File Header (8 bytes) represents a configuration of a header generated by the fourth capacity reduction index header configuration unit 2140a. The original type of data (Result Set) is stored in the fetch type, the Dictionay Key (Index) type is stored in the P Type, and the size of the Dictionay Key (Index) type is stored in the P Size.

The column binary file storage unit 210 converts the header generated by the fourth capacity reduction index header configuration unit 214a into a binary value, and the index column format and size of the size designated by the fourth capacity reduction policy 234 According to this, the index value on the dictionary column of the original cell data value is sequentially converted into a binary value, and the header and data are combined into a single index binary file and stored on disk.

The column binary file storage unit 210 includes a fourth capacity reduction dictionary header construction unit 214b that generates a header storing a dictionary column format and a size of the dictionary column format.

Referring to FIG. 8, the Second File Header (2 or 4 bytes) represents the configuration of a header generated by the fourth capacity reduction dictionary header configuration unit 214b. In the header, the format of the Dictionary value and the size of the Dictionary format are stored.

According to the fourth capacity reduction policy 234, duplicates are removed from the corresponding column defined in the table (Result Set), and only different values are sequentially configured, and the dictionary column is sequentially converted into binary values at intervals of size per cell. It is created as a single dictionary binary file that combines the data and stores it on disk.

The column binary file reader 220 interprets the capacity reduction policy for each column stored in the capacity reduction policy information unit 230 to obtain cell values one by one from the numeric column binary file stored on the disk, and the original stored in the header of the column binary file. Converts to a column format, composes a record of a specific sequence, and loads it into memory.

The column binary file reader 220 reads the P Type and P Size values stored in the header from the binary file for each column stored on the disk, checks the cell data stored on the disk, and converts it into the original column format corresponding to the fetch type, Records can be constructed and loaded into memory.

Until now, the present invention has been described in detail with reference to preferred embodiments, but those skilled in the art to which the present invention pertains can practice the present invention in other specific forms without changing the technical spirit or essential features thereof, The embodiments are illustrative in all respects and should be understood as non-limiting.

In addition, the scope of the present invention is specified by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts are included in the scope of the present invention. Should be interpreted as.

Structure storing 100, 200... as binary files for each column
110, 210...column binary file storage
111... fixed-length column binary storage
111a... fixed-length header components
112...variable length column binary storage
112a...cell indicator header configuration section
112b...cell data header configuration section
120, 220...column binary file reader
121...fixed-length column binary reader
122...variable length column binary reader
211...first capacity reduction header component
212...second capacity reduction header component
213...3rd capacity reduction header component
214a...4th capacity reduction index header component
214b..4th capacity reduction dictionary header component
230...Ministry of Information on Capacity Reduction Policy
231...first capacity reduction policy
232...second capacity reduction policy
233...3rd capacity reduction policy
234...the fourth capacity reduction policy
A system that displays 300... large amounts of data in a chart without memory capacity limitations
330...chart display
331...2D array index number management unit
332...array structure conversion unit
333... record indexing unit
334...record index reader

Claims (8)

A column binary file storage unit that generates table data received as a result of a data search result from a database management system (DBMS) as a column-based binary file and stores it in a disk;
A column binary file reader configured to read cell values one by one from a binary file for each column stored on a disk by the column binary file storage unit, construct a record, and then load a memory in a record unit; And
Visually display in the form of pixels at positions corresponding to X coordinates and Y coordinates on the chart by using each cell value corresponding to the X coordinate and Y coordinate in the record loaded into the memory by the column binary file reader. Including the chart display,
The chart display unit
By assigning record index numbers in the order of visual display in pixel form at the positions corresponding to the X and Y coordinates on the chart, each position of the pixels in the two-dimensional array corresponding to the X and Y coordinates. A two-dimensional array index number management unit for storing at least one record index number correspondingly;
Convert the two-dimensional array to a one-dimensional array, convert the X and Y coordinates of the two-dimensional array to the index number of the one-dimensional array, and store the index number of the one or more records in correspondence with the index number of the one-dimensional array. An array structure conversion unit;
The address value indicating the start position in the record index data file of the record index number corresponding to the index number of the one-dimensional array and the number of record index numbers belong to one cell of the same size and are sequentially configured, and all cells After converting the data to binary values in turn, a record index indicator file is created and stored on disk, and the record index number corresponding to the index number of the one-dimensional array is sequentially converted into binary values in byte units corresponding to the specified size. A record index creation unit for generating a record index data file after conversion and storing it in a disk; And
When designating a specific pixel on the chart, the index number of the one-dimensional array corresponding to the X coordinate and Y coordinate of the pixel is used to indicate the start position of the record index data file of the record index number in the record index indicator file stored on the disk. A large amount of data is stored using a binary file storage structure for each column, comprising a record index reader that obtains the address value and the number of record index numbers and retrieves the corresponding number of record index numbers from the record index data file. Charting system without capacity constraints.
delete The method according to claim 1,
The column binary file reader receives one or more record index numbers obtained by the record index reader and reads cell values corresponding to the record index numbers one by one from the column-unit binary file stored in the disk to form a record. Records are sequentially formed for all the received record index numbers and transferred to the chart display unit,
The chart display unit displays record information corresponding to a record index number, using a binary file storage structure for each column, and displaying a large amount of data as a chart without a memory capacity limitation.
A column binary file storage unit that generates table data received as a result of a data search result from a database management system (DBMS) as a column-based binary file and stores it in a disk;
A column binary file reader configured to read cell values one by one from a binary file for each column stored on a disk by the column binary file storage unit, construct a record, and then load a memory in a record unit; And
Visually display in the form of pixels at positions corresponding to X coordinates and Y coordinates on the chart by using each cell value corresponding to the X coordinate and Y coordinate in the record loaded into the memory by the column binary file reader. Including the chart display,
The column binary file storage unit,
Fixed-length column binary storage that converts all cell data in the column into binary values sequentially at the cell size interval of the original column type included in the numeric column table (Result Set), then creates a single binary file and stores it on disk ; And
The size in bytes corresponding to the number of characters included in each cell of the character column and the address value indicating the starting position of the character of each cell in the cell data file belong to one cell of the same size. Consecutively, after converting all cell data to binary values in sequence, a cell indicator file is created and stored on the disk, and cells are sequentially converted to binary values as much as the size in bytes corresponding to the number of characters contained in each cell. It includes a variable-length column binary storage unit that creates a data file and stores it on a disk,
The column binary file reader unit,
When reading the Nth record value from the binary file of the numeric column stored on the disk (header size of the binary file + N × size per cell of the original column format defined in the header of the binary file), it moves to the Nth record position. A fixed-length column binary reader that reads as much as the cell size of the original column format; And
When reading the value of the Nth record from the binary file of the character column stored on the disk (Header size of the cell indicator file + N × (Cell data file position defined in the header of the binary file) Integer format size + Number of characters to read Integer format Size)), move to the Nth record position, read as much as the size of the cell data file location integer format, and find the starting address of the Nth record in the cell data file, and read as much as the number of characters to be read that corresponds to the integer format size A column comprising a variable-length column binary reader that obtains the byte size corresponding to the number of characters to be read from the cell data file and reads the byte size corresponding to the number of characters to be read from the start address of the N-th record of the cell data file. A system that displays large amounts of data in a chart without memory capacity restrictions using the storage structure of each binary file.
delete A column binary file storage unit that generates table data received as a result of a data search result from a database management system (DBMS) as a column-based binary file and stores it in a disk;
A column binary file reader configured to read cell values one by one from a binary file for each column stored on a disk by the column binary file storage unit, construct a record, and then load a memory in a record unit; And
Visually display in the form of pixels at positions corresponding to X coordinates and Y coordinates on the chart by using each cell value corresponding to the X coordinate and Y coordinate in the record loaded into the memory by the column binary file reader. Including the chart display,
The column binary file storage unit,
For the received table (Result Set), according to the characteristics of the cell data values belonging to each numeric column, a different capacity reduction conversion format for each numeric column is applied to the original column, corresponding to the cell size interval of the converted format. Converts all cell data in a column to binary values in turn, creates a binary file for each column and stores it on disk.
In order to reduce the size of the binary file when generating by converting one or more numeric columns of the table (Result Set) received as a result of the data inquiry into a binary file, a capacity reduction policy suitable for the characteristics of the cell data values belonging to each column is implemented. Further comprising a capacity reduction policy information unit matching and storing by column,
The column binary file storage unit extracts the cell data value for each column from the received table (Result Set), and then refers to the capacity reduction policy information unit to reduce the capacity than the original column type of the table according to the capacity reduction policy matched to the corresponding column. A system that displays large-capacity data as a chart without limiting memory capacity using a binary file storage structure for each column, characterized in that binary files for each column are created and stored on disk.
delete The method of claim 6,
The column binary file reader unit
By analyzing the capacity reduction policy for each column stored in the capacity reduction policy information unit, one by one cell value is obtained from the numeric column binary file stored on the disk, and converted to the original column format stored in the header of the column binary file to construct a specific sequence of records. A system that displays large-capacity data as a chart without limiting memory capacity by using a binary file storage structure for each column, which is then loaded into memory.

Images