KR101634122B1 - System and method for data input with high speed - Google Patents
System and method for data input with high speed Download PDFInfo
- Publication number
- KR101634122B1 KR101634122B1 KR1020150149891A KR20150149891A KR101634122B1 KR 101634122 B1 KR101634122 B1 KR 101634122B1 KR 1020150149891 A KR1020150149891 A KR 1020150149891A KR 20150149891 A KR20150149891 A KR 20150149891A KR 101634122 B1 KR101634122 B1 KR 101634122B1
- Authority
- KR
- South Korea
- Prior art keywords
- data
- communication buffer
- input
- communication
- server
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
- G06F3/023—Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
-
- G06F17/30289—
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer And Data Communications (AREA)
Abstract
The present invention relates to a high-speed data input system and method, and a high-speed data input system according to an embodiment of the present invention includes a communication channel generator for generating a communication channel for an append to a specific table through a function call, And an input processor for writing the data to be input into the communication buffer using an application program interface for transmitting data recorded in the communication buffer to the server when the set condition is satisfied .
Description
The present invention relates to a technical idea for high-speed data input in a DBMS (Data Base Management System) supporting a Structured Query Language (SQL).
The Internet Of Things (IOT) generates large data, that is, a large amount of data at high speed, and requires extremely fast data input performance in order to process the data in real time.
The Internet Of Things (IOT) is based on the technology of storing and analyzing machine data in a database, for example, analyzing data generated on the production line in the hundreds of thousands per second and various information necessary for the production process based on the data .
Through the interface of the existing database, it is difficult to record such huge amount of data in real time smoothly.
In the existing DBMS (Data Base Management System), an INSERT statement is used for data input. At this time, the application generates a query statement and transmits it to the server or transmits the query to the server through the Prepare API in advance and sets the data to be input through the Bind API.
The Bind API is very good for specifying server type and application type so that the procedure is functionally useful for data conversion and error handling between application and DB server. However, it takes a lot of time to provide a lot of functions, The input performance is lowered.
Generally, it is general that the client and communication parts take 1/3 of the total execution time, the server takes 1/3 of the time to save the disk, and the real disk IO and the index update operation takes about 1/3.
An object of the present invention is to input data at a significantly faster rate than a conventional data input speed in a DBMS supporting SQL.
An object of the present invention is to improve data input speed and enable real-time data input.
An object of the present invention is to reduce the communication cost by reducing the data conversion cost between the server and the client and the function call cost of the database server.
The high-speed data input system according to an embodiment includes a communication channel generating unit for generating a communication channel for an append to a specific table through a function call, and a data input unit for inputting data using an application program interface for an append And an input processor for transmitting the data recorded in the communication buffer to the server through the communication channel when the set condition is satisfied.
The communication channel generation unit according to an embodiment transmits the transmission / reception data using the protocol for access and generates the communication channel for transferring the transmission result value corresponding to the data transmission.
The input processing unit according to an embodiment transmits the data to a value in the same format as the type of data used in the server by using the application program interface for the append.
The input processing unit according to an embodiment writes to the communication buffer a call of a continuous allowable function using the application program interface for the append.
The input processing unit according to an embodiment uses the application program interface for append to write data in the communication buffer when data is inputted to the communication buffer at a threshold value or more.
The input processing unit according to an embodiment writes data in the communication buffer when a function for emptying the communication buffer is called by using the application program interface for the append.
The input processing unit according to an embodiment writes data in the communication buffer until the application process is terminated using the application program interface for the append.
The input processing unit according to an embodiment transmits data in the communication buffer to the server when a condition for emptying the communication buffer occurs.
The protocol processing module of the server according to an embodiment transfers data in the communication buffer directly to the storage system by the protocol for access and records the same.
The protocol processing module of the server according to an embodiment returns the error occurrence data and the execution success count on the communication channel in the case of an operation error.
A high-speed data input method according to an exemplary embodiment of the present invention includes generating a communication channel for an append to a specific table through a function call, inputting data to be input to a communication buffer using an application program interface for an append And transmitting the data recorded in the communication buffer to the server through the communication channel when the set condition is satisfied.
The step of generating the communication channel according to an embodiment includes transmitting the transmission / reception data using the protocol for the access, and generating the communication channel transmitting the transmission result value corresponding to the data transmission.
The application program interface for append according to an embodiment transmits the data to a value of the same format as the type of data used in the server.
The application program interface for the append according to one embodiment writes to the communication buffer for a call to a contiguous allow function.
The application program interface for an append according to an embodiment writes data in the communication buffer when data is inputted to the communication buffer at a threshold value or more.
The application program interface for the append according to an embodiment writes data to the communication buffer when a function for emptying the communication buffer is called.
In one embodiment, the application program interface for the append appends data to the communication buffer until the append process is terminated.
The input processing unit according to an embodiment transmits data in the communication buffer to the server when a condition for emptying the communication buffer occurs.
The high-speed data input method according to an embodiment further includes transmitting data in the communication buffer directly to the storage system using the protocol processing module of the server and recording the same.
The high-speed data input method according to an embodiment of the present invention further includes the step of returning an error occurrence data, an execution success count through the communication channel in the case of an operation error using the protocol processing module of the server.
A program according to an exemplary embodiment of the present invention includes a command set for generating a communication channel for an append to a specific table through a function call, a data set to be input using an application program interface for append, An instruction set, and a set of instructions for transmitting data recorded in the communication buffer to the server over the communication channel when the set condition is satisfied.
According to embodiments, data can be input at a significantly faster rate than a conventional data input speed in a DBMS supporting SQL.
According to embodiments, data input speed is improved and real time data input is possible.
According to the embodiments, the communication cost can be reduced by reducing the data conversion cost between the server and the client and the function call cost of the database server.
1 is a diagram illustrating a high-speed data input system according to an embodiment.
2 is an explanatory view illustrating the operation of the entire system to which the high-speed data input system is applied.
3 is a diagram illustrating a high-speed data input method according to an embodiment.
4 is an embodiment for explaining the operation of the protocol processing module of the server.
It is to be understood that the specific structural or functional descriptions of embodiments of the present invention disclosed herein are presented for the purpose of describing embodiments only in accordance with the concepts of the present invention, May be embodied in various forms and are not limited to the embodiments described herein.
Embodiments in accordance with the concepts of the present invention are capable of various modifications and may take various forms, so that the embodiments are illustrated in the drawings and described in detail herein. It should be understood, however, that there is no intention to limit the embodiments according to the concepts of the present invention to the particular forms disclosed, but includes modifications, equivalents, or alternatives falling within the spirit and scope of the present invention.
The terms first, second, or the like may be used to describe various elements, but the elements should not be limited by the terms. The terms may be named for the purpose of distinguishing one element from another, for example without departing from the scope of the right according to the concept of the present invention, the first element being referred to as the second element, Similarly, the second component may also be referred to as the first component.
It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between. Expressions that describe the relationship between components, for example, "between" and "immediately" or "directly adjacent to" should be interpreted as well.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises ", or" having ", and the like, are used to specify one or more of the features, numbers, steps, operations, elements, But do not preclude the presence or addition of steps, operations, elements, parts, or combinations thereof.
Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the meaning of the context in the relevant art and, unless explicitly defined herein, are to be interpreted as ideal or overly formal Do not.
Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.
1 is a diagram illustrating a high-speed
The high-speed
In addition, the high-speed
To this end, the high-speed
First, the communication
Next, the
For example, the
That is, the
The
In addition, the
The protocol processing module of the server can preferentially perform operations such as decompression according to the setting with respect to data transferred from the
The
On the other hand, the communication channel can transfer the transmission / reception data using the access protocol and return the transmission result value corresponding to the data transmission. That is, when an operation error due to the occurrence of a constraint error or the like occurs in the server, the error occurrence data and the execution success count are transmitted through the communication channel. Since this data may affect performance, it is transmitted only when set by the user. In case of not transmitting, the number of successes and the number of errors can be recorded in a space that can be accessed by another query.
2 is an embodiment for explaining the operation of the
The
Specifically, the application program calls the SQLAppend API functions 210 to create an access communication channel for the desired table, and inputs it according to the application-
The
At this time, the
That is, the
On the other hand, the
3 is a diagram illustrating a high-speed data input method according to an embodiment.
The high-speed data input method according to an embodiment allows data to be input at a significantly faster rate than a conventional data input speed in a DBMS supporting SQL, and real-time data input can be performed by improving data input speed. In addition, by reducing the data conversion cost between the server and the client and the function call cost of the database server, the communication cost can be reduced.
To this end, the high-speed data input method according to an exemplary embodiment may generate a communication channel for an append to a specific table through a function call. For example, a function to be called can be interpreted as a function that can open an append based on SQL (Structured Query Language).
Next, a high-speed data input method according to an exemplary embodiment may write data to be input into a communication buffer using an application program interface for an append. At this time, the data recorded in the communication buffer is not actually deleted but can be transmitted to the server via the communication channel if the condition is satisfied while waiting in the communication buffer.
To this end, the high-speed data input method according to an exemplary embodiment may determine, through
For example, the high-speed data input method according to an exemplary embodiment may determine that a predetermined condition is satisfied when data is input to the communication buffer by a threshold value or more using an application program interface for an append.
In addition, the high-speed data input method according to an exemplary embodiment may determine that a predetermined condition is satisfied when a condition for emptying the communication buffer occurs.
In the high-speed data input method according to an exemplary embodiment, if the predetermined condition is satisfied as a result of the determination in
Meanwhile, the high-speed data input method according to an embodiment can directly write a function call that can open an append to a communication buffer based on a continuous function, that is, a Structured Query Language (SQL) . In addition, the high-speed data input method according to an exemplary embodiment may call an application program interface (API) for filling the buffer or for the user to explicitly empty the buffer, and may be based on Structured Query Language A function for an append that can close an append can be called and data can be written to a communication buffer to which the appended protocol is applied until the appending operation is terminated.
4 is an embodiment for explaining the operation of the protocol processing module of the server.
The operation method of the protocol processing module of the server can be directly transmitted to the storage system for data in the communication buffer and recorded (step 401). For example, the operation method of the protocol processing module of the server can transmit the same value as the data type of the server to the storage manager without performing a separate type conversion process.
Next, the operation method of the protocol processing module of the server is as follows. In case of an operation error due to a constraint error or the like in the server, the data in which an error occurred, and the execution success count are changed to AppendResult Lt; / RTI > To this end, the method of operating the protocol processing module of the server may first determine whether an operation error has occurred in data transfer or recording (step 402).
If it is determined in
Specifically, the operation method of the protocol processing module of the server is a function for the SQL-based assist function through the AppendResult according to the application protocol, when the operation error due to the constraint error occurs in the server, Lt; / RTI > Since this data may affect performance, it is transmitted only when it is set by the user. In case of not transmitting the data, the number of successes and the number of errors can be recorded in a space that can be accessed by another query.
As a result, when the present invention is used, data can be input at a significantly faster rate than a conventional data input speed in a DBMS supporting SQL. In addition, it is possible to reduce the communication cost by improving the data input speed and real time data input, reducing the data conversion cost between the server and the client and the function call cost of the database server.
The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.
The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.
The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.
While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.
Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.
Claims (8)
A communication channel generation unit for generating a communication channel for an append to a specific table through a function call; And
An input processor for writing data to be input into the communication buffer using an application program interface for appending and transmitting data recorded in the communication buffer to the server via the communication channel when the set condition is satisfied;
/ RTI >
Wherein the communication channel generation unit comprises:
And transmits the transmission / reception data using the available protocol, and generates the communication channel for transferring the transmission result value corresponding to the data transmission.
Wherein the application program interface for the append transmits the data to a value of the same format as the type of data used in the server.
The input processing unit,
And writes data in the communication buffer when data is inputted to the communication buffer by a threshold value or more using the application program interface for the append.
The input processing unit,
And writes data to the communication buffer when a function for emptying the communication buffer is called using the application program interface for the append.
The input processing unit,
The data is written to the communication buffer until the application process is terminated by using the application program interface for the application, and when a condition for emptying the communication buffer occurs, the data in the communication buffer is transmitted to the server Speed data input system.
The protocol processing module of the server includes:
And transmits the data in the communication buffer as it is to the storage system as it is, and returns at least one of the error occurrence data and the execution success count through the communication channel in the event of an operation error.
Generating a communication channel for an append to a specific table through a function call;
Writing data to be input into a communication buffer using an application program interface for an application; And
Transmitting the data recorded in the communication buffer to the server through the communication channel if the set condition is satisfied,
/ RTI >
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150149891A KR101634122B1 (en) | 2015-10-28 | 2015-10-28 | System and method for data input with high speed |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020150149891A KR101634122B1 (en) | 2015-10-28 | 2015-10-28 | System and method for data input with high speed |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| KR101634122B1 true KR101634122B1 (en) | 2016-06-29 |
Family
ID=56365935
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| KR1020150149891A KR101634122B1 (en) | 2015-10-28 | 2015-10-28 | System and method for data input with high speed |
Country Status (1)
| Country | Link |
|---|---|
| KR (1) | KR101634122B1 (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140048396A (en) | 2012-10-11 | 2014-04-24 | 주식회사 케이티 | System and method for searching file in cloud storage service, and method for controlling file therein |
| EP2846267A2 (en) * | 2013-08-28 | 2015-03-11 | Biosense Webster (Israel), Ltd. | Double buffering with atomic transactions for the persistent storage of real-time data flows |
| KR20150079050A (en) | 2013-12-31 | 2015-07-08 | 주식회사 포스코아이씨티 | System for distributing big data |
-
2015
- 2015-10-28 KR KR1020150149891A patent/KR101634122B1/en active IP Right Grant
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140048396A (en) | 2012-10-11 | 2014-04-24 | 주식회사 케이티 | System and method for searching file in cloud storage service, and method for controlling file therein |
| EP2846267A2 (en) * | 2013-08-28 | 2015-03-11 | Biosense Webster (Israel), Ltd. | Double buffering with atomic transactions for the persistent storage of real-time data flows |
| KR20150079050A (en) | 2013-12-31 | 2015-07-08 | 주식회사 포스코아이씨티 | System for distributing big data |
Non-Patent Citations (1)
| Title |
|---|
| InfiniFlux v2.0.0 사용자 매뉴얼 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9002802B2 (en) | High throughput, reliable replication of transformed data in information systems | |
| US9811557B2 (en) | Optimizing query statements in relational databases | |
| US9864876B2 (en) | Live editing and publishing of documents within a content management system using a hybrid draft authorization workflow | |
| US9881035B2 (en) | Systems and methods for in-place migration with downtime minimization | |
| CN107301214A (en) | Data migration method, device and terminal device in HIVE | |
| US10241699B2 (en) | Method, apparatus or software for transferring a storage replication system between storage systems | |
| US10324933B2 (en) | Technique for processing query in database management system | |
| US10262007B2 (en) | Systems and methods for automatically passing hints to a file system | |
| KR101634122B1 (en) | System and method for data input with high speed | |
| US10761940B2 (en) | Method, device and program product for reducing data recovery time of storage system | |
| US10997057B2 (en) | Debugging asynchronous functions | |
| US20230065492A1 (en) | Method for obtaining browser running data, electronic device, and storage medium | |
| CN104572638A (en) | Data reading and writing method and device | |
| US9275103B2 (en) | Optimization of JOIN queries for related data | |
| WO2018094958A1 (en) | Data processing method, device and system | |
| US11606437B2 (en) | System and method for boosting up performance of NVMe-TCP transport | |
| US10803030B2 (en) | Asynchronous SQL execution tool for zero downtime and migration to HANA | |
| CN112948410A (en) | Data processing method, device, equipment and medium | |
| CN112948467A (en) | Data processing method and device, computer equipment and storage medium | |
| US10977249B2 (en) | Method, system, and apparatus for performing flow-based processing using stored procedure | |
| US9411837B2 (en) | System and method for optimizing memory utilization in a database | |
| JP5546909B2 (en) | Data processing system, method and program | |
| CN111625524B (en) | Data processing method, device, equipment and storage medium | |
| US20220377088A1 (en) | Data management computer and data management method | |
| US20160139969A1 (en) | In-memory approach to extend semantic event processing with domain insights |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| E701 | Decision to grant or registration of patent right | ||
| GRNT | Written decision to grant | ||
| FPAY | Annual fee payment |
Payment date: 20190605 Year of fee payment: 4 |