WO2013066010A1 - Method for pre-loading in memory and method for parallel processing for high-volume batch processing - Google Patents

Method for pre-loading in memory and method for parallel processing for high-volume batch processing Download PDF

Info

Publication number
WO2013066010A1
WO2013066010A1 PCT/KR2012/008913 KR2012008913W WO2013066010A1 WO 2013066010 A1 WO2013066010 A1 WO 2013066010A1 KR 2012008913 W KR2012008913 W KR 2012008913W WO 2013066010 A1 WO2013066010 A1 WO 2013066010A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
comparison information
memory
comparison
file
Prior art date
Application number
PCT/KR2012/008913
Other languages
French (fr)
Korean (ko)
Inventor
채조욱
정호철
박수용
김경희
박진철
최종건
곽송해
황민정
Original Assignee
에스케이씨앤씨 주식회사
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 에스케이씨앤씨 주식회사 filed Critical 에스케이씨앤씨 주식회사
Publication of WO2013066010A1 publication Critical patent/WO2013066010A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements

Definitions

  • the present invention relates to a method of batch processing a large amount of information in a batch, and more particularly, to a memory shipping material and a parallel processing method for a large batch processing of selecting and batching a large amount of data at a time based on a relational database. .
  • the maintenance logic is low because the dialogue logic and input / output logic are not easily separated, and when executing, large sized queries are generated based on each action read from the file, which hinders performance and file sorting / dividing / It requires additional solutions for tasks such as mergers.
  • the present invention has been made in view of the above-described prior art, and has a relational database based on a database storing large amounts of information. It is to provide a method.
  • the present invention is to provide a memory shipping material and a parallel processing method for a large-scale batch processing by dividing the first and second comparison information required for the metabolic operation by section and loading them in a memory batch.
  • the present invention divides the input data based on a key value, and then divides the work in parallel, and performs a memory loading material for a large-capacity batch processing performed by sequentially dividing the divided input data into intervals within each parallel job. And a parallel processing method.
  • a memory shipping material and a parallel processing method for batch processing of a large amount of information may include retrieving and processing second comparison information from a second database and loading the same into a memory. Searching and processing the first comparison information of the specific identifier from a database, and retrieving the second comparison information of the specific identifier from the memory; and the dialogue of the first comparison information and the second comparison information of the specific identifier. And outputting a result of the metabolism, and deleting the first comparison information and the second comparison information of the specific identifier that performed the metabolism after the metabolism is performed in the memory.
  • the loading of the second comparison information may include loading the blue code information into the memory in advance, dividing the second comparison information of the second database into at least one or more sections, and performing the second comparison for each divided section. Loading information into the memory as a hash table.
  • the blueprint code before loading the first comparison information of the specific identifier, the information center and the tax calendar, the blueprint code, various information such as corporation / closing company in the form of a hash table in advance in the memory.
  • the memory may be implemented as a heap memory.
  • the special identifier is characterized in that the identification information for identifying each individual, such as a business operator ID.
  • the metabolic result outputting step may include outputting the first comparison information disagreement file, the second comparison information disagreement file, the first comparison information updating file, and the second comparison information updating file.
  • the method may further include inserting the first comparison information mismatch file and the second comparison information mismatch file into a database.
  • the method may further include updating the first and second databases based on the first comparison information update file and the second comparison information update file. .
  • the sections are divided based on the specific identifier, and the same tasks are distributed in parallel, and the distributed input data is sequentially divided into sections in each parallel task. .
  • the present invention implements a large-scale database based on a relational database to select and process data in large quantities at a time, thereby improving the number of database calls.
  • the present invention can divide the information required for the metabolic operation by the interval to be shipped to the memory to perform a batch process without generating an intermediate file, it can be provided as an online service.
  • the present invention does not require file sorting / merge / split so that there is no need to consider resource contention. Accordingly, the present invention can improve performance, and it is possible to precisely separate business logic from SQL statements, thereby improving maintainability. .
  • the present invention divides the interval based on the key value of the input data when batch processing a large amount of data, and then distributes the tasks in parallel, and since the divided input data is sequentially divided into the intervals within each parallel operation. This can improve throughput and throughput.
  • FIG. 1 is a block diagram showing a mass information batch processing system related to the present invention.
  • FIG. 2 is a flow chart showing a large-capacity information batch processing method related to the present invention.
  • FIG. 3 is a diagram illustrating a process of collectively processing a large amount of information using a parallel process related to the present invention.
  • FIG. 4 illustrates the structure of a hash processing logic and a hash table in accordance with the present invention.
  • FIG. 1 is a block diagram showing a mass information batch processing system related to the present invention.
  • the large-capacity information batch processing system stores a first database (hereinafter, referred to as a first DB) 10 in which a large amount of first comparison information is stored, and a large amount of second comparison information. And a second database (hereinafter referred to as a second DB) 20 and a metabolic processing device 30 which performs cross check of the first comparison information and the second comparison information.
  • a first database hereinafter, referred to as a first DB
  • second database hereinafter referred to as a second DB
  • the first DB 10 and the second DB 20 are implemented based on a relational database (RDB) that expresses data (first and second comparison information) in a simple table form.
  • RDB relational database
  • the first comparison information may correspond to report information and include a business ID and a payment tax amount
  • the second comparison information May correspond to payment information, and includes a business ID, account number, and payment amount, and the information is provided by a bank.
  • the first comparison information of the first DB 10 is directly provided by the taxpayer through a computer provided by a public office (for example, a tax processing office (National Tax Service, etc.)) or a lower office such as a tax office.
  • the metabolic processor 30 compares the first and second comparison information stored in the memory 31 with the memory 31 temporarily storing the first and second comparison information used for metabolic processing. Processor 32 to review.
  • first comparison information and second comparison information drawn from the first DB 10 and the second DB 20 are loaded in the form of a hash table.
  • the memory 31 includes a heap memory 31a, and the small sized codeable data resides in the heap memory 31a.
  • 4 is a diagram illustrating the structure of a hash processing logic and a hash table according to the present invention.
  • the heap memory 31a is loaded with various related information in advance so that the first comparison information can be inquired or processed.
  • various related information such as blueprint code and payment information, information center, tax calendar, corporation / discard company, etc. are stored in the heap memory 31a. Preloaded. In the case of corporation / closer information, it is reloaded by section.
  • the processor 32 inquires and processes the second comparison information from the second DB 20 and loads the second comparison information into the heap memory 31a. In addition, the processor 32 inquires, processes and withdraws the first comparison information of the specific identifier from the first DB 10, and the second comparison information of the specific identifier among the second comparison information loaded in the heap memory 31a. Search for. The processor 32 checks the extracted first comparison information and the retrieved second comparison information to check the state (eg, payment status) of the corresponding entity.
  • the state of the individual includes normal payment, overpayment, nonpayment, mispayment, and the like.
  • the processor 32 outputs four kinds of metabolic results according to the metabolic performance.
  • the processor 32 inserts the metabolic result into the first comparison information / second comparison information disagreement DB or updates the metabolic result in the first / second DB by reflecting the metabolic result.
  • the metabolic result includes a first comparison information disagreement file, a second comparison information disagreement file, a first comparison information update file, a second comparison information update file, and the like.
  • the metabolic results may be used for confirming the evidence data and processing results according to the corresponding large-scale information processing.
  • the processor 32 metabolizes the retrieved first comparison information and the retrieved second comparison information, and then deletes the first comparison information, which is completed, from the memory 31, and the retrieved second comparison information is stored in the heap memory 31a.
  • the storage space of the memory 31 is secured by removing the comparison information.
  • FIG. 2 is a flowchart illustrating a method of processing a large amount of information in a batch according to the present invention.
  • the processor 32 inquires and processes the second comparison information from the second DB 20 and loads the second comparison information into the heap memory 31a of the memory 31 (S101). In this case, the processor 32 does not load the entire second comparison information stored in the second DB 20 into the memory 31, but divides it into at least one or more sections and loads each of the sections into the memory 31 for each section.
  • the second comparison information is loaded into the heap memory 31a in the form of a hash table. For example, when the second comparison information stored in the second DB 20 is 220,000, the 220,000 second comparison information is divided into ten sections, and the second comparison information is stored in the memory 31. )).
  • the processor 32 Before querying the second comparison information, the processor 32 loads predetermined classification information (eg, blue code information) into the heap memory 31a in advance, and sets a memory limit to perform buffering. do.
  • predetermined classification information eg, blue code information
  • the processor 32 retrieves and processes the first comparison information of the specific identifier from the first DB 10 (S102).
  • the specific identifier refers to identification information that can distinguish each entity, such as a business ID.
  • the processor 32 extracts first comparison information having an operator ID of '1102401444' from the first DB 10 to generate a metabolic input file (eg, a hash table).
  • the processor 32 may provide various related information (eg, information center and tax calendar, blue book code, Information such as corporation / disclosure) is preloaded into the memory 31 as a hash table.
  • various related information eg, information center and tax calendar, blue book code, Information such as corporation / disclosure
  • the processor 32 searches for the second comparison information of the specific identifier in the heap memory 31a (S103). For example, the processor 32 searches that the operator ID is '1102401444' among the second comparison information stored in the heap memory 31a.
  • the processor 32 performs a cross-check between the first comparison information of the specific identifier fetched from the first DB 10 and the second comparison information of the specific identifier retrieved from the heap memory 31a. (S104).
  • the processor 32 performs metabolism in a multi-loop manner based on the 'first comparison information' case by one second comparison information for each first comparison information.
  • the processor 32 generates a metabolic result in a file and outputs it (S105).
  • the metabolic result is output as a first comparison information disagreement file, a second comparison information disagreement file, a first comparison information updating file, and a second comparison information updating file.
  • the processor 32 secures the memory by deleting the first comparison information and the second comparison information of the specific identifier that performed the metabolism from the memory 31 (S106). For example, the processor 32 removes the first comparison information and the second comparison information having the operator ID '1102401444' from the memory 31.
  • the processor 32 then performs additional logic to maximize performance, as shown in FIG.
  • the processor 32 performs the metabolism, and then deletes the finished hash table record.
  • the hash table search speed is approximately O (1), much faster than the DBMS search speed O (logn).
  • the maximum search rate 0 (1/2) may be implemented by deleting the hash table.
  • the second comparison information is divided into a plurality of sections, and the metabolic operations are separately performed on the divided second comparison information for each divided section.
  • the metabolic operations may be performed in parallel. .
  • the processor 32 when the number of distributed tasks is determined by the task scheduler, the processor 32 performs a preceding task for parallel processing.
  • the processor 32 processes the same tasks in parallel by creating threads for the determined number of distributed tasks.
  • the first comparison information of the specific identifier is queried and retrieved, and the extracted first comparison information is processed in parallel with the second comparison information for each of the plurality of sections and each metabolic task. That is, since the first comparison information of the specific identifier and the second section information of the ten sections are metabolized in parallel, the metabolic operation of a large amount of information for one information entity is performed on the second comparison information as a whole.
  • the metabolic operation of the first comparison information for each of the plurality of identifiers and the second comparison information for the specific section is processed in parallel.
  • the first comparison information for ten different operator IDs is extracted, and the first comparison information for each extracted operator ID and the second comparison information of a specific section are simultaneously processed in parallel.
  • the metabolic operation for the second comparison information for each section is processed in parallel.
  • the processor 32 When the metabolic task is completed, the processor 32 performs parallel processing with a plurality of metabolic result insertion task threads for inserting the first comparison information / second comparison information discord information of metabolic results output from each metabolic task thread into a DB. In addition, after inserting the dialogue result, the dialogue result update (update) tasks are executed in parallel. The processor 32 performs a metabolic result report task when the metabolic result update task is completed.
  • the present invention may be performed by dividing a section based on a specific identifier and then distributing the same work in parallel, and sequentially dividing the input data distributed into sections in each parallel work. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The present invention relates to a high-volume information batch-processing method which is based on a relational database. The high-volume information batch processing method according to the present invention pre-loads second comparative data from a second database into a memory by checking and processing same, withdraws first comparative data for a specific identifier from a first database by checking and processing same, retrieves second comparative data for the specific identifier from the memory, outputs a cross-checking result by cross-checking the second comparative data and the first comparative data for the specific identifier, and thus secures the memory by deleting from the memory the second comparative data and first comparative data for the specific identifier, which have undergone cross-checking.

Description

대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법Memory shipments and parallel processing methods for large batch processing
본 발명은 대용량 정보를 배치(Batch)로 일괄처리하는 방법에 관한 것으로, 특히, 관계형 데이터베이스 기반으로 한번에 대량의 데이터를 선택하여 일괄처리하는 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법에 관한 것이다.The present invention relates to a method of batch processing a large amount of information in a batch, and more particularly, to a memory shipping material and a parallel processing method for a large batch processing of selecting and batching a large amount of data at a time based on a relational database. .
종래에, 대용량 정보를 배치로 일괄처리하기 위해 사용되는 파일 기반 시스템은 순차접근방식(Sequential Access Method, SAM) 기반 처리방식으로 구축되어 대용량 정보를 일괄적으로 대사처리한다.Conventionally, a file-based system used to batch process large amounts of information in batches is built on a sequential access method (SAM) based processing to metabolize large amounts of information in batches.
이러한 파일기반 시스템으로 대규모 작업을 수행하는 경우, 각 작업들 간의 디스크 입출력 경합으로 인해 전체적인 시스템 성능을 저하시킨다.In the case of performing large-scale work with such a file-based system, disk I / O contention among the tasks degrades overall system performance.
그리고, 대사처리과정에서 불필요한 중간 작업파일들을 다량으로 생성하여 디스크 및 중앙처리장치(Central Processing Unit, CPU)의 자원을 소모하게 된다.In addition, a large amount of unnecessary intermediate work files are generated during metabolic processing, consuming resources of a disk and a central processing unit (CPU).
또한, 대사 로직과 입출력 로직이 간결하게 분리되지 않아 유지 보수성이 낮으며, 수행시 파일에서 읽은 각 수행 건 기반으로 작은 크기의 쿼리(query)를 대규모로 발생시켜 성능을 저해하며 파일 정렬/분할/합병 등의 작업을 위한 추가적인 솔루션을 필요로 한다.In addition, the maintenance logic is low because the dialogue logic and input / output logic are not easily separated, and when executing, large sized queries are generated based on each action read from the file, which hinders performance and file sorting / dividing / It requires additional solutions for tasks such as mergers.
본 발명은 상기한 종래기술을 감안하여 안출된 것으로, 대용량 정보가 저장되는 데이터베이스를 관계형 데이터베이스 기반으로 구축하여 한번에 대량으로 데이터를 선택하여 대사작업을 수행하는 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법을 제공하기 위한 것이다.SUMMARY OF THE INVENTION The present invention has been made in view of the above-described prior art, and has a relational database based on a database storing large amounts of information. It is to provide a method.
또한, 본 발명은 대사작업에 필요한 제1비교 및 제2비교정보를 구간별로 분할하여 메모리에 선적재하여 일괄처리하는 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법을 제공하기 위한 것이다.In addition, the present invention is to provide a memory shipping material and a parallel processing method for a large-scale batch processing by dividing the first and second comparison information required for the metabolic operation by section and loading them in a memory batch.
또한, 본 발명은 입력 데이터를 키값을 기준으로 구간을 분할한 후 병렬로 작업을 분배하여 수행하며, 각 병렬 작업 내에서도 분배된 입력 데이터를 구간으로 순차 분할하여 수행하는 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법을 제공하기 위한 것이다.In addition, the present invention divides the input data based on a key value, and then divides the work in parallel, and performs a memory loading material for a large-capacity batch processing performed by sequentially dividing the divided input data into intervals within each parallel job. And a parallel processing method.
상기한 과제를 실현하기 위한 본 발명의 실시예에 따른 대용량 정보 배치처리를 위한 메모리 선적재 및 병렬처리 방법은 제2데이터베이스로부터 제2비교정보를 조회 및 가공하여 메모리에 선적재하는 단계와, 제1데이터베이스로부터 특정 식별자의 제1비교정보를 조회 및 가공하여 인출하며 상기 메모리에서 상기 특정 식별자의 제2비교정보를 검색하는 단계와, 상기 특정 식별자의 제1비교정보와 제2비교정보에 대한 대사를 수행하여 대사결과를 출력하는 단계와, 상기 대사 수행 후 대사를 수행한 상기 특정 식별자의 제1비교정보와 제2비교정보를 메모리에서 삭제하는 단계를 포함한다.According to an embodiment of the present invention, a memory shipping material and a parallel processing method for batch processing of a large amount of information may include retrieving and processing second comparison information from a second database and loading the same into a memory. Searching and processing the first comparison information of the specific identifier from a database, and retrieving the second comparison information of the specific identifier from the memory; and the dialogue of the first comparison information and the second comparison information of the specific identifier. And outputting a result of the metabolism, and deleting the first comparison information and the second comparison information of the specific identifier that performed the metabolism after the metabolism is performed in the memory.
또한, 상기 제2비교 정보의 적재단계는, 청서코드 정보를 상기 메모리에 미리 적재하는 단계와, 상기 제2데이터베이스의 제2비교 정보를 적어도 하나 이상의 구간으로 분할하고 그 분할된 구간별 제2비교 정보를 해시테이블로 상기 메모리에 적재하는 단계를 포함한다.The loading of the second comparison information may include loading the blue code information into the memory in advance, dividing the second comparison information of the second database into at least one or more sections, and performing the second comparison for each divided section. Loading information into the memory as a hash table.
또한, 상기 특정 식별자의 제1비교 정보 조회 전 정보화센터 및 세무달력, 청서코드, 법인/폐업자와 같은 각종 정보들을 해시테이블 형태로 메모리에 미리 적재하는 것을 특징으로 한다.In addition, it is characterized in that before loading the first comparison information of the specific identifier, the information center and the tax calendar, the blueprint code, various information such as corporation / closing company in the form of a hash table in advance in the memory.
또한, 상기 메모리는, 힙 메모리로 구현되는 것을 특징으로 한다.The memory may be implemented as a heap memory.
또한, 상기 특별 식별자는, 사업자 아이디와 같이 각각의 개체를 식별할 수 있는 식별정보인 것을 특징으로 한다.In addition, the special identifier is characterized in that the identification information for identifying each individual, such as a business operator ID.
또한, 상기 대사결과 출력단계는, 제1비교정보 불부합 파일, 제2비교정보 불부합 파일, 제1비교정보 갱신 파일, 제2비교정보 갱신 파일로 출력하는 것을 특징으로 한다.The metabolic result outputting step may include outputting the first comparison information disagreement file, the second comparison information disagreement file, the first comparison information updating file, and the second comparison information updating file.
또한, 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법은 상기 제1비교정보 불부합 파일 및 제2비교정보 불부합 파일을 데이터베이스에 삽입하는 단계를 더 포함하는 것을 특징으로 한다.The method may further include inserting the first comparison information mismatch file and the second comparison information mismatch file into a database.
또한, 대용량 배치처리를 위한 메모리 선적재 및 병렬처리 방법은 상기 제1비교정보 갱신 파일 및 제2비교정보 갱신 파일에 근거하여 제1 및 제2데이터베이스를 갱신하는 단계를 더 포함하는 것을 특징으로 한다.The method may further include updating the first and second databases based on the first comparison information update file and the second comparison information update file. .
또한, 상기 대사를 수행할 때, 상기 특정 식별자를 기준으로 구간을 분할한 후 병렬로 동일 작업을 분배하여 수행하며, 각 병렬 작업 내에서도 분배된 입력 데이터를 구간으로 순차 분할하여 수행하는 것을 특징으로 한다.In addition, when the metabolism is performed, the sections are divided based on the specific identifier, and the same tasks are distributed in parallel, and the distributed input data is sequentially divided into sections in each parallel task. .
이상과 같이, 본 발명은 대용량의 데이터베이스를 관계형 데이터베이스 기반으로 구현하여 한번에 대량으로 데이터를 선택하여 처리하므로, 데이터베이스 호출 횟수를 개선할 수 있다.As described above, the present invention implements a large-scale database based on a relational database to select and process data in large quantities at a time, thereby improving the number of database calls.
또한, 본 발명은 대사작업시 필요한 정보를 구간별로 분할하여 메모리에 선적재하여 중간 파일을 생성하지 않고 일괄처리를 수행할 수 있으며, 온라인 서비스로 제공할 수 있다.In addition, the present invention can divide the information required for the metabolic operation by the interval to be shipped to the memory to perform a batch process without generating an intermediate file, it can be provided as an online service.
그리고, 본 발명은 파일 정렬/병합/분할이 필요하지 않아 이로 인한 자원 경합을 고려할 필요가 없어 성능을 개선할 수 있으며, 업무 로직과 SQL 구문과의 정확한 분리가 가능하여 유지보수성을 향상시킬 수 있다.In addition, the present invention does not require file sorting / merge / split so that there is no need to consider resource contention. Accordingly, the present invention can improve performance, and it is possible to precisely separate business logic from SQL statements, thereby improving maintainability. .
또한, 본 발명은 대량의 데이터를 일괄처리할 때 입력 데이터의 키값을 기준으로 구간을 분할한 후 병렬로 작업을 분배하여 수행하며, 각 병렬 작업 내에서도 분배된 입력 데이터를 구간으로 순차 분할하여 수행하므로, 처리량과 처리속도를 개선할 수 있다.In addition, the present invention divides the interval based on the key value of the input data when batch processing a large amount of data, and then distributes the tasks in parallel, and since the divided input data is sequentially divided into the intervals within each parallel operation. This can improve throughput and throughput.
도 1은 본 발명과 관련된 대용량 정보 배치처리 시스템을 도시한 블록 구성도.1 is a block diagram showing a mass information batch processing system related to the present invention.
도2는 본 발명과 관련된 대용량 정보 배치처리 방법을 도시한 흐름도.2 is a flow chart showing a large-capacity information batch processing method related to the present invention.
도3은 본 발명과 관련된 병렬처리를 이용하여 대용량 정보를 대사로 일괄처리하는 과정을 도시한 도면.3 is a diagram illustrating a process of collectively processing a large amount of information using a parallel process related to the present invention.
도4는 본 발명에 따른 해시 처리 로직과 해시 테이블의 구조를 나타낸 도면.4 illustrates the structure of a hash processing logic and a hash table in accordance with the present invention.
이하, 도면들을 참조하여 본 발명의 실시예를 상세하게 설명한다.Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
도 1은 본 발명과 관련된 대용량 정보 배치처리 시스템을 도시한 블록구성도이다.1 is a block diagram showing a mass information batch processing system related to the present invention.
도 1에 도시된 바와 같이, 본 발명에 따른 대용량 정보 배치처리 시스템은 대용량의 제1비교 정보가 저장되는 제1 데이터베이스(이하, 제1DB라 함, 10)와, 대용량의 제2비교 정보가 저장되는 제2데이터베이스(이하, 제2DB라 함, 20)와, 상기 제1비교 정보와 제2비교 정보의 대사(cross check)를 수행하는 대사처리 장치(30)를 포함한다.As shown in FIG. 1, the large-capacity information batch processing system according to the present invention stores a first database (hereinafter, referred to as a first DB) 10 in which a large amount of first comparison information is stored, and a large amount of second comparison information. And a second database (hereinafter referred to as a second DB) 20 and a metabolic processing device 30 which performs cross check of the first comparison information and the second comparison information.
상기 제1DB(10)와 제2DB(20)는 데이터(제1 및 제2비교 정보)를 단순한 표(table) 형태로 표현하는 관계형 데이터베이스(Relational Database, RDB) 기반으로 구현된다.The first DB 10 and the second DB 20 are implemented based on a relational database (RDB) that expresses data (first and second comparison information) in a simple table form.
본 발명에 따른 대용량 정보 배치처리 시스템이 관공서의 세금처리 관련 업무에 적용되는 경우, 상기 제1비교 정보는 신고 정보에 해당될 수 있으며 사업자 아이디 및 납부세액 등을 포함하게 되고, 상기 제2비교 정보는 납부 정보에 해당될 수 있으며, 사업자 아이디 및 계좌번호, 납부금액 등을 포함하며, 해당 정보는 은행으로부터 제공받는다. 그리고, 상기 제1DB(10)의 제1비교 정보는 공공관청(예: 세무처리 관청(국세청 등))에서 제공하는 전산매체를 통해 납세 의무자로부터 직접 제공받거나 또는 세무서와 같은 하급 관청으로부터 제공받는다.When the large-capacity information batch processing system according to the present invention is applied to a tax processing related task of a public office, the first comparison information may correspond to report information and include a business ID and a payment tax amount, and the second comparison information. May correspond to payment information, and includes a business ID, account number, and payment amount, and the information is provided by a bank. In addition, the first comparison information of the first DB 10 is directly provided by the taxpayer through a computer provided by a public office (for example, a tax processing office (National Tax Service, etc.)) or a lower office such as a tax office.
대사처리장치(30)는 대사처리에 사용되는 제1비교 정보와 제2비교 정보를 일시적으로 저장하는 메모리(31)와 상기 메모리(31)에 저장된 제1비교 정보와 제2비교 정보를 대조하고 검토하는 프로세서(32)를 포함한다.The metabolic processor 30 compares the first and second comparison information stored in the memory 31 with the memory 31 temporarily storing the first and second comparison information used for metabolic processing. Processor 32 to review.
상기 메모리(31)에는 상기 제1DB(10)와 제2DB(20)로부터 인출한 제1비교 정보와 제2비교 정보가 해시 테이블(hash table) 형태로 로딩(loading)된다. 그리고, 상기 메모리(31)는 힙 메모리(heap memory)(31a)를 포함하며, 규모가 작은 코드성 데이터를 상기 힙 메모리(31a)에 상주시킨다. 도 4는 본 발명에 따른 해시 처리 로직과 해시 테이블의 구조를 나타낸 도면이다.In the memory 31, first comparison information and second comparison information drawn from the first DB 10 and the second DB 20 are loaded in the form of a hash table. In addition, the memory 31 includes a heap memory 31a, and the small sized codeable data resides in the heap memory 31a. 4 is a diagram illustrating the structure of a hash processing logic and a hash table according to the present invention.
상기 힙 메모리(31a)에는 상기 제1비교 정보를 조회하거나 가공할 수 있도록 각종 관련정보들을 미리 적재한다. 본 발명에 따른 대용량 정보 배치처리 시스템이 관공서의 세금처리 관련 업무에 적용되는 경우, 청서코드 및 납부정보, 정보화센터, 세무달력, 법인/폐업자 등과 같은 각종 관련 정보들이 상기 힙 메모리(31a)에 미리 적재된다. 법인/폐업자 정보의 경우, 구간별로 재로드된다.The heap memory 31a is loaded with various related information in advance so that the first comparison information can be inquired or processed. When the large-capacity information batch processing system according to the present invention is applied to a tax processing related task of a government office, various related information such as blueprint code and payment information, information center, tax calendar, corporation / discard company, etc. are stored in the heap memory 31a. Preloaded. In the case of corporation / closer information, it is reloaded by section.
상기 프로세서(32)는 제2DB(20)로부터 제2비교 정보를 조회 및 가공하여 힙 메모리(31a)에 적재한다. 그리고, 상기 프로세서(32)는 제1DB(10)로부터 특정 식별자의 제1비교 정보를 조회 및 가공하여 인출하며 상기 힙 메모리(31a)에 적재된 제2비교 정보 중 상기 특정 식별자의 제2비교 정보를 검색한다. 상기 프로세서(32)는 상기 인출된 제1비교 정보와 검색된 제2비교 정보를 대조검토하여 해당 개체의 상태(예: 납부상태)를 확인한다. 상기 개체의 상태(예: 납부상태)는 정상납부, 과납, 미납, 오납 등을 포함한다.The processor 32 inquires and processes the second comparison information from the second DB 20 and loads the second comparison information into the heap memory 31a. In addition, the processor 32 inquires, processes and withdraws the first comparison information of the specific identifier from the first DB 10, and the second comparison information of the specific identifier among the second comparison information loaded in the heap memory 31a. Search for. The processor 32 checks the extracted first comparison information and the retrieved second comparison information to check the state (eg, payment status) of the corresponding entity. The state of the individual (eg payment status) includes normal payment, overpayment, nonpayment, mispayment, and the like.
또한, 상기 프로세서(32)는 대사 수행에 따른 대사결과를 4종류의 파일로 출력한다. 상기 프로세서(32)는 상기 대사결과를 제1비교정보/제2비교정보 불부합 DB에 삽입하거나 상기 대사결과를 반영하여 제1/제2DB에 업데이트한다. 상기 대사결과는 제1비교정보 불부합 파일, 제2비교정보 불부합 파일, 제1비교정보 업데이트 파일, 제2비교정보 업데이트 파일 등을 포함한다. 그리고, 상기 대사결과는 해당 대용량 정보처리에 따른 근거자료와 처리결과 확인용으로 사용될 수 있다.In addition, the processor 32 outputs four kinds of metabolic results according to the metabolic performance. The processor 32 inserts the metabolic result into the first comparison information / second comparison information disagreement DB or updates the metabolic result in the first / second DB by reflecting the metabolic result. The metabolic result includes a first comparison information disagreement file, a second comparison information disagreement file, a first comparison information update file, a second comparison information update file, and the like. In addition, the metabolic results may be used for confirming the evidence data and processing results according to the corresponding large-scale information processing.
상기 프로세서(32)는 상기 인출된 제1비교 정보와 검색된 제2비교 정보를 대사한 후 대사를 마친 제1비교 정보는 메모리(31)에서 삭제하고, 상기 힙 메모리(31a)에 상기 검색된 제2비교 정보를 제거하여 메모리(31)의 저장 공간을 확보한 다.The processor 32 metabolizes the retrieved first comparison information and the retrieved second comparison information, and then deletes the first comparison information, which is completed, from the memory 31, and the retrieved second comparison information is stored in the heap memory 31a. The storage space of the memory 31 is secured by removing the comparison information.
도 2는 본 발명과 관련된 대용량 정보를 배치로 처리하는 방법을 도시한 흐름도이다.2 is a flowchart illustrating a method of processing a large amount of information in a batch according to the present invention.
도 2에 따르면, 프로세서(32)는 제2DB(20)로부터 제2비교 정보를 조회 및 가공하여 메모리(31)의 힙 메모리(31a)에 적재한다(S101). 이때, 프로세서(32)는 제2DB(20)에 저장된 제2비교 정보 전체를 메모리(31)에 적재하지 않고, 적어도 하나 이상의 구간으로 분할하여 각 구간별로 메모리(31)에 적재한다. 상기 제2비교 정보는 해시테이블(hash table) 형태로 힙 메모리(31a)에 적재된다. 예를 들어, 제2DB(20)에 저장된 제2비교 정보가 22만 건인 경우, 22만 건의 제2비교 정보를 10개의 구간으로 분할하여 분할된 구간별로 2.2만 건의 제2비교 정보를 메모리(31)에 적재한다.According to FIG. 2, the processor 32 inquires and processes the second comparison information from the second DB 20 and loads the second comparison information into the heap memory 31a of the memory 31 (S101). In this case, the processor 32 does not load the entire second comparison information stored in the second DB 20 into the memory 31, but divides it into at least one or more sections and loads each of the sections into the memory 31 for each section. The second comparison information is loaded into the heap memory 31a in the form of a hash table. For example, when the second comparison information stored in the second DB 20 is 220,000, the 220,000 second comparison information is divided into ten sections, and the second comparison information is stored in the memory 31. )).
그리고, 상기 제2비교 정보를 조회하기 전 프로세서(32)는 소정의 분류 정보(예: 청서코드 정보)를 상기 힙 메모리(31a)에 미리 적재하고, 메모리 한계치를 설정하여 버퍼링(buffering)을 수행한다.Before querying the second comparison information, the processor 32 loads predetermined classification information (eg, blue code information) into the heap memory 31a in advance, and sets a memory limit to perform buffering. do.
상기 프로세서(32)는 제1DB(10)로부터 특정 식별자의 제1비교 정보를 조회 및 가공하여 인출한다(S102). 상기 특정 식별자는 사업자 아이디와 같이 각 개체를 구별할 수 있는 식별정보를 의미한다. 예컨대, 프로세서(32)는 사업자 아이디가 '1102401444'인 제1비교 정보를 제1DB(10)로부터 각각 인출하여 대사 입력 파일(예: 해시 테이블)을 생성한다.The processor 32 retrieves and processes the first comparison information of the specific identifier from the first DB 10 (S102). The specific identifier refers to identification information that can distinguish each entity, such as a business ID. For example, the processor 32 extracts first comparison information having an operator ID of '1102401444' from the first DB 10 to generate a metabolic input file (eg, a hash table).
그리고, 제1DB(10)로부터 특정 식별자의 제1비교 정보를 인출하기 전, 상기 프로세서(32)는 정보의 조회하거나 가공을 위해 필요한 각종 관련정보들(예: 정보화센터 및 세무달력, 청서코드, 법인/폐업자와 같은 정보들)을 해시 테이블(hash table)로 메모리(31)에 미리 적재한다.In addition, before withdrawing the first comparison information of the specific identifier from the first DB 10, the processor 32 may provide various related information (eg, information center and tax calendar, blue book code, Information such as corporation / disclosure) is preloaded into the memory 31 as a hash table.
그리고, 상기 프로세서(32)는 상기 힙 메모리(31a)에서 상기 특정 식별자의 제2비교 정보를 검색한다(S103). 예를 들면, 프로세서(32)는 상기 힙 메모리(31a)에 저장된 제2비교 정보 중 사업자 아이디가 '1102401444'인 것을 검색한다.The processor 32 searches for the second comparison information of the specific identifier in the heap memory 31a (S103). For example, the processor 32 searches that the operator ID is '1102401444' among the second comparison information stored in the heap memory 31a.
상기 프로세서(32)는 상기 제1DB(10)로부터 인출된 상기 특정 식별자의 제1비교정보와 상기 힙 메모리(31a)에서 검색된 상기 특정 식별자의 제2비교 정보간의 대사(cross-check)를 수행한다(S104). 상기 프로세서(32)는 제1비교 정보 한 건당 제2비교 정보 한 건씩 '제1비교정보' 건을 기준으로 다중 루프방식으로 대사를 수행한다.The processor 32 performs a cross-check between the first comparison information of the specific identifier fetched from the first DB 10 and the second comparison information of the specific identifier retrieved from the heap memory 31a. (S104). The processor 32 performs metabolism in a multi-loop manner based on the 'first comparison information' case by one second comparison information for each first comparison information.
그리고, 상기 프로세서(32)는 대사결과를 파일로 생성하여 출력한다(S105).The processor 32 generates a metabolic result in a file and outputs it (S105).
상기 대사결과는 제1비교정보 불부합 파일, 제2비교정보 불부합 파일, 제1비교정보 갱신 파일, 제2비교정보 갱신 파일로 출력된다.The metabolic result is output as a first comparison information disagreement file, a second comparison information disagreement file, a first comparison information updating file, and a second comparison information updating file.
상기 대사를 수행한 후 상기 프로세서(32)는 대사를 수행한 상기 특정 식별자의 제1비교 정보와 제2비교 정보를 메모리(31)에서 삭제하여 메모리를 확보한다(S106). 예컨대, 프로세서(32)는 사업자 아이디가 '1102401444'인 제1비교 정보와 제2비교 정보를 메모리(31)에서 제거한다.After performing the metabolism, the processor 32 secures the memory by deleting the first comparison information and the second comparison information of the specific identifier that performed the metabolism from the memory 31 (S106). For example, the processor 32 removes the first comparison information and the second comparison information having the operator ID '1102401444' from the memory 31.
이후, 프로세서(32)는 도4에 도시된 바와 같이, 성능 극대화를 위한 추가 로직을 수행한다. 프로세서(32)는 대사를 수행한 후, 작업이 끝난 해시 테이블 레코드를 삭제한다. 원래, 해시 테이블 검색속도는 대략 O(1)로 DBMS검색속도 O(logn) 보다 훨씬 빠르다. 상기 해시 테이블 삭제로 인해 최대 검색속도 0(1/2)가 구현될 수 있다.The processor 32 then performs additional logic to maximize performance, as shown in FIG. The processor 32 performs the metabolism, and then deletes the finished hash table record. Originally, the hash table search speed is approximately O (1), much faster than the DBMS search speed O (logn). The maximum search rate 0 (1/2) may be implemented by deleting the hash table.
상기한 실시 예에서는 복수의 구간으로 제2비교 정보를 분할하고, 그 분할된 구간별 제2비교 정보에 대해 개별적으로 대사 작업을 수행하는 것을 설명하였으나, 상기한 대사 작업이 병렬로 수행될 수 있다.In the above-described embodiment, the second comparison information is divided into a plurality of sections, and the metabolic operations are separately performed on the divided second comparison information for each divided section. However, the metabolic operations may be performed in parallel. .
도 3에 도시된 바와 같이, 작업 일정 스케줄러에 의해 분산작업 개수가 결정되면, 프로세서(32)는 병렬처리를 위한 선행작업을 수행한다. 상기 프로세서(32)는 결정된 분산작업 개수만큼 스레드를 생성하여 동일한 작업을 병렬로 처리한다.As shown in FIG. 3, when the number of distributed tasks is determined by the task scheduler, the processor 32 performs a preceding task for parallel processing. The processor 32 processes the same tasks in parallel by creating threads for the determined number of distributed tasks.
예를 들어, 특정 식별자의 제1비교 정보를 조회하여 인출하고, 그 인출된 제1비교 정보를 복수의 구간별 제2비교 정보와 각각의 대사 작업을 스레드로 병렬처리한다. 즉, 특정 식별자의 제1비교 정보와 10개 구간 제2비교 정보를 병렬로 대사 처리하므로, 하나의 정보 개체에 대한 대용량 정보의 대사작업이 제2비교 정보 전체에 대해 한번에 수행된다.For example, the first comparison information of the specific identifier is queried and retrieved, and the extracted first comparison information is processed in parallel with the second comparison information for each of the plurality of sections and each metabolic task. That is, since the first comparison information of the specific identifier and the second section information of the ten sections are metabolized in parallel, the metabolic operation of a large amount of information for one information entity is performed on the second comparison information as a whole.
또는, 복수의 식별자별 제1비교 정보와 특정 구간 제2비교 정보의 대사작업이 병렬로 처리된다. 예를 들어, 각각 다른 10개의 사업자 아이디에 대한 제1비교 정보를 인출하고, 그 인출된 사업자 아이디별 제1비교 정보와 특정 구간의 제2비교 정보를 병렬로 동시에 처리한다. 다시 말해서, 복수의 개체에 대한 대용량 정보 대사작업을 동시에 처리하는 것으로, 구간별 제2비교 정보에 대한 대사작업을 반복적으로 수행해야 한다.Alternatively, the metabolic operation of the first comparison information for each of the plurality of identifiers and the second comparison information for the specific section is processed in parallel. For example, the first comparison information for ten different operator IDs is extracted, and the first comparison information for each extracted operator ID and the second comparison information of a specific section are simultaneously processed in parallel. In other words, by simultaneously processing a large amount of information metabolic operations for a plurality of individuals, it is necessary to repeatedly perform the metabolic operation for the second comparison information for each section.
대사작업이 완료되면 프로세서(32)는 각각의 대사작업 스레드로부터 출력되는 대사결과들의 제1비교정보/제2비교정보 불부합 정보를 DB에 삽입하는 복수의 대사결과 삽입작업 스레드로 병렬처리한다. 또한, 대사결과 삽입작업 후 대사결과 업데이트(갱신) 작업들을 스레드로 병렬처리한다. 상기 프로세서(32)는 대사결과 업데이트 작업이 종료되면 대사결과 보고작업을 수행한다.When the metabolic task is completed, the processor 32 performs parallel processing with a plurality of metabolic result insertion task threads for inserting the first comparison information / second comparison information discord information of metabolic results output from each metabolic task thread into a DB. In addition, after inserting the dialogue result, the dialogue result update (update) tasks are executed in parallel. The processor 32 performs a metabolic result report task when the metabolic result update task is completed.
이와 같이, 본 발명은 대사를 수행할 때, 특정 식별자를 기준으로 구간을 분할한 후 병렬로 동일 작업을 분배하여 수행하며, 각 병렬 작업 내에서도 분배된 입력 데이터를 구간으로 순차 분할하여 수행할 수 있다.As described above, the present invention may be performed by dividing a section based on a specific identifier and then distributing the same work in parallel, and sequentially dividing the input data distributed into sections in each parallel work. .

Claims (9)

  1. 제2데이터베이스로부터 제2비교 정보를 조회 및 가공하여 메모리에 선적재하는 단계와,Querying and processing the second comparison information from the second database and reloading it into memory;
    제1데이터베이스로부터 특정 식별자의 제1비교 정보를 조회 및 가공하여 인출하며 상기 메모리에서 상기 특정 식별자의 제2비교 정보를 검색하는 단계와,Retrieving and processing first comparison information of a specific identifier from a first database and retrieving second comparison information of the specific identifier from the memory;
    상기 특정 식별자의 제1비교 정보와 제2비교 정보에 대한 대사를 수행하여 대사결과를 출력하는 단계와,Outputting a metabolic result by performing metabolism on the first comparison information and the second comparison information of the specific identifier;
    상기 대사 수행 후 대사를 수행한 상기 특정 식별자의 제1비교 정보와 제2비교 정보를 메모리에서 삭제하는 단계를 포함하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.And deleting the first comparison information and the second comparison information of the specific identifier that performed the metabolism from the memory after the metabolism is performed.
  2. 제1항에 있어서, 상기 납부정보 적재단계는,According to claim 1, The payment information loading step,
    소정의 분류코드 정보를 상기 메모리에 미리 적재하는 단계와,Loading predetermined classification code information into the memory in advance;
    상기 제2데이터베이스의 제2비교 정보를 적어도 하나 이상의 구간으로 분할하고 그 분할된 구간별 제2비교 정보를 해시테이블로 상기 메모리에 적재하는 단계를 포함하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.Dividing the second comparison information of the second database into at least one section and loading the divided second comparison information into the memory as a hash table in a large capacity based on the relational database. Information batch processing method.
  3. 제1항에 있어서,The method of claim 1,
    상기 특정 식별자의 제1비교 정보 조회 전, 정보의 조회하거나 가공을 위해 필요한 각종 관련정보들을 해시 테이블의 형태로 메모리에 미리 적재하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.A method of batch processing of large amounts of information based on a relational database, characterized in that before loading the first comparison information of the specific identifier, various related information necessary for querying or processing the information is preloaded into a memory in the form of a hash table.
  4. 제1항 내지 제3항 중 어느 한 항에 있어서, 상기 메모리는,The memory of claim 1, wherein the memory comprises:
    힙 메모리로 구현되는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.A method for batch processing of large amounts of information based on a relational database characterized by being implemented in heap memory.
  5. 제1항에 있어서, 상기 특별 식별자는,The method of claim 1, wherein the special identifier,
    사업자 아이디와 같이 각각의 개체를 식별할 수 있는 식별정보인 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.A large-scale information batch processing method based on a relational database, characterized in that the identification information that can identify each entity, such as the operator ID.
  6. 제1항에 있어서, 상기 대사결과 출력단계는,According to claim 1, The metabolic result output step,
    제1비교정보 불부합 파일, 제2비교정보 불부합 파일, 제1비교정보 갱신 파일, 제2비교정보 갱신 파일로 출력되는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.And a first comparison information disagreement file, a second comparison information disagreement file, a first comparison information update file, and a second comparison information update file.
  7. 제6항에 있어서,The method of claim 6,
    상기 제1비교정보 불부합 파일 및 제2비교정보 불부합 파일을 데이터베이스에 삽입하는 단계를 더 포함하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.And inserting the first comparison information disagreement file and the second comparison information disagreement file into a database.
  8. 제6항에 있어서,The method of claim 6,
    상기 제1비교정보 갱신 파일 및 제2비교정보 갱신 파일에 근거하여 제1 및 제2데이터베이스를 갱신하는 단계를 더 포함하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.And updating the first and second databases based on the first comparison information update file and the second comparison information update file.
  9. 제1항에 있어서,The method of claim 1,
    상기 대사를 수행할 때, 상기 특정 식별자를 기준으로 구간을 분할한 후 병렬로 동일작업을 분배하여 수행하며, 각 병렬작업 내에서도 분배된 입력 데이터를 구간으로 순차 분할하여 수행하는 것을 특징으로 하는 관계형 데이터베이스를 기반으로 하는 대용량 정보 배치처리 방법.When the metabolism is performed, a division is performed based on the specific identifier, and then the same job is distributed in parallel, and even in each parallel job, the distributed input data is sequentially divided into sections to perform the relational database. A large amount of information batch processing based on.
PCT/KR2012/008913 2011-10-31 2012-10-29 Method for pre-loading in memory and method for parallel processing for high-volume batch processing WO2013066010A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020110112454A KR20130047431A (en) 2011-10-31 2011-10-31 Method for storaging in memory and pararell-processing for batch process of mass information
KR10-2011-0112454 2011-10-31

Publications (1)

Publication Number Publication Date
WO2013066010A1 true WO2013066010A1 (en) 2013-05-10

Family

ID=48192313

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2012/008913 WO2013066010A1 (en) 2011-10-31 2012-10-29 Method for pre-loading in memory and method for parallel processing for high-volume batch processing

Country Status (2)

Country Link
KR (1) KR20130047431A (en)
WO (1) WO2013066010A1 (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106844740A (en) * 2017-02-14 2017-06-13 华南师范大学 Data pre-head method based on memory object caching system
US10372488B2 (en) 2016-09-16 2019-08-06 Oracle International Corporation Parallel processing using memory mapping

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20200124551A (en) 2019-04-24 2020-11-03 주식회사 데이터스트림즈 A conversion systsm for processing large amounts of data in a batch job

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06119215A (en) * 1992-10-08 1994-04-28 Nec Corp Batch processing method
JPH0895810A (en) * 1994-09-28 1996-04-12 Nec Corp Batch job execution system
JP2003208323A (en) * 2002-01-11 2003-07-25 Hitachi Ltd Method, system and program for executing batch job
KR20030071245A (en) * 2002-02-28 2003-09-03 한국전자통신연구원 Store system and method of dynamic bitmap for huge file system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06119215A (en) * 1992-10-08 1994-04-28 Nec Corp Batch processing method
JPH0895810A (en) * 1994-09-28 1996-04-12 Nec Corp Batch job execution system
JP2003208323A (en) * 2002-01-11 2003-07-25 Hitachi Ltd Method, system and program for executing batch job
KR20030071245A (en) * 2002-02-28 2003-09-03 한국전자통신연구원 Store system and method of dynamic bitmap for huge file system

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10372488B2 (en) 2016-09-16 2019-08-06 Oracle International Corporation Parallel processing using memory mapping
CN106844740A (en) * 2017-02-14 2017-06-13 华南师范大学 Data pre-head method based on memory object caching system

Also Published As

Publication number Publication date
KR20130047431A (en) 2013-05-08

Similar Documents

Publication Publication Date Title
KR101616967B1 (en) Improvements relating to handling and processing of massive numbers of processing instructions in real time
US10002019B2 (en) System and method for assigning a transaction to a serialized execution group based on an execution group limit for parallel processing with other execution groups
US7680793B2 (en) Commit-time ordered message queue supporting arbitrary read and dequeue patterns from multiple subscribers
CN108197306B (en) SQL statement processing method and device, computer equipment and storage medium
CN102354289B (en) Concurrent transaction scheduling method and related device
CN106095878B (en) Divide the database manipulation device and method of table based on point library
CN101996214B (en) Method and device for processing database operation request
US8650224B2 (en) Batching content management operations to facilitate efficient database interactions
CN105956123A (en) Local updating software-based data processing method and apparatus
CN111797168A (en) Block parallel processing method, equipment and storage medium
US9489413B2 (en) Asynchronous global index maintenance during partition maintenance
CN103038718B (en) Localization data affinity system and mixed method
WO2013066010A1 (en) Method for pre-loading in memory and method for parallel processing for high-volume batch processing
CN109388636A (en) Business datum is inserted into database method, apparatus, computer equipment and storage medium
CN114253930A (en) Data processing method, device, equipment and storage medium
CN108073658B (en) Data synchronization system and method
US8285755B2 (en) Partial updating in a database proxy driver
CN107291938A (en) Order Query System and method
US8229946B1 (en) Business rules application parallel processing system
US20200409939A1 (en) Systems and methods for scalable database technology
CN110348693B (en) Multi-robot rapid task scheduling method based on multi-core computing
CN1324466A (en) Method for checking tables paces involved in referential integrity
Zhang et al. Parallel Top-K Algorithms on GPU: A Comprehensive Study and New Methods
US20230031463A1 (en) Method of batch processing data that is stored in multiple tables as a plurality of rows of data by reading out and batch processing data from only a portion of a row from each of tables that is to be used in batch processing logic
US11734282B1 (en) Methods and systems for performing a vectorized delete in a distributed database system

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12844748

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12844748

Country of ref document: EP

Kind code of ref document: A1