CN110362553B

CN110362553B - Method for processing large-batch credit investigation data, electronic device and computer equipment

Info

Publication number: CN110362553B
Application number: CN201910521797.2A
Authority: CN
Inventors: 徐筱莉
Original assignee: Aiyunbao Shanghai Technology Co ltd
Current assignee: Aiyunbao Shanghai Technology Co ltd; Shenzhen Lian Intellectual Property Service Center
Priority date: 2019-06-17
Filing date: 2019-06-17
Publication date: 2023-09-15
Anticipated expiration: 2039-06-17
Also published as: CN110362553A

Abstract

The invention discloses a method for processing large-batch credit investigation data. And respectively storing task data submitted by the clients into a first database and a second database, and taking out one piece of task data from the first database for processing. And inquiring whether task data exists in the first database. And if no task data is detected, updating the task state of the task data in the second database. Checking whether the first database has missing task data which is not processed because of the missing task data, and if the missing task data is detected, reinserting the missing task data into the first database for processing. The method and the electronic device for processing the large-batch credit investigation data can release the pressure of the database, improve the data reading efficiency and ensure the data persistence.

Description

Method for processing large-batch credit investigation data, electronic device and computer equipment

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a method for processing large-scale credit investigation data, an electronic device, a computer device, and a storage medium.

Background

At present, a large amount of data needing to be processed in real time is stored through a database, and asynchronous processing threads continuously access the database to retrieve the data for processing by using a faster frequency so as to ensure the real-time performance and persistence of data processing. To ensure the immediacy of processing, it is necessary to access the database with a very fast frequency to query whether there is a task to be processed, and database pressure is always very high

The consumer loan application clients submit various types of client data, such as various purchased life insurance policies, car insurance policies and other detailed information, and the system needs to sort the submitted data, inquire, audit, calculate, credit rating and the like. The data volume submitted by the clients every day is huge, and the data volume needs to be processed in time by the system.

The system will put the received data (request task) in ORACLE database before, the processing system will access the database without stopping the cycle, inquire whether there is new request task, if there is, process immediately and write the processing result in the database, if not, go on the next cycle. Because the database is hard disk storage, frequent queries to the database create high access pressure, and the database cannot handle other tasks well.

Disclosure of Invention

In view of this, the present invention provides a method for processing large-scale credit investigation data, which uses REDIS storage for processing system reading and also stores in ORACLE hard disk database, and the two are matched, so that rapid reading and writing can be realized, and data can be ensured not to be lost.

In order to achieve the above object, the present invention provides a method for processing large-scale credit investigation data, which is applied to an electronic device, and the method comprises the following steps: respectively storing task data submitted by a client into a first database and a second database; taking out a piece of task data from the first database and processing the task data; inquiring whether task data exists in the first database; if no task data is detected, executing the next query on the first database; updating the task state of the task data in the second database; checking whether missing task data is not processed due to the loss of the task data in the first database; and if missing task data is detected, reinserting the missing task data into the first database for processing.

Further, when the state of the first task data in the second database is not updated beyond the preset time, the first task data is lost.

Further, after the task data is processed, the task data is removed from the first database but not the second database.

Further, the first database stores task data to be processed, and the second database stores all task data.

In order to achieve the above objective, the present invention further provides an electronic device, which includes a storage module, a processing module, a query module, an update module, and an inspection module. The storage module is used for storing task data submitted by a client into the first database and the second database respectively. The processing module is used for taking out a piece of task data from the first database and processing the task data. The query module is used for querying whether task data exist in the first database. The updating module is used for updating the task state of the task data in the second database. The checking module is used for checking whether missing task data is not processed because of task data loss in the first database. And if the missing task data is detected, the processing module reinserts the missing task data into the first database for processing.

Further, after the task data is processed, the query module moves the task data out of the first database but not out of the second database.

To achieve the above object, the present invention further provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for processing large-batch credit data when executing the computer program.

To achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the above-described method for processing high-volume credit data.

Compared with the prior art, the large-batch credit investigation data processing method not only uses REDIS storage for processing system reading, but also can store the REDIS storage in the hard disk database of ORACLE, so that the reading pressure of the ORACLE database is released, the data reading efficiency is improved, and the data persistence can still be ensured.

Drawings

FIG. 1 is a schematic diagram of a hardware architecture of an electronic device according to an embodiment of the invention;

FIG. 2 is a functional block diagram of an electronic device according to an embodiment of the invention; and

FIG. 3 is a flowchart showing steps of a method for processing large-scale credit data according to an embodiment of the invention.

Reference numerals:

electronic device	10
		Memory device	110
Processor and method for controlling the same	120
		Large-batch credit investigation data processing system	130
Memory module	210
		Processing module	220
Query module	230
		Update module	240
Inspection module	250

The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that the description of "first", "second", etc. in this disclosure is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implying an indication of the number of technical features being indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present invention.

Fig. 1 is a schematic diagram of a hardware architecture of an electronic device according to an embodiment of the invention. The electronic device 10, but is not limited to, the memory 110, the processor 120, and the bulk credit data processing system 130 may be communicatively coupled to one another via a system bus, with fig. 1 only showing the electronic device 10 having components 110-130, but it should be understood that not all of the illustrated components are required to be implemented and that more or fewer components may alternatively be implemented.

The memory 110 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or DX memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 110 may be an internal storage unit of the electronic device 10, such as a hard disk or a memory of the electronic device 10. In other embodiments, the memory may also be an external storage device of the electronic apparatus 10, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the electronic apparatus 10. Of course, the memory 110 may also include both an internal memory unit and an external memory device of the electronic apparatus 100. In this embodiment, the memory 110 is typically used to store an operating system and various application software installed on the electronic device 10, such as program codes of the bulk credit data processing system 130. In addition, the memory 110 may be used to temporarily store various types of data that have been output or are to be output.

The processor 120 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 120 is generally used to control the overall operation of the electronic device 10. In this embodiment, the processor 120 is configured to execute the program code or process data stored in the memory 110, for example, execute the bulk credit data processing system 130 or the like.

FIG. 2 is a functional block diagram of an electronic device according to an embodiment of the invention. The electronic device 10 of the embodiment of the invention comprises a storage module 210, a processing module 220, a query module 230, an update module 240 and a checking module 250.

The storage module 210 is configured to store task data (requested task) submitted by a client in the REDIS (REmote DIctionary Server) database, and also in the ORACLE database (hard disk storage).

The REDIS database and the ORACLE database store task data waiting for the application main body to process, and the data content is the same. When the data is not processed, the number of the data stored in the two data storage units is the same. After the data are processed, no data exists in REDIS database, and all data are reserved in ORACLE database. The REDIS database is a memory database, and the read-write speed is fast, but because the data is stored in the memory, the data can be completely disappeared due to sudden power failure or abnormality.

The storage module 210 further pushes the task data waiting for the application to process to the REDIS database in a REDIS supported format, and stores the same task data in the ORACLE database in an ORACLE supported format.

Each task data has a unique number, the numbers in the REDIS database are the same as the numbers in the ORACLE database, and the task data all have a common state: "to be treated".

Task data is stored in a REDIS database for reading by a processing system (such as a large-batch credit data processing system 130 in an electronic device) and also stored in a hard disk database such as an ORACLE, and the REDIS database and the ORACLE are matched, so that quick reading and writing can be realized, and the task data can be prevented from being lost. Because REDIS database is a non-relational database, the LIST structure of REDIS is used, data of different tables of the original database under different screening conditions are stored in different key words, and then the data are taken out by utilizing the characteristic of first-in first-out, so that the data reading efficiency is improved, and meanwhile, the storage and reading pressure of the database are not increased. LIST structure in REDIS is a LIST of left in and right out or right in and left out. Just as a pipe, the ball 1, the ball 2 and the ball 3 are plugged in sequence from the head, and then the ball 1, the ball 2 and the ball 3 can be taken out in sequence from the tail. LIST is such a tube that stores data.

Because of the many categories of task data submitted by clients, there are also multiple processing systems at the same time, one processing system being responsible for processing task data of one category. These different categories of task data all exist in multiple LIST of REDISs. Just as there are a plurality of different colored tubes in a container, each colored tube holds its own colored ball, the system for handling red balls only takes the ball in the red tube at a time, and the system for handling yellow balls only takes the ball in the yellow tube at a time. The screening conditions are the colors of the tubes and the categories of processing tasks. Keywords describing task categories, such as car, life, etc., replace car insurance data and life insurance data, respectively.

The processing module 220 is configured to fetch and process a piece of task data from the REDIS database.

The query module 230 is configured to query the REDIS database for task data. After the query module 230 fetches one piece of task data for processing, the piece of task data is removed from the REDIS database, and only unprocessed task data remains in the REDIS database. Only the task data to be processed will be saved in the REDIS database.

The query module 230 is also configured to query the REDIS database for task data at a frequency of about tens of milliseconds.

If there is more task data, the processing module 220 is caused to continue to fetch the task data for processing. Because REDIS is an in-memory database and stored in a HASH structure, frequent queries will not put stress on REDIS servers.

If no task data is detected, the query module 230 performs the next query on the REDIS database.

The query module 230 queries the ORACLE database at a speed of several tens of milliseconds before querying the REDIS database, and then queries the REDIS database, failing to continuously query the ORACLE database.

The update module 240 updates the task state currently processed in the ORACLE database after the task data processing is completed, which indicates that one task processing is completed. All data and results are left in the ORACLE database for later use.

The task state is stored in the ORACLE database, and after the task data is fetched from the REDIS database and processed, the "to-be-processed" state of the task data in the ORACLE database is updated to the "processed" state.

The checking module 250 periodically checks whether the REDIS database has missing task data due to a loss of task data. Since task data in the memory database is easy to be lost due to faults, it is necessary to check whether the task data is lost in the REDIS database at regular time, so that part of task data is omitted. For example, when the state of the task data in the ORACLE database is not updated beyond a preset time (e.g., half an hour), indicating that the piece of task data has not been processed, the piece of task data may be considered lost. Since normally this piece of task data should be processed for half an hour.

If the checking module 240 detects missing task data, the processing module 220 reinserts the missing task data into the REDIS database after recovering from the failure state for reprocessing.

The timing checks to ensure that the scheme does not result in data being permanently unprocessed due to a fault. Because the task data processed by the provisioning application are all task data stored in the REDIS database, if the REDIS database loses task data due to a fault, the lost task data can never be processed. However, since all task data are stored in backup in the ORACLE database, the ORACLE database can be checked regularly to acquire the data which cannot be processed in time.

In step 301, the REDIS database is used for data management, and task data (request task) submitted by the client is stored in the REDIS database, and is also stored in the ORACLE database (hard disk storage).

The REDIS database and the ORACLE database store task data waiting for the application main body to process, and the data content is the same. When the data is not processed, the number of the data stored in the two data storage units is the same. After the data are processed, no data exists in REDIS database, and all task data are reserved in ORACLE database. The REDIS database is a memory database, and the read-write speed is fast, but because the data is stored in the memory, the data can be completely disappeared due to sudden power failure or abnormality.

Step 301 further includes pushing the task data waiting for processing by the application to the first database in a format supported by REDIS. In the embodiment of the present invention, the first database is a REDIS database.

Step 301 also includes storing the same task data piece by piece in a second database in a format supported by ORACLE. In the embodiment of the present invention, the second database is an ORACLE database.

Each task data has a unique number, the number in the first database is the same as the number in the second database, and the task data all have a common state: "to be treated".

The REDIS database is used for storing task data for reading by a processing system, and the task data are stored in a hard disk database such as ORACLE, and the REDIS database and the ORACLE are matched, so that quick reading and writing can be realized, and the task data can be prevented from being lost. Because REDIS database is a non-relational database, the LIST structure of REDIS is used, data of different tables of the original database under different screening conditions are stored in different key words, and then the data are taken out by utilizing the characteristic of first-in first-out, so that the data reading efficiency is improved, and meanwhile, the storage and reading pressure of the database are not increased. LIST structure in REDIS is a LIST of left in and right out or right in and left out. Just as a pipe, the ball 1, the ball 2 and the ball 3 are plugged in sequence from the head, and then the ball 1, the ball 2 and the ball 3 can be taken out in sequence from the tail. LIST is such a tube that stores data.

Step 302, a piece of task data is fetched from the first database and processed.

In step 303, the processing system queries whether there is any task data in the first database. After the processing system fetches one piece of task data for processing, the task data is moved out of the first database, and only unprocessed task data is left in the first database. Only task data to be processed is stored in the first database.

Step 303 further comprises the processing system querying the first database for the presence of task data at a frequency of about once a few tens of milliseconds.

If there is more task data, the process returns to step 301 to continue to fetch the task data. Because REDIS is an in-memory database and stored in a HASH structure, frequent queries will not put stress on REDIS servers.

Step 304, if no task data is detected, the processing system executes the next query on the first database.

The processing system queries the second database at a rate of tens of milliseconds before querying the first database, and then queries the first database, failing to continuously query the second database.

Step 305, updating the task state currently processed in the second database after the task data processing is completed, which indicates that one task processing is completed. All data and results are left in the second database for later use.

The task state is stored in the second database, and after the task data is taken out from the first database and processed, the 'to-be-processed' state of the task data in the second database is updated to the 'processed' state.

Step 306, checking whether the first database has missing task data due to the task data loss is not processed at regular time. Since task data in the memory database is easy to be lost due to faults, it is necessary to check whether the task data is lost in the first database at regular time, so that part of task data is omitted. For example, when a preset time (e.g., half an hour) is exceeded and the status of the task data in the second database is not updated, it indicates that the task data has not been processed, and it can be considered that the task data is lost. Since normally the task bar data should be processed for half an hour.

If missing task data is detected, the missing task data is reinserted into the first database after recovering from the fault state for reprocessing 307.

The timing checks to ensure that the scheme does not result in data being permanently unprocessed due to a fault. Because the task data processed by the application program are all the task data stored in the first database, if the task data is lost due to failure of the first database, the lost task data can not be processed forever. However, since all task data are stored in the second database in a backup manner, the second database can be checked regularly, and the data which cannot be processed in time can be obtained.

As shown in step 301, since all data to be processed is also stored in the second database, it is possible to query the second database for data that has not been processed for a period of time (e.g., half an hour) at a frequency of 15 minutes or half an hour. The task data to be processed may be a piece of task data submitted by a client to be audited or calculated.

All task data to be processed are stored in the first database, all task data (task data to be processed and processed) are stored in the second database, and the task data are provided by other external systems and are required to be processed by application programs.

Many processes are performed, such as formatting the task data content, extracting key information in the task data, obtaining more information again through other systems according to the task data, performing computation, and the like.

If there is task data that has not been processed for a period of time, this indicates that the data is lost when the first database fails, and if the first database processes the task data, the state of the task data is updated as described in step 305, so the task data that has not been updated for a long period of time is task data that has not been processed. And after detecting the left task data, reinserting the task data into the first database recovered from the fault state for reprocessing.

The invention uses REDIS database to store the data for the processing system to read, and stores the data in the hard disk database such as ORACLE database, which cooperates with each other to realize quick reading and writing and ensure the data not to be lost.

The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

From the above description of the embodiments, it is clear that the above embodiment method may be implemented by means of software plus a necessary general purpose hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.

The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims

1. The method for processing the large-batch credit information is applied to the electronic device and is characterized by comprising the following steps:

respectively storing the received task data into a first database and a second database, wherein the first database is REDIS database, and the second database is ORACLE database;

taking out a piece of task data from the first database and processing the task data;

inquiring whether task data exists in the first database;

if no task data is detected, executing the next query on the first database;

updating the task state of the task data in the second database;

checking whether missing task data is not processed due to the loss of the task data in the first database; and

If missing task data is detected, reinserting the missing task data into the first database for processing;

when the state of the first task data in the second database is not updated beyond the preset time, the first task data is lost;

after the task data is processed, the task data is moved out of the first database but not moved out of the second database.

2. The method of claim 1, wherein the first database holds task data to be processed and the second database holds all task data.

3. An electronic device, comprising:

the storage module is used for respectively storing task data submitted by a client into a first database and a second database, wherein the first database is a REDIS database, and the second database is an ORACLE database;

the processing module is used for taking out a piece of task data from the first database and processing the task data;

the query module is used for querying whether task data exist in the first database;

the updating module is used for updating the task state of the task data in the second database; and

The checking module is used for checking whether the first database has missing task data due to the loss of the task data and the missing task data is not processed;

if missing task data is detected, the processing module reinserts the missing task data into the first database for processing;

after the task data is processed, the query module moves the task data out of the first database but not out of the second database.

4. The electronic device of claim 3, wherein the first database holds task data to be processed and the second database holds all of the task data.

5. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method for processing bulk credit data of any one of claims 1 to 2 when the computer program is executed.

6. A computer-readable storage medium having stored thereon a computer program, characterized by: the computer program, when executed by a processor, implements the steps of the method for processing bulk credit data of any of claims 1 to 2.