CN111125107A - Data processing method, device, electronic equipment and medium - Google Patents

Data processing method, device, electronic equipment and medium Download PDF

Info

Publication number
CN111125107A
CN111125107A CN201911343605.XA CN201911343605A CN111125107A CN 111125107 A CN111125107 A CN 111125107A CN 201911343605 A CN201911343605 A CN 201911343605A CN 111125107 A CN111125107 A CN 111125107A
Authority
CN
China
Prior art keywords
data object
identification information
information
data
present disclosure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911343605.XA
Other languages
Chinese (zh)
Inventor
张麟龙
孙俊
陈琪
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
JD Digital Technology Holdings Co Ltd
Original Assignee
JD Digital Technology Holdings Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by JD Digital Technology Holdings Co Ltd filed Critical JD Digital Technology Holdings Co Ltd
Priority to CN201911343605.XA priority Critical patent/CN111125107A/en
Publication of CN111125107A publication Critical patent/CN111125107A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2228Indexing structures
    • G06F16/2255Hash tables
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/22Indexing; Data structures therefor; Storage structures
    • G06F16/2219Large Object storage; Management thereof

Abstract

The data processing method comprises the steps of storing identification information and reference information of data objects into a memory database, responding to a received attribute modification task, determining identification information of the data objects needing to modify attribute information from the memory database according to the reference information, and modifying the attribute information of the data objects in a disk database based on the identification information of the data objects needing to modify the attribute information. The present disclosure also provides a data processing apparatus, an electronic device, and a computer-readable storage medium.

Description

Data processing method, device, electronic equipment and medium
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, an electronic device, and a medium.
Background
In the context of massive amounts of data, limited by storage and computing capabilities, simple processing operations can also become very time consuming. For example, attribute modification for large volumes of data stored in disk database servers can take a significant amount of time. If the timeliness requirement of the modification is high, it loses meaning even if the modification is waited for to be completed. Therefore, how to improve the processing efficiency of large-batch data is an urgent problem to be solved.
Disclosure of Invention
In view of the above, the present disclosure provides a data processing method, apparatus, electronic device and medium.
One aspect of the disclosed embodiments provides a data processing method, including storing identification information and reference information of a data object in a memory database, determining, in response to receiving an attribute modification task, identification information of a data object whose attribute information needs to be modified from the memory database according to the reference information, and modifying attribute information of a data object in a disk database based on the identification information of the data object whose attribute information needs to be modified.
Optionally, the method further includes monitoring a data object change message, and parsing the data object change message to obtain the identification information and the reference information of the data object.
Optionally, the storing the identification information and the reference information of the data object into the memory database includes storing the identification information and the attribute information of the data object into the memory database according to the batch according to the identification information of the data object.
Optionally, the memory database is a key-value pair database, the storing the identification information and the reference information of the data object into the memory database according to the identification information of the data object and the batch includes determining a hash value of the identification information of the data object, and storing the identification information and the reference information of the data object into a value in a data structure form with the hash value as a key.
Optionally, the data structure comprises an ordered set.
Optionally, the ordered set comprises a skip list.
Optionally, the reference information includes an expiration time of the data object, and the attribute information includes validity of the data object.
Another aspect of an embodiment of the present disclosure provides a data processing apparatus including a storage module, a determination module, and a modification module. And the storage module is used for storing the identification information and the reference information of the data object into the memory database. And the determining module is used for responding to the received attribute modification task and determining the identification information of the data object of which the attribute information needs to be modified from the memory database according to the reference information. And the modification module is used for modifying the attribute information of the data object in the disk database based on the identification information of the data object of which the attribute information needs to be modified.
Another aspect of an embodiment of the present disclosure provides an electronic device comprising at least one processor and at least one memory storing one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the at least one processor, cause the processor to perform the method as described above.
Another aspect of the embodiments of the present disclosure provides a computer-readable storage medium storing computer-executable instructions for implementing the method as described above when executed.
Another aspect of embodiments of the present disclosure provides a computer program comprising computer executable instructions for implementing the method as described above when executed.
According to the method, the identification information and the reference information of the data objects are stored in the memory database, when the attribute modification task is received, the data objects of which the attribute information needs to be modified can be determined according to the reference information in the memory database, so that the attribute information of the data objects in the disk database is modified, the process of searching the data objects needing to be modified in the disk database is avoided, and the efficiency of modifying the attributes of the data objects in batches is effectively improved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically shows an application scenario of a data processing method according to an embodiment of the present disclosure;
FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates a diagram of a storage structure in an in-memory database according to an embodiment of the present disclosure;
FIG. 4 schematically shows a schematic diagram of a data processing apparatus according to an embodiment of the present disclosure; and
FIG. 5 schematically shows a block diagram of a computer system suitable for a data processing system according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). It will be further understood by those within the art that virtually any disjunctive word and/or phrase presenting two or more alternative terms, whether in the description, claims, or drawings, should be understood to contemplate the possibilities of including one of the terms, either of the terms, or both terms. For example, the phrase "a or B" should be understood to include the possibility of "a" or "B", or "a and B".
The embodiment of the disclosure provides a data processing method, which includes storing identification information and reference information of a data object in a memory database, determining identification information of the data object of which the attribute information needs to be modified from the memory database according to the reference information in response to receiving an attribute modification task, and modifying attribute information of the data object in a disk database based on the identification information of the data object of which the attribute information needs to be modified.
Fig. 1 schematically shows an application scenario of a data processing method according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of an application scenario in which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, but does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server cluster 105. The network 104 serves to provide a medium of communication links between the terminal devices 101, 102, 103 and the server cluster 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.
A user may use the terminal devices 101, 102, 103 to interact with the server cluster 105 over the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have installed thereon various communication client applications, such as shopping-like applications, web browser applications, search-like applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only).
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server cluster 105 may be a cluster of servers providing various services, such as a background management server cluster (for example only) that provides support for websites browsed by users using the terminal devices 101, 102, 103. The background management server cluster may analyze and perform other processing on the received data such as the user request, and feed back a processing result (for example, a webpage, information, or data obtained or generated according to the user request) to the terminal device.
Various database applications may be included in the Server cluster 105, including disk databases (e.g., SQL Server) and in-memory databases (e.g., Redis), in accordance with embodiments of the present disclosure. Among them, the Redis database is a distributively deployable database that stores data in a Key-Value pair (Key-Value) manner.
It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the server cluster 105. Accordingly, the data processing apparatus provided by the embodiments of the present disclosure may be generally disposed in the server cluster 105.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 to S230.
In operation S210, the identification information and the reference information of the data object are stored in the in-memory database.
According to the embodiment of the disclosure, the data object is stored in the disk database, and the reference information of the disk database is also stored in the disk database as a basis for modifying the data object. The reference information may also be one of the attributes of the data object. In the related art, the data object is modified according to the reference information in the disk database, and a large amount of read-write operation needs to be performed in the disk database, so that not only is time consumed, but also the connection resources of the database are occupied. The method of the embodiment of the disclosure imports the identification information and the reference information of the data object into the memory database to accelerate the processing speed and alleviate the occupation of the database resources. The identification information of the data object may be a unique identification code of the data object, and one data object may be uniquely determined according to the identification information.
For example, the data object may be a coupon, the identification information of the data object may be a coupon code of the coupon, the reference information may be an expiration time of the coupon, and the determination of whether to modify the state of the coupon, which is valid, may be made periodically based on the expiration time of the coupon. For example, daily modifications to ensure that the status of the coupons in the daily disk database is correct.
According to the embodiment of the disclosure, a data object change message can be monitored, and the data object change message is analyzed to obtain the identification information and the reference information of the data object. For example, when a data object is generated or an attribute of an existing data object is changed, a data object change message may be generated, and the method of the embodiment of the present disclosure may monitor the message and add a new data object or modify reference information of the data object in the in-memory database according to the content of the message.
According to an embodiment of the present disclosure, the storing the identification information and the reference information of the data object into the memory database includes storing the identification information and the attribute information of the data object into the memory database according to the batch according to the identification information of the data object. The identification information and the reference information of the data objects are stored in the memory database in batches, so that batch execution is allowed during processing, and different batches can be executed in parallel, thereby further shortening the time required by one processing operation.
Fig. 3 schematically illustrates a schematic diagram of a storage structure in an in-memory database according to an embodiment of the present disclosure.
As shown in fig. 3, the data in the in-memory database may be stored in the form of key-value pairs. In the disclosed embodiments, the values are data having a certain data structure.
According to the embodiment of the present disclosure, the storing, according to the identification information of the data object, the identification information and the reference information of the data object into the memory database according to the batch includes determining a hash value of the identification information of the data object, and storing the identification information and the reference information of the data object into a value in a data structure form with the hash value as a key.
As shown in fig. 3, the data structure may be an ordered set, and in particular, the ordered set may be implemented in the form of a skip table, according to an embodiment of the present disclosure. For operations such as increasing, deleting, checking, changing and the like, the data structure of the jump table supports the time complexity of O (logN), so that data searching can be efficiently carried out on mass data.
Taking the skip list with the key of 0 in fig. 3 as an example, assuming that the number 22 needs to be searched, the number 7 can be searched in the first layer data first, and if the judgment is that 22 is greater than 7, the number 19 is searched continuously, and if the judgment is that 22 is greater than 19, the number 26 is searched continuously, and if the judgment is that 22 is less than 26, the position of 19 sinks into the second layer number to search continuously, and the next number 22 is searched, and the process is ended.
In fact, the skip list can be used to store any ordered data, such as time data for recording time of day. For example, the reference information may include an expiration time of the data object, and the attribute information includes validity of the data object. The state of validity of the coupon in the disk database can be modified by storing the expiration time of the coupon into the jump table and periodically searching for the expiration data in the jump table.
When storing data in the skip list, the number of layers in which the data is stored may be determined in a random manner. Fig. 3 illustrates a skip table of only two layers, and embodiments of the present disclosure may use a skip table with more layers.
In the embodiment illustrated in fig. 3, there are three key value pairs, and processing operations can be executed in batches according to different keys, so as to implement parallel processing, improve processing efficiency, and reduce operational risk.
Reference is made back to fig. 2. In operation S220, in response to receiving the attribute modification task, identification information of the data object whose attribute information needs to be modified is determined from the in-memory database according to the reference information. A search may be conducted, for example, in the manner described above with reference to fig. 3, to determine whether an expired coupon exists and, if so, to determine identification information of the expired coupon.
In operation S230, attribute information of the data object in the disk database is modified based on the identification information of the data object whose attribute information needs to be modified. After the identification information is determined, the attribute information of the data object can be directly searched and modified in the disk database, so that the process of searching for the expired data object in the disk database is avoided, the processing efficiency is improved, and the occupation of the disk database connection resources is reduced.
Based on the same inventive concept, the embodiment of the present disclosure further provides a data processing apparatus, and the data processing apparatus according to the embodiment of the present disclosure is described below with reference to fig. 4.
Fig. 4 schematically shows a block diagram of a data processing apparatus 400 according to an embodiment of the present disclosure.
As shown in fig. 4, the data processing apparatus 400 includes a storage module 410, a determination module 420, and a modification module 430. The apparatus 400 may perform the various methods described above with reference to fig. 2.
The storing module 410, for example, performs the operation S210 described above with reference to fig. 2, to store the identification information and the reference information of the data object in the in-memory database.
The determining module 420, for example, performs operation S220 described above with reference to fig. 2, and is configured to determine, according to the reference information, identification information of a data object whose attribute information needs to be modified from the in-memory database in response to receiving the attribute modification task.
The modification module 430, for example, performs operation S230 described above with reference to fig. 2, to modify the attribute information of the data object in the disk database based on the identification information of the data object whose attribute information needs to be modified.
According to the embodiment of the present disclosure, the apparatus may further include a monitoring module, configured to monitor a data object change message, and parse the data object change message to obtain the identification information and the reference information of the data object.
According to an embodiment of the present disclosure, the storing the identification information and the reference information of the data object into the memory database includes storing the identification information and the attribute information of the data object into the memory database according to the batch according to the identification information of the data object.
According to the embodiment of the present disclosure, the memory database is a key-value pair database, and the storing, according to the identification information of the data object, the identification information and the reference information of the data object into the memory database according to the batch includes determining a hash value of the identification information of the data object, and storing the identification information and the reference information of the data object into a value in a data structure form with the hash value as a key.
According to an embodiment of the present disclosure, the data structure includes an ordered set.
According to an embodiment of the present disclosure, the ordered set includes a skip table.
According to an embodiment of the present disclosure, the reference information includes an expiration time of the data object, and the attribute information includes validity of the data object.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any of the storage module 410, the determination module 420, the modification module 430, and the listening module may be combined in one module to be implemented, or any one of them may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the storage module 410, the determination module 420, the modification module 430, and the listening module may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware. Alternatively, at least one of the storage module 410, the determination module 420, the modification module 430 and the listening module may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
FIG. 5 schematically illustrates a block diagram of a computer system suitable for implementing the data processing method and apparatus according to an embodiment of the present disclosure. The computer system illustrated in FIG. 5 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure. The computer system shown in fig. 5 may be implemented as a server cluster including at least one processor (e.g., processor 501) and at least one memory (e.g., storage portion 508).
As shown in fig. 5, a computer system 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)502 or a program loaded from a storage section 508 into a Random Access Memory (RAM) 503. The processor 501 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 501 may also include onboard memory for caching purposes. Processor 501 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.
In the RAM 503, various programs and data necessary for the operation of the system 500 are stored. The processor 501, the ROM 502, and the RAM 503 are connected to each other by a bus 504. The processor 501 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 502 and/or the RAM 503. Note that the programs may also be stored in one or more memories other than the ROM 502 and the RAM 503. The processor 501 may also perform various operations of method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, system 500 may also include an input/output (I/O) interface 505, input/output (I/O) interface 505 also being connected to bus 504. The system 500 may also include one or more of the following components connected to the I/O interface 505: an input portion 506 including a keyboard, a mouse, and the like; an output portion 507 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage portion 508 including a hard disk and the like; and a communication section 509 including a network interface card such as a LAN card, a modem, or the like. The communication section 509 performs communication processing via a network such as the internet. The driver 510 is also connected to the I/O interface 505 as necessary. A removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 510 as necessary, so that a computer program read out therefrom is mounted into the storage section 508 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 509, and/or installed from the removable medium 511. The computer program, when executed by the processor 501, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable medium, which may be embodied in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer readable medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to embodiments of the present disclosure, a computer readable medium may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer-readable signal medium may include a propagated data signal with computer-readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, optical fiber cable, radio frequency signals, etc., or any suitable combination of the foregoing.
For example, according to embodiments of the present disclosure, a computer-readable medium may include ROM 502 and/or RAM 503 and/or one or more memories other than ROM 502 and RAM 503 described above.
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.

Claims (10)

1. A method of data processing, comprising:
storing the identification information and the reference information of the data object into a memory database;
in response to receiving the attribute modification task, determining identification information of the data object of which the attribute information needs to be modified from the memory database according to the reference information; and
and modifying the attribute information of the data object in the disk database based on the identification information of the data object of which the attribute information needs to be modified.
2. The method of claim 1, further comprising:
monitoring a data object change message;
and analyzing the data object change message to obtain the identification information and the reference information of the data object.
3. The method of claim 1, wherein storing the identification information and the reference information of the data object in the in-memory database comprises:
and storing the identification information and the attribute information of the data objects into a memory database according to the identification information of the data objects and the batches.
4. The method of claim 3, wherein the in-memory database is a key-value pair database, and the storing the identification information of the data object and the reference information into the in-memory database according to the identification information of the data object by batch comprises:
determining a hash value of identification information of the data object;
and storing the identification information and the reference information of the data object into a value in a data structure form taking the hash value as a key.
5. The method of claim 4, wherein the data structure comprises an ordered set.
6. The method of claim 5, wherein the ordered set comprises a skip list.
7. The method of claim 1, wherein the reference information comprises an expiration time of a data object and the attribute information comprises a validity of the data object.
8. A data processing apparatus comprising:
the storage module is used for storing the identification information and the reference information of the data object into a memory database;
the determining module is used for responding to the received attribute modification task and determining the identification information of the data object of which the attribute information needs to be modified from the memory database according to the reference information; and
and the modification module is used for modifying the attribute information of the data object in the disk database based on the identification information of the data object of which the attribute information needs to be modified.
9. An electronic device, comprising:
a processor; and
a memory having computer readable instructions stored thereon that, when executed by the processor, cause the processor to perform the method of any of claims 1-7.
10. A computer readable storage medium having computer readable instructions stored thereon which, when executed by a processor, cause the processor to perform the method of any of claims 1 to 7.
CN201911343605.XA 2019-12-23 2019-12-23 Data processing method, device, electronic equipment and medium Pending CN111125107A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911343605.XA CN111125107A (en) 2019-12-23 2019-12-23 Data processing method, device, electronic equipment and medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911343605.XA CN111125107A (en) 2019-12-23 2019-12-23 Data processing method, device, electronic equipment and medium

Publications (1)

Publication Number Publication Date
CN111125107A true CN111125107A (en) 2020-05-08

Family

ID=70501498

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911343605.XA Pending CN111125107A (en) 2019-12-23 2019-12-23 Data processing method, device, electronic equipment and medium

Country Status (1)

Country Link
CN (1) CN111125107A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111949252A (en) * 2020-07-23 2020-11-17 网易(杭州)网络有限公司 Data processing method and device and electronic equipment
CN112434182A (en) * 2020-10-29 2021-03-02 北京奇艺世纪科技有限公司 Data storage and acquisition method, device, server and storage medium
CN112650559A (en) * 2020-12-29 2021-04-13 贝壳技术有限公司 Data processing method and device, computer readable storage medium and electronic equipment
CN113727208A (en) * 2021-08-30 2021-11-30 杭州云合智网技术有限公司 Communication method, device, equipment and medium for quickly reading and writing parameter information of optical module
CN113779021A (en) * 2021-02-03 2021-12-10 西安京迅递供应链科技有限公司 Data processing method, device, computer system and readable storage medium
CN113806372A (en) * 2021-09-29 2021-12-17 中国平安人寿保险股份有限公司 New data information construction method and device, computer equipment and storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104468773A (en) * 2014-12-04 2015-03-25 北京国双科技有限公司 Data processing method and server
CN105022757A (en) * 2014-04-29 2015-11-04 腾讯科技(深圳)有限公司 Webpage revision method and webpage revision device
CN107133309A (en) * 2017-04-28 2017-09-05 东软集团股份有限公司 Storage, querying method and the device of flow instance, storage medium and electronic equipment
CN107679081A (en) * 2017-08-30 2018-02-09 平安科技(深圳)有限公司 Information amending method, device, computer equipment and computer-readable recording medium
CN110377610A (en) * 2019-06-27 2019-10-25 腾讯科技(深圳)有限公司 A kind of database update method, apparatus, equipment and medium based on cloud platform
CN110414965A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Information displaying method, device for displaying information, electronic equipment and medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105022757A (en) * 2014-04-29 2015-11-04 腾讯科技(深圳)有限公司 Webpage revision method and webpage revision device
CN104468773A (en) * 2014-12-04 2015-03-25 北京国双科技有限公司 Data processing method and server
CN107133309A (en) * 2017-04-28 2017-09-05 东软集团股份有限公司 Storage, querying method and the device of flow instance, storage medium and electronic equipment
CN107679081A (en) * 2017-08-30 2018-02-09 平安科技(深圳)有限公司 Information amending method, device, computer equipment and computer-readable recording medium
CN110377610A (en) * 2019-06-27 2019-10-25 腾讯科技(深圳)有限公司 A kind of database update method, apparatus, equipment and medium based on cloud platform
CN110414965A (en) * 2019-07-30 2019-11-05 中国工商银行股份有限公司 Information displaying method, device for displaying information, electronic equipment and medium

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111949252A (en) * 2020-07-23 2020-11-17 网易(杭州)网络有限公司 Data processing method and device and electronic equipment
CN111949252B (en) * 2020-07-23 2023-09-08 网易(杭州)网络有限公司 Data processing method and device and electronic equipment
CN112434182A (en) * 2020-10-29 2021-03-02 北京奇艺世纪科技有限公司 Data storage and acquisition method, device, server and storage medium
CN112650559A (en) * 2020-12-29 2021-04-13 贝壳技术有限公司 Data processing method and device, computer readable storage medium and electronic equipment
CN112650559B (en) * 2020-12-29 2021-09-28 贝壳找房(北京)科技有限公司 Data processing method and device, computer readable storage medium and electronic equipment
CN113779021A (en) * 2021-02-03 2021-12-10 西安京迅递供应链科技有限公司 Data processing method, device, computer system and readable storage medium
CN113727208A (en) * 2021-08-30 2021-11-30 杭州云合智网技术有限公司 Communication method, device, equipment and medium for quickly reading and writing parameter information of optical module
CN113806372A (en) * 2021-09-29 2021-12-17 中国平安人寿保险股份有限公司 New data information construction method and device, computer equipment and storage medium
CN113806372B (en) * 2021-09-29 2024-02-06 中国平安人寿保险股份有限公司 New data information construction method, device, computer equipment and storage medium

Similar Documents

Publication Publication Date Title
CN111125107A (en) Data processing method, device, electronic equipment and medium
CN112463729B (en) Data file warehousing method and device, electronic equipment and medium
CN112597126A (en) Data migration method and device
CN109218041B (en) Request processing method and device for server system
CN110866031B (en) Database access path optimization method and device, computing equipment and medium
CN110795331A (en) Software testing method and device
CN112948138A (en) Method and device for processing message
CN112100211B (en) Data storage method, apparatus, electronic device, and computer readable medium
CN107256244B (en) Data processing method and system
CN111460020B (en) Method, device, electronic equipment and medium for resolving message
CN113132400B (en) Business processing method, device, computer system and storage medium
CN112131257B (en) Data query method and device
CN114780361A (en) Log generation method, device, computer system and readable storage medium
CN110705935B (en) Logistics document processing method and device
CN113986833A (en) File merging method, system, computer system and storage medium
CN110309121B (en) Log processing method and device, computer readable medium and electronic equipment
CN109213815B (en) Method, device, server terminal and readable medium for controlling execution times
CN111901403A (en) Session management method and apparatus, electronic device, and medium
CN113378346A (en) Method and device for model simulation
CN113535768A (en) Production monitoring method and device
CN112799863A (en) Method and apparatus for outputting information
CN112988857A (en) Service data processing method and device
CN109240878B (en) Data processing method and device
CN111736976B (en) Task processing method, device, computing equipment and medium
CN114268558B (en) Method, device, equipment and medium for generating monitoring graph

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176

Applicant after: Jingdong Technology Holding Co.,Ltd.

Address before: Room 221, 2 / F, block C, 18 Kechuang 11th Street, Daxing District, Beijing, 100176

Applicant before: JINGDONG DIGITAL TECHNOLOGY HOLDINGS Co.,Ltd.

CB02 Change of applicant information