CN113779021A - Data processing method, device, computer system and readable storage medium - Google Patents
Data processing method, device, computer system and readable storage medium Download PDFInfo
- Publication number
- CN113779021A CN113779021A CN202110150449.6A CN202110150449A CN113779021A CN 113779021 A CN113779021 A CN 113779021A CN 202110150449 A CN202110150449 A CN 202110150449A CN 113779021 A CN113779021 A CN 113779021A
- Authority
- CN
- China
- Prior art keywords
- target data
- data
- identification information
- target
- present disclosure
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title abstract description 32
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000000034 method Methods 0.000 claims abstract description 34
- 238000012795 verification Methods 0.000 claims abstract description 34
- 238000004590 computer program Methods 0.000 claims abstract description 27
- 230000008569 process Effects 0.000 claims abstract description 11
- 238000000605 extraction Methods 0.000 claims description 25
- 230000002159 abnormal effect Effects 0.000 claims description 18
- 238000004364 calculation method Methods 0.000 claims description 13
- 230000015654 memory Effects 0.000 claims description 11
- 238000004806 packaging method and process Methods 0.000 claims description 7
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 5
- 238000010586 diagram Methods 0.000 description 10
- 238000004891 communication Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000012546 transfer Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004140 cleaning Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000000758 substrate Substances 0.000 description 2
- 108010001267 Protein Subunits Proteins 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2228—Indexing structures
- G06F16/2255—Hash tables
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24552—Database cache management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present disclosure provides a data processing method, including: verifying the received data to be processed and determining target data passing the verification; marking the target data to generate identification information of the target data; determining attribute characteristics of target data; and caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information receives the target data and processes the target data. The present disclosure also provides a data processing apparatus, a computer system, a readable storage medium, and a computer program product.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a data processing method, an apparatus, a computer system, a readable storage medium, and a computer program product.
Background
The intelligent terminal is developed very rapidly, new applications are developed endlessly, and many applications require the intelligent terminal to have higher performance, so that the intelligent terminal processor is required to have higher reaction and processing capability on reported abnormal data, and complete functions and better experience can be provided for users.
In implementing the disclosed concept, the inventors found that there are at least the following problems in the related art: after the abnormal data is reported, the abnormal data is directly transferred to the intelligent terminal processor without logic analysis, so that the coupling service is easily operated by mistake or the data is not transferred effectively, and the processing efficiency is low.
Disclosure of Invention
In view of the above, the present disclosure provides a data processing method, an apparatus, a computer system, a readable storage medium, and a computer program product.
One aspect of the present disclosure provides a data processing method, including:
verifying the received data to be processed and determining target data passing the verification;
marking the target data to generate identification information of the target data;
determining attribute characteristics of target data; and
and caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information receives the target data and processes the target data.
According to an embodiment of the present disclosure, wherein the attribute feature includes a hash value;
based on the attribute characteristics, caching and extracting the target data and the identification information of the target data comprises the following steps:
performing hash value calculation on the target data and the identification information of the target data, and determining hash values of the target data and the identification information of the target data;
caching the target data and the identification information of the target data into a cache node corresponding to the hash value based on the hash value of the target data and the identification information of the target data;
extracting data in a target format from a cache node corresponding to the hash value based on the hash value;
the cache node is one or more of a plurality of nodes virtually divided by the cache.
According to an embodiment of the present disclosure, the data processing method further includes:
monitoring the cache state of the cache node, wherein the cache state comprises an extraction state or an idle state;
and under the condition that the cache state of the cache node is an idle state, determining the cache node as a target cache node, wherein the target cache node is characterized by the cache node capable of extracting the target data cached in the cache node and the identification information of the target data.
According to an embodiment of the present disclosure, the data processing method further includes:
determining extraction time for extracting the target data and the identification information of the target data;
and transferring to the next target cache node to extract the target data and the identification information of the target data under the condition that the extraction time is greater than or equal to the preset threshold value.
According to an embodiment of the present disclosure, verifying the received data to be processed, and determining the target data includes:
judging whether the data comprises preset verification information or not;
determining the data as target data under the condition that the data comprises preset verification information; and
and filtering the data which does not comprise the preset verification information under the condition that the data does not comprise the preset verification information.
According to an embodiment of the present disclosure, wherein,
marking the target data, wherein generating the identification information of the target data comprises:
acquiring abnormal reason information of target data; and
marking the target data based on the abnormal reason information of the target data to generate the identification information of the target data, wherein the identification information of the target data comprises the abnormal reason identification information or the identification information of the to-be-processed mode.
According to an embodiment of the present disclosure, the data processing method further includes:
and packaging the target data and the identification information of the target data to generate data in a target format.
Yet another aspect of the present disclosure provides a data processing apparatus including:
the verification module is used for verifying the received data to be processed and determining the target data passing the verification;
the marking module is used for marking the target data to generate the identification information of the target data;
the determining module is used for determining the attribute characteristics of the target data; and
and the caching and extracting module is used for caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information can receive the target data and process the target data.
Yet another aspect of the present disclosure provides a computer system comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the data processing method described above.
Yet another aspect of the present disclosure provides a computer-readable storage medium having stored thereon executable instructions, which when executed by a processor, cause the processor to implement the data processing method described above.
Yet another aspect of the present disclosure provides a computer program product comprising a computer program comprising computer executable instructions for implementing the data processing method described above when executed.
According to the embodiment of the disclosure, the received data to be processed is verified, and the target data passing the verification is determined; marking the target data to generate identification information of the target data; determining attribute characteristics of target data; based on the attribute characteristics, caching and extracting the target data and the identification information of the target data so that a target processor corresponding to the identification information receives the target data and processes the target data, and by means of the technical means of verifying the received data, determining the target data which passes the verification and marking the target data, the identification information of the target data is generated, and invalid circulation and processing of the data are reduced; therefore, the technical problem that after abnormal data are reported in the prior art, the abnormal data are directly transferred to the intelligent terminal processor without logic analysis, coupling service misoperation is easily caused or invalid transfer of the data is caused, and the processing efficiency is low is at least partially solved, and the technical effect of high-efficiency and rapid data processing is further achieved.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
FIG. 1 schematically illustrates an exemplary system architecture to which the data processing methods and apparatus of the present disclosure may be applied;
FIG. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure;
FIG. 3 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure;
FIG. 4 schematically shows an architecture diagram of an application data processing method according to another embodiment of the present disclosure;
FIG. 5 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure; and
FIG. 6 schematically shows a block diagram of a computer system suitable for implementing a data processing method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides a data processing method. The method comprises the following steps: verifying the received data to be processed and determining target data passing the verification; marking the target data to generate identification information of the target data; determining attribute characteristics of target data; and caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information receives the target data and processes the target data.
Fig. 1 schematically illustrates an exemplary system architecture 100 to which the data processing methods and apparatus may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. The terminal devices 101, 102, 103 may have installed thereon various communication client applications, such as a shopping-like application, a web browser application, a search-like application, an instant messaging tool, a mailbox client, and/or social platform software, etc. (by way of example only).
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 105 may be a server providing various services, such as a background management server (for example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and perform other processing on the received data such as the user request, and feed back a processing result (e.g., a webpage, information, or data obtained or generated according to the user request) to the terminal device.
It should be noted that the data processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may be disposed in the server 105 in general. The data processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the data processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.
For example, the data to be processed may be originally stored in any one of the terminal devices 101, 102, or 103 (e.g., the terminal device 101, but not limited thereto), or stored on an external storage device and may be imported into the terminal device 101. Then, the terminal device 101 may transmit the data to be processed to another terminal device, a server, or a server cluster, and execute the data processing method provided by the embodiment of the present disclosure by another server or a server cluster that receives the data to be processed.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically shows a flow chart of a data processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 to S240.
In operation S210, the received data to be processed is verified, and target data that passes the verification is determined.
According to the embodiment of the disclosure, the data to be processed may be abnormal data reported by the user terminal.
According to the embodiment of the disclosure, received data to be processed is verified, and target data passing verification is determined; the data to be processed and the preset information can be matched and verified, and the successfully matched data is the target data passing verification.
But not limited thereto, according to other embodiments of the present disclosure, the received data to be processed may be filtered to remove "dirty data" or invalid data.
According to an optional embodiment of the present disclosure, verifying the received data to be processed may further include performing data cleaning, format unification, and the like on the received data to be processed.
According to the embodiment of the disclosure, the received data to be processed is verified, and the target data passing the verification is determined, so that the subsequent processing process of invalid data is avoided, the processing capacity is improved, and the processing space is saved.
In operation S220, the target data is marked, and identification information of the target data is generated.
According to the embodiment of the disclosure, the identification information generation policy of the target data can be set based on the target processor. The identification information of the target data corresponds to the target processor in the subsequent operation, which is beneficial to transmitting the target data to the corresponding target processor for processing.
In operation S230, attribute characteristics of the target data are determined.
According to the embodiment of the disclosure, the attribute feature of the target data may be the only attribute feature of the target data, and when a plurality of target data exist, the target data may be distinguished by the attribute feature, and the corresponding data may be stored in the corresponding cache node and the data to be extracted may be extracted from the corresponding cache node.
In operation S240, the target data and the identification information of the target data are cached and extracted based on the attribute characteristics, so that the target processor corresponding to the identification information receives the target data and processes the target data.
According to the embodiment of the disclosure, the target data and the identification information of the target data are cached and extracted based on the attribute characteristics, so that a basis is provided for caching and extracting.
According to the embodiment of the disclosure, the target data and the identification information of the target data are cached and extracted, so that the asynchronous data processing mode of the target processor is realized, the data processing capacity of the target processor is improved, the waiting time of a user terminal request is reduced, and the response speed is improved.
According to the embodiment of the disclosure, compared with the prior art that the reported data is directly transferred to the processor, the received data to be processed is verified, marked, cached and extracted, so that the problems of disordered logic processing, invalid transfer in the processor and the like caused by complicated services are avoided, the user experience is further improved, and the operation efficiency is improved.
The method shown in fig. 2 is further described with reference to fig. 3-4 in conjunction with specific embodiments.
Fig. 3 schematically shows a flow chart of a data processing method according to another embodiment of the present disclosure.
As shown in fig. 3, the data processing method includes operations S310 to S360.
In operation S310, data to be processed is received.
In operation S320, the received data to be processed is verified, and the target data that is verified is determined.
In operation S330, the target data is marked, and identification information of the target data is generated.
According to an alternative embodiment of the present disclosure, marking the target data and generating the identification information of the target data may include the following operations. For example, acquiring abnormal reason information of the target data; and marking the target data based on the abnormal reason information of the target data to generate the identification information of the target data, wherein the identification information of the target data comprises the abnormal reason identification information or the identification information of the to-be-processed mode.
According to the embodiment of the disclosure, the main purpose of marking the target data is that, on one hand, the identification information represents the corresponding target processor to be subsequently streamed, and provides basis for the subsequent target data stream, so as to prevent invalid stream; on the other hand, after the identification information is identified, the target data with different identification information is prevented from being transferred to the same target processor due to the coupling of the codes, and finally, the involvement and the change among unrelated services are caused.
In operation S340, the target data and the identification information of the target data are encapsulated to generate data in a target format.
In operation S350, attribute characteristics of the target data are determined.
In operation S360, data in the target format is cached and extracted based on the attribute characteristics.
According to the embodiment of the disclosure, the data to be processed can be streamed data reported from different channels. As in fig. 3, the data to be processed may be data sent by the user terminal through the Web environment, captured data, or data transmitted by the data interface. The data reporting mode is complex, and various channels cause irregular data and have diversity.
According to the embodiment of the disclosure, after the identification information of the target data is generated, the target data and the identification information of the target data are encapsulated to generate the data in the target format. Ensuring the integrity and uniqueness of the data in subsequent streams.
As shown in fig. 3, operation S320 may further include operations S3210, S3221, and S3222, according to other embodiments of the present disclosure.
In operation S3210, it is determined whether the data includes preset authentication information.
In operation S3221, in the case where the data includes the preset authentication information, the data is determined as the target data.
In operation S3222, in the case where the data does not include the preset authentication information, the data not including the preset authentication information is filtered.
According to an embodiment of the present disclosure, the preset verification information may be preset certain field information, such as department field information; however, the present invention is not limited to this, and may be call interface identification information, single number information, or the like.
According to the embodiment of the disclosure, by verifying the data to be processed, the data which cannot be processed by the target processor can be verified and filtered, and the follow-up invalid data circulation and the waste of resources are avoided.
According to an embodiment of the present disclosure, caching and extracting the target data and the identification information of the target data based on the attribute characteristics may specifically include the following operations.
Performing hash value calculation on the target data and the identification information of the target data, and determining hash values of the target data and the identification information of the target data; caching the target data and the identification information of the target data into a cache node corresponding to the hash value based on the hash value of the target data and the identification information of the target data; extracting target data and identification information of the target data from a cache node corresponding to the hash value based on the hash value; the cache node is one or more of a plurality of nodes virtually divided by the cache.
According to an embodiment of the present disclosure, the attribute feature may be a hash value calculated by a hash value.
According to an embodiment of the present disclosure, the hash value calculation of the present disclosure may employ a consistent hash algorithm for calculation.
According to the embodiment of the disclosure, hash value calculation can be performed by taking a set of the ID information of the target data and the identification information of the target data as input to obtain hash values of the target data and the identification information of the target data; however, the present invention is not limited to this, and other target data and feature information of identification information of the target data may be used as input for the hash value calculation as long as unique identification is possible.
According to the embodiment of the present disclosure, it is determined that the target data and the identification information of the target data should be stored in a specific location in the plurality of cache nodes. Not only the hash value of the target data and the identification information of the target data is obtained; hash value calculation is also required for the cache nodes.
According to the embodiment of the present disclosure, the hash value calculation is performed on the cache node, and the IP address of the cache node may be used as an input, but the present disclosure is not limited thereto, and other characteristic information of the cache node may also be used as an input as long as the unique identification information of the corresponding cache node can be used.
According to the embodiments of the present disclosure, the correspondence between the hash values of the target data and the identification information of the target data and the hash value of the cache node may be as long as the correspondence can be mapped to each other. In the present disclosure, a hash ring (hash ring) may be used to obtain a one-to-one mapping relationship.
According to the embodiment of the disclosure, the caching mode is based on the mapping relationship between the hash values of the target data and the identification information of the target data and the hash value of the cache node, and the extraction mode can also be based on the mapping relationship between the hash values of the target data and the identification information of the target data and the hash value of the cache node.
According to the embodiment of the disclosure, a hash value calculation mode is adopted, and the corresponding relation of the hash values is utilized, so that the extraction is rapid, efficient and accurate compared with the mode of traversing and querying from the cache nodes.
According to the optional embodiment of the disclosure, the target data and the identification information of the target data are stored in the cache node, and simultaneously, the states of the target data and the identification information of the target data can be recorded and filed, so that the problem of data loss when the system is unstable or down is solved.
According to an optional embodiment of the present disclosure, the preset data saving cache node is configured to, when the target data and the identification information of the target data are stored in the cache node, cache a copy of the same data in the preset data saving cache node at the same time to play a role of backup.
According to an optional embodiment of the present disclosure, in the operation of extracting data from a cache node and sending the data to a corresponding target processor, an operation of monitoring a cache state of the cache node may also be performed, where the cache state includes an extraction state or an idle state. And under the condition that the cache state of the cache node is an idle state, determining the cache node as a target cache node, wherein the target cache node is characterized by the cache node capable of extracting the target data cached in the cache node and the identification information of the target data.
Fig. 4 schematically shows an architecture diagram of an application data processing method according to another embodiment of the present disclosure.
According to an alternative embodiment of the present disclosure, as shown in fig. 4, the data processing method may extract data in the cache node by designing the dispatchers, e.g., dispatcher 1 and dispatcher 2, and dispatch the data to the corresponding target processor, e.g., target processor 1, target processor 2, or target processor 3, accordingly.
According to the embodiment of the disclosure, the same dispatcher can be correspondingly responsible for extracting and dispatching data in multiple cache nodes to corresponding target processors. In this case, cluster management may be implemented using ZooKeeper (distributed coordination service).
According to the embodiment of the disclosure, the cache nodes can be registered in the ZooKeeper registration center for service registration, for example, the corresponding IP address information and port information are registered in the ZooKeeper registration center, so that the cache states of the cache nodes can be monitored and recorded. The dispatcher determines a target cache node according to the cache state, so that the problem that the cache nodes compete with each other is avoided.
According to an optional embodiment of the present disclosure, the data processing method of the present disclosure may further determine, by designing a time window, extraction time for extracting the target data and the identification information of the target data; and transferring to the next target cache node to extract the target data and the identification information of the target data under the condition that the extraction time is greater than or equal to the preset threshold value.
According to the embodiment of the disclosure, when a plurality of cache nodes are correspondingly matched with one dispatcher, the dispatcher takes too long time to fetch data from one cache node, which may cause that the data cached in other cache nodes cannot be timely transmitted to a target server for processing, resulting in data backlog. And by determining the extraction time and controlling the length of the extraction time, the problem of backlog of other data caused by the fact that a route is occupied by a cache node for a long time is solved.
In summary, the data processing method disclosed by the present disclosure verifies the received data to be processed, and determines the target data that passes the verification, so as to achieve the effect of data cleaning and facilitate the change of the requirements of the business party. The data is processed asynchronously by using a cache technology, so that the data processing capacity of the system is improved. And the data can be processed in different stages according to the service requirements, the data processing mode can be changed in time, and the adjustment is flexible. And preventing data consumption competition by registering the cache node in the zookeeper. The processing efficiency of the abnormal data is improved, and the user experience is improved.
Fig. 5 schematically shows a block diagram of a data processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 5, the data processing apparatus 500 includes a verification module 510, a marking module 520, a determination module 530, and a caching and extraction module 540.
A verification module 510, configured to verify the received data to be processed, and determine target data that passes the verification;
a marking module 520, configured to mark the target data and generate identification information of the target data;
a determining module 530 for determining attribute characteristics of the target data; and
and a caching and extracting module 540, configured to cache and extract the target data and the identification information of the target data based on the attribute characteristics, so that a target processor corresponding to the identification information receives the target data and processes the target data.
According to an embodiment of the present disclosure, wherein the attribute characteristic includes a hash value.
According to an embodiment of the present disclosure, the caching and extraction module 540 includes a hash value calculation unit, a caching unit, and an extraction unit.
The hash value calculation unit is used for performing hash value calculation on the target data and the identification information of the target data and determining the hash values of the target data and the identification information of the target data;
the cache unit is used for caching the target data and the identification information of the target data into a cache node corresponding to the hash value based on the hash value of the target data and the identification information of the target data;
an extraction unit, configured to extract data in a target format from a cache node corresponding to a hash value based on the hash value; the cache node is one or more of a plurality of nodes virtually divided by the cache.
According to an embodiment of the present disclosure, the data processing apparatus 500 further includes a status monitoring module and a status determining module.
The state monitoring module is used for monitoring the cache state of the cache node, wherein the cache state comprises an extraction state or an idle state;
and the state determining module is used for determining the cache node as a target cache node under the condition that the cache state of the cache node is an idle state, wherein the target cache node is characterized by the cache node capable of extracting the target data cached in the cache node and the identification information of the target data.
According to an embodiment of the present disclosure, the data processing apparatus 500 further includes a time determination module and a transfer extraction module.
The time determining module is used for determining the extraction time of the extracted target data and the identification information of the target data;
and the transfer extraction module is used for transferring to the next target cache node to extract the target data and the identification information of the target data under the condition that the extraction time is greater than or equal to the preset threshold value.
According to an embodiment of the present disclosure, the verification module 510 includes a determination unit, a data determination unit, and a filtering unit.
A judging unit configured to judge whether the data includes preset verification information;
the data determining unit is used for determining the data as target data under the condition that the data comprises preset verification information; and
and the filtering unit is used for filtering the data which does not comprise the preset verification information under the condition that the data does not comprise the preset verification information.
According to an embodiment of the present disclosure, the marking module 520 includes an acquisition unit and a generation unit.
An acquisition unit configured to acquire abnormality cause information of target data; and
the generating unit is used for marking the target data based on the abnormal reason information of the target data and generating the identification information of the target data, wherein the identification information of the target data comprises the abnormal reason identification information or the identification information of the to-be-processed mode.
According to an embodiment of the present disclosure, the data processing apparatus 500 further comprises a packaging module.
And the packaging module is used for packaging the target data and the identification information of the target data to generate data in a target format.
Any number of modules, sub-modules, units, sub-units, or at least part of the functionality of any number thereof according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules, sub-modules, units, and sub-units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, sub-modules, units, sub-units according to embodiments of the present disclosure may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in any other reasonable manner of hardware or firmware by integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware implementations. Alternatively, one or more of the modules, sub-modules, units, sub-units according to embodiments of the disclosure may be at least partially implemented as a computer program module, which when executed may perform the corresponding functions.
For example, any number of the validation module 510, the tagging module 520, the determination module 530, and the caching and extraction module 540 may be combined in one module/unit/sub-unit for implementation, or any one of the modules/units/sub-units may be split into multiple modules/units/sub-units. Alternatively, at least part of the functionality of one or more of these modules/units/sub-units may be combined with at least part of the functionality of other modules/units/sub-units and implemented in one module/unit/sub-unit. According to an embodiment of the present disclosure, at least one of the verification module 510, the marking module 520, the determination module 530, and the caching and extraction module 540 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of three implementations of software, hardware, and firmware, or in any suitable combination of any of them. Alternatively, at least one of the verification module 510, the marking module 520, the determination module 530, and the caching and extraction module 540 may be at least partially implemented as a computer program module that, when executed, may perform corresponding functions.
It should be noted that, the data processing apparatus portion in the embodiment of the present disclosure corresponds to the data processing method portion in the embodiment of the present disclosure, and the description of the data processing apparatus portion specifically refers to the data processing method portion, which is not described herein again.
Fig. 6 schematically shows a block diagram of a computer system suitable for implementing the above described method according to an embodiment of the present disclosure. The computer system illustrated in FIG. 6 is only one example and should not impose any limitations on the scope of use or functionality of embodiments of the disclosure.
As shown in fig. 6, a computer system 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM)602 or a program loaded from a storage section 608 into a Random Access Memory (RAM) 603. Processor 601 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 601 may also include onboard memory for caching purposes. Processor 601 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the disclosure.
In the RAM 603, various programs and data necessary for the operation of the system 600 are stored. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. The processor 601 performs various operations of the method flows according to the embodiments of the present disclosure by executing programs in the ROM 602 and/or RAM 603. It is to be noted that the programs may also be stored in one or more memories other than the ROM 602 and RAM 603. The processor 601 may also perform various operations of the method flows according to embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, system 600 may also include an input/output (I/O) interface 605, input/output (I/O) interface 605 also connected to bus 604. The system 600 may also include one or more of the following components connected to the I/O interface 605: an input portion 606 including a keyboard, a mouse, and the like; an output portion 607 including a display such as a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker; a storage section 608 including a hard disk and the like; and a communication section 609 including a network interface card such as a LAN card, a modem, or the like. The communication section 609 performs communication processing via a network such as the internet. The driver 610 is also connected to the I/O interface 605 as needed. A removable medium 611 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 610 as necessary, so that a computer program read out therefrom is mounted in the storage section 608 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication section 609, and/or installed from the removable medium 611. The computer program, when executed by the processor 601, performs the above-described functions defined in the system of the embodiments of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include the ROM 602 and/or RAM 603 described above and/or one or more memories other than the ROM 602 and RAM 603.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the data processing method provided by the embodiments of the present disclosure.
The computer program, when executed by the processor 601, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of a signal on a network medium, downloaded and installed through the communication section 609, and/or installed from the removable medium 611. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (11)
1. A method of data processing, comprising:
verifying the received data to be processed and determining target data passing the verification;
marking the target data to generate identification information of the target data;
determining attribute characteristics of the target data; and
and caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information receives the target data and processes the target data.
2. The method of claim 1, wherein the attribute characteristics comprise hash values;
the caching and extracting the target data and the identification information of the target data based on the attribute characteristics comprises:
performing hash value calculation on the target data and the identification information of the target data, and determining hash values of the target data and the identification information of the target data;
caching the target data and the identification information of the target data into a cache node corresponding to the hash value based on the hash value of the target data and the identification information of the target data;
extracting the data in the target format from a cache node corresponding to the hash value based on the hash value;
the cache node is one or more of a plurality of nodes virtually divided by the cache.
3. The method of claim 2, further comprising:
monitoring the cache state of the cache node, wherein the cache state comprises an extraction state or an idle state;
and under the condition that the caching state of the caching node is an idle state, determining that the caching node is a target caching node, wherein the target caching node is characterized by the caching node capable of extracting the target data cached in the caching node and the identification information of the target data.
4. The method of claim 3, further comprising:
determining extraction time for extracting the target data and the identification information of the target data;
and transferring to the next target cache node to extract the target data and the identification information of the target data under the condition that the extraction time is greater than or equal to a preset threshold value.
5. The method of claim 1, wherein the validating the received data to be processed and determining the target data comprises:
judging whether the data comprises preset verification information or not;
determining the data as target data under the condition that the data comprises preset verification information; and
and filtering the data which does not comprise the preset verification information under the condition that the data does not comprise the preset verification information.
6. The method of claim 1, wherein,
the marking the target data and the generating the identification information of the target data comprises:
acquiring abnormal reason information of the target data; and
marking the target data based on the abnormal reason information of the target data, and generating the identification information of the target data, wherein the identification information of the target data comprises abnormal reason identification information or to-be-processed mode identification information.
7. The method of claim 1, further comprising:
and packaging the target data and the identification information of the target data to generate data in a target format.
8. A data processing apparatus comprising:
the verification module is used for verifying the received data to be processed and determining the target data passing the verification;
the marking module is used for marking the target data to generate the identification information of the target data;
the determining module is used for determining the attribute characteristics of the target data; and
and the caching and extracting module is used for caching and extracting the target data and the identification information of the target data based on the attribute characteristics so that a target processor corresponding to the identification information receives the target data and processes the target data.
9. A computer system, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-7.
10. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 7.
11. A computer program product comprising a computer program comprising computer executable instructions for implementing the method of any one of claims 1 to 7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110150449.6A CN113779021B (en) | 2021-02-03 | 2021-02-03 | Data processing method, device, computer system and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110150449.6A CN113779021B (en) | 2021-02-03 | 2021-02-03 | Data processing method, device, computer system and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113779021A true CN113779021A (en) | 2021-12-10 |
| CN113779021B CN113779021B (en) | 2024-05-17 |
Family
ID=78835585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110150449.6A Active CN113779021B (en) | 2021-02-03 | 2021-02-03 | Data processing method, device, computer system and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113779021B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115225586A (en) * | 2022-07-14 | 2022-10-21 | 中科驭数（北京）科技有限公司 | Data packet transmitting method, device, equipment and computer readable storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049431A (en) * | 2015-06-30 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Data access control method and device |
| CN106851636A (en) * | 2017-01-10 | 2017-06-13 | 南京邮电大学 | A kind of dynamic key false data filter method for being applied to wireless sensor network |
| CN108881397A (en) * | 2018-05-25 | 2018-11-23 | 沈阳东软医疗系统有限公司 | Method, apparatus, equipment and the storage medium of Medical Devices processing information |
| CN109101641A (en) * | 2018-08-20 | 2018-12-28 | 联想(北京)有限公司 | Form processing method, device, system and medium |
| CN110008257A (en) * | 2019-04-10 | 2019-07-12 | 深圳市腾讯计算机系统有限公司 | Data processing method, device, system, computer equipment and storage medium |
| CN111078147A (en) * | 2019-12-16 | 2020-04-28 | 南京领行科技股份有限公司 | Processing method, device and equipment for cache data and storage medium |
| CN111125107A (en) * | 2019-12-23 | 2020-05-08 | 京东数字科技控股有限公司 | Data processing method, device, electronic equipment and medium |
| CN111382146A (en) * | 2018-12-29 | 2020-07-07 | 浙江宇视科技有限公司 | Data storage control method, device, equipment and storage medium |
-
2021
- 2021-02-03 CN CN202110150449.6A patent/CN113779021B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105049431A (en) * | 2015-06-30 | 2015-11-11 | 深圳市深信服电子科技有限公司 | Data access control method and device |
| CN106851636A (en) * | 2017-01-10 | 2017-06-13 | 南京邮电大学 | A kind of dynamic key false data filter method for being applied to wireless sensor network |
| CN108881397A (en) * | 2018-05-25 | 2018-11-23 | 沈阳东软医疗系统有限公司 | Method, apparatus, equipment and the storage medium of Medical Devices processing information |
| CN109101641A (en) * | 2018-08-20 | 2018-12-28 | 联想(北京)有限公司 | Form processing method, device, system and medium |
| CN111382146A (en) * | 2018-12-29 | 2020-07-07 | 浙江宇视科技有限公司 | Data storage control method, device, equipment and storage medium |
| CN110008257A (en) * | 2019-04-10 | 2019-07-12 | 深圳市腾讯计算机系统有限公司 | Data processing method, device, system, computer equipment and storage medium |
| CN111078147A (en) * | 2019-12-16 | 2020-04-28 | 南京领行科技股份有限公司 | Processing method, device and equipment for cache data and storage medium |
| CN111125107A (en) * | 2019-12-23 | 2020-05-08 | 京东数字科技控股有限公司 | Data processing method, device, electronic equipment and medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115225586A (en) * | 2022-07-14 | 2022-10-21 | 中科驭数（北京）科技有限公司 | Data packet transmitting method, device, equipment and computer readable storage medium |
| CN115225586B (en) * | 2022-07-14 | 2024-04-26 | 中科驭数（北京）科技有限公司 | Data packet transmitting method, device, equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113779021B (en) | 2024-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2021259013A1 (en) | Data processing method and apparatus, electronic device, and computer-readable medium | |
| CN115277566B (en) | Load balancing method and device for data access, computer equipment and medium | |
| WO2019192133A1 (en) | Electronic apparatus, data link risk pre-warning method, and storage medium | |
| WO2019075845A1 (en) | Construction method and device for link call relationship, computer device and storage medium | |
| CN109039803A (en) | A kind of method, system and the computer equipment of processing readjustment notification message | |
| JP6758139B2 (en) | Systems and methods for efficient call processing | |
| CN108900627B (en) | Network request method, terminal device and storage medium | |
| CN115357761A (en) | Link tracking method and device, electronic equipment and storage medium | |
| CN112953769B (en) | Data transmission method, device, computer system and readable storage medium | |
| CN113779021A (en) | Data processing method, device, computer system and readable storage medium | |
| CN113191889A (en) | Wind control configuration method, configuration system, electronic device and readable storage medium | |
| CN108289165B (en) | Method and device for realizing camera control based on mobile phone and terminal equipment | |
| CN109933444A (en) | A kind of instant communication method between applying of lodging | |
| CN114257632B (en) | Method and device for reconnecting broken wire, electronic equipment and readable storage medium | |
| CN115934378A (en) | Service data processing method and device, electronic equipment and storage medium | |
| CN113986995A (en) | Request distribution method and device, storage medium and electronic equipment | |
| CN113055350B (en) | Data transmission method, device, equipment and readable storage medium | |
| CN112306791B (en) | Performance monitoring method and device | |
| CN115250276A (en) | Distributed system and data processing method and device | |
| WO2019169696A1 (en) | Platform client data backflow method, electronic apparatus, device, and storage medium | |
| CN111338642A (en) | Method, device, terminal and storage medium for determining application downloading path | |
| CN115174588B (en) | Bandwidth control method, device, apparatus, storage medium and program product | |
| CN114844810B (en) | Heartbeat data processing method, device, equipment and medium | |
| CN115333871B (en) | Firewall operation and maintenance method and device, electronic equipment and readable storage medium | |
| CN112162868B (en) | Data communication method, device and storage medium applied to cluster processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |