CN115811525A - Data exchange and processing method based on distributed architecture - Google Patents
Data exchange and processing method based on distributed architecture Download PDFInfo
- Publication number
- CN115811525A CN115811525A CN202310090854.2A CN202310090854A CN115811525A CN 115811525 A CN115811525 A CN 115811525A CN 202310090854 A CN202310090854 A CN 202310090854A CN 115811525 A CN115811525 A CN 115811525A
- Authority
- CN
- China
- Prior art keywords
- node
- data
- data exchange
- node group
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Collating Specific Patterns (AREA)
Abstract
The invention discloses a data exchange and processing method based on a distributed architecture, which introduces liveness as a node designation basis of a data exchange request sending object, improves the pertinence of data exchange by re-determining a designated node in each data exchange request of the same polling, improves the distributed performance, simultaneously considers the data processing speed of a main node and simplifies the data processing process. The liveness well represents rules and characteristics of data processing of the main node under different scenes and different periods, the problem of probability prediction of each node in the distributed architecture, which is needed, is converted into a relatively quantized liveness calculation problem, and the main node can more quickly and relatively accurately determine an object to be subjected to data exchange when requesting in each data exchange. A communication verification mechanism is introduced, and each designated node transmits a data packet to the main node after the data exchange authority verification of the main node is passed, so that the safety of data exchange and processing is improved.
Description
Technical Field
The invention relates to the technical field of information, in particular to a data exchange and processing method based on a distributed architecture.
Background
The distributed architecture comprises a plurality of nodes, each node has data processing capacity and data storage capacity, and data resources can be shared among the nodes. In order to simplify the distributed computing process, a certain node in the distributed architecture may be designated as a master node, and other nodes may be designated as slave nodes to exchange data to the master node, and the master node performs data processing. In some special scenarios. For example, in an identity recognition scene, the fingerprint, the face and the iris of the same person need to be compared in features, and the identity authentication is judged to be passed after the comparison of the fingerprint, the face and the iris of the same person is successful. In order to facilitate classified storage of the identity characteristic data, the fingerprint image can be stored in a node 1 in a distributed architecture, the face image can be stored in a node 2 in the distributed architecture, the iris image can be stored in a node 3 in the distributed architecture, then one node is designated as a main node in the nodes 1, 2 and 3, the other 2 nodes exchange data to the main node, and the main node performs data processing. However, when the external third party service calls the 3 nodes at the same time with high frequency, the data processing performance of the master node is greatly influenced. To solve this problem, increasing the number of nodes in the distributed architecture is the most direct and effective means to improve the distributed processing capability. However, increasing the number of nodes sacrifices the pertinence of the master node in requesting data exchange, and it is usually necessary to design a complex set of data exchange rules to identify the nodes necessary for the master node to process data from the large number of nodes, and if a communication checking mechanism is added to the data exchange, the complexity of the master node in processing data is higher. Therefore, how to simplify the data processing process of the master node as much as possible while improving the distributed processing capability becomes an urgent technical problem to be solved.
Disclosure of Invention
The invention provides a data exchange and processing method based on a distributed architecture, aiming at improving the distributed performance and simultaneously considering the speed and the accuracy of data exchange and the data processing speed of a main node.
In order to achieve the purpose, the invention adopts the following technical scheme:
a data exchange and processing method based on a distributed architecture is provided, which comprises the following steps:
s1, a node is designated from a node group in a distributed architecture as a main node for data processing;
s2, the service processing module forwards the data exchange request in the same polling received at the main node to a designated node in a designated node group in the distributed architecture;
s3, each appointed node receiving the data exchange request packs the data stored by the appointed node and then sends the packed data to the service processing module;
s4, the service processing module forms a data packet sent by the designated node in each designated node group into a data set requested by the current data exchange request and forwards the data set to the main node;
s5, the master node processes the data of the requested data set and judges whether the data processing result is successfully obtained or not,
if yes, feeding back a data processing result to each designated node corresponding to the data packet generated in the data set, finishing polling and sending the data exchange request, and accumulating the activity of each designated node by '1' after receiving result feedback;
if not, the next data exchange request is initiated under the polling, and then the step S2 is returned until the polling request is finished.
Preferably, the distributed architecture includes a plurality of first node groups respectively corresponding to different first tags, a plurality of second node groups respectively corresponding to different second tags, a plurality of third node groups respectively corresponding to different third tags, and a plurality of fourth node groups respectively corresponding to different fourth tags, and each of the node groups includes at least one node corresponding to the same tag;
in step S1, the method for designating the master node includes the steps of:
s11, calculating the activity degree of each node group, and forming a node group sequence after sequencing from low to high according to the activity degree, wherein each element in the node group sequence carries a label of the corresponding node group;
s12, calculating identity detail characteristic values of the personnel to be checked;
s13, extracting elements which have label corresponding relation with the calculated identity detail characteristic values from the node group sequence in an extraction mode of reserving a sorting sequence to form an extraction sequence;
s14, taking the node with the lowest activity degree in the node group corresponding to the first element in the extraction sequence as the master node.
Preferably, the method for calculating the activity of each node group comprises the following steps:
s111, calculating a weighted summation value of the activity of each node in the node group;
and S112, calculating the product of the weighted summation value and the activity correction coefficient corresponding to the node group as the activity calculated for the node group.
Preferably, in step S2, the method for determining the designated node in each designated node group to which the data exchange request initiated by the master node is forwarded by the service processing includes:
s21, matching label data pairs for representing a group of people from a database by taking labels corresponding to the identity detail characteristic values of the personnel to be checked as matching bases of a specified node group, and taking the node group corresponding to each type of label in the label data pairs as the matched specified node group;
and S22, extracting the nodes with the highest activity left in each appointed node group after node filtering as appointed nodes.
Preferably, in step S23, the method for filtering nodes of the designated node group includes:
s231, the master node performs individual identity characteristic comparison on the data packet sent by the node with the highest liveness in each designated node group, and judges whether the comparison is successful or not,
if yes, generating a termination signal for terminating the data exchange of the specified node group in the polling, and sending the termination signal to the service processing module;
if not, generating a node filtering signal, sending the node filtering signal to the service processing module, and turning to the step S232;
s232, the service processing module filters out the nodes with identity feature comparison in the current data exchange request of step S231 from the designated node group according to the node filtering signal.
Preferably, in step S3, after each designated node that receives the data exchange request verifies the data exchange permission of the master node and passes the data exchange permission, the designated node packages the data stored in the designated node and sends the data to the service processing module, where the permission verification method includes the steps of:
s31, analyzing a main node access path, a label corresponding to the main node and a unique node number of the main node carried in the data exchange request forwarded by the service processing module;
s32, searching the node group to which the main node belongs according to the label corresponding to the main node;
s33, acquiring the access path of each other node in the node group from a database according to the analyzed binding relationship between the access path of the main node and the access positions of other nodes in the node group found in the step S32;
s34, accessing all nodes in the node group and reading the liveness recorded at each node;
s35, judging whether the node with the lowest read liveness is the main node carried in the data exchange request or not,
if yes, judging that the authority verification is passed;
if not, the authority verification is judged to be failed.
The invention has the following beneficial effects:
1. the liveness is introduced as a node designation basis of a data exchange request sending object, and in each data exchange request of the same polling, by re-determining a designated node, the pertinence of data exchange is improved, the distributed performance is improved by increasing the number of nodes, the data processing speed of the main node is considered, and the data processing process is simplified.
2. The liveness well represents the rules and characteristics of the main node for processing data under different scenes and different periods, the problem of probability prediction of each node in the distributed architecture is converted into a relatively quantitative liveness calculation problem, and the main node can determine an object to be subjected to data exchange more quickly, simply and accurately in each data exchange request.
3. A communication verification mechanism is introduced, and each designated node forwards the data packet to the main node through the service processing module after the data exchange authority verification of the main node is passed, so that the safety of data exchange and processing is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a diagram illustrating implementation steps of a data exchange and processing method based on a distributed architecture according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of equally dividing a fingerprint image into a number of rectangular sub-blocks and ordering the rectangular sub-blocks;
FIG. 3 is a schematic diagram illustrating the calculation of the distance between the boundary pixel point and the left vertex of the rectangular sub-block in the fingerprint image;
FIG. 4 is a diagram of calculating sum ratio of iris images
Schematic ofDrawing;
fig. 5 is a schematic diagram of the distributed architecture provided in this embodiment.
Detailed Description
The technical scheme of the invention is further explained by the specific implementation mode in combination with the attached drawings.
Wherein the showings are for the purpose of illustration only and not for the purpose of limiting the same, the same is shown by way of illustration only and not in the form of limitation; to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numerals in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. are used to indicate an orientation or a positional relationship based on that shown in the drawings, it is only for convenience of description and simplification of description, but not to indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and are not to be construed as limitations on the present patent, and specific meanings of the terms may be understood according to specific situations by those of ordinary skill in the art.
In the description of the present invention, unless otherwise explicitly specified or limited, the term "connected" or the like, if appearing to indicate a connection relationship between the components, is to be understood broadly, for example, as being fixed or detachable or integral; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be connected through any combination of two or more members or structures. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
As shown in fig. 2, the distributed architecture provided in this embodiment includes a plurality of node groups, which are respectively a first node group, a second node group, a third node group, and a fourth node group, where each first node group corresponds to a different first tag, each second node group corresponds to a different second tag, each third node group corresponds to a different third tag, and each fourth node group corresponds to a different fourth tag; each node group comprises at least one node corresponding to the same label;
taking the first label as a distance value and value interval, the second label as a quantity interval, the third label as a ratio value and value interval, and the fourth label as a sum value ratio interval as an example, all nodes in the same first node group correspond to the same distance value and value interval, all nodes in the same second node group correspond to the same quantity interval, all nodes in the same third node group correspond to the same ratio value and value interval, and all nodes in the same fourth node group correspond to the same sum value ratio interval.
In this embodiment, the distance value sum is a weighted sum of distances between each boundary pixel point of the fingerprint information of each rectangular sub-block divided in each fingerprint image and the left vertex of the rectangular sub-block, and the first label, i.e., the distance value sum is a numerical value interval in which the distance value sum falls. The present embodiment expresses the distance value and the value as
，
The method is calculated by the following method steps:
a1, equally dividing each edge into a plurality of segments at equal intervals for the width and the height of each fingerprint image under the same crowd classification (such as male teenagers with age groups of 13-17) framed in a rectangular frame selection mode, and referring to fig. 2 for an example of equal division;
a2, starting from each bisection point, connecting lines to the opposite bisection points of opposite sides in a mode of being perpendicular to the side where the starting point is located, so as to disperse the fingerprint image into a plurality of rectangular sub-blocks, taking the rectangular sub-blocks at the left top corners of the image as initial sub-blocks of a standard sequence (the sub-blocks with the sequence of 1 in the figure 2 represent the initial sub-blocks), and marking each rectangular sub-block in a circular sequence in a mode of rotating the standard sequence counterclockwise inwards;
a3, filtering out the fingerprint image which does not carry fingerprint information (the fingerprint information carrying sub-blocks are pixels which do not represent fingerprint information, such as rectangular sub-blocks with the sequence of "1", "2", "12", "18" and "29" in FIG. 2) and rectangular sub-blocks which are fully loaded with fingerprint information, wherein the fingerprint information carrying pixels touch each side of the rectangular sub-blocks (such as the rectangular sub-blocks with the sequence of "8", "36", "22" and "24" in FIG. 2);
a4, searching boundary pixel points (represented by marks P1 and P2 in fig. 3) of the fingerprint information in each rectangular sub-block for each rectangular sub-block remaining after filtering in the step A3, where there are many existing methods for searching boundary pixel points, for example, a point where the fingerprint information inside the rectangular sub-block is interrupted is identified as a boundary pixel point;
a5, calculating the distance (as represented by L1 and L2 in figure 2) between each boundary pixel point and the left vertex (as represented by the mark P0 in figure 2) of the rectangular sub-block, and summing each distance according to the following formula (1) to obtain the distance value corresponding to each rectangular sub-block
：
In the formula (1), the first and second groups of the compound,
representing the second in a fingerprint image
Distance values of the individual rectangular sub-blocks;
a6, calculating the distance value and the value of all the rectangular sub-blocks remaining after the filtering of the step A3 by the following formula (2)
：
In the formula (2), the first and second groups of the compound,
represent
In the calculation of
The weight occupied by hour;
In this embodiment, the number accumulated value is an accumulated value of the number of rectangular sub-blocks in the fingerprint image whose distance value difference is smaller than the difference threshold, and the second label, that is, the number interval is a number interval in which the number accumulated value falls. The present embodiment expresses the number accumulated value as
，
Is calculated byThe method comprises the following steps:
b1, calculated in each fingerprint image and standard fingerprint image
And the distance value difference is calculated by the following formula (3) for two rectangular sub-blocks with the same rank number
：
In the formula (3), the first and second groups of the compound,
indicating participation
In the calculated fingerprint image
Distance values of the individual rectangular sub-blocks;
while that of each rectangular sub-block in a standard fingerprint image
The value is calculated by the following equation (4):
in the formula (4), the first and second groups,
representing the first image set or the second image set
In a fingerprint image
The corresponding rectangular sub-blocks have the distance values of the rectangular sub-blocks with the same row sequence number;
Due to each rectangular sub-block in the standard fingerprint image
The value is the average value of the distance values of the corresponding rectangular sub-blocks of all the fingerprint images in the first image set or the second image set, so that each rectangular sub-block in the standard index image has a corresponding rectangular sub-block in each fingerprint image.
B2, to
Rectangular sub-block columns in fingerprint image smaller than disparity value threshold are quantity accumulation objects: (
The smaller the similarity between two rectangular sub-blocks with position correspondence is), and the fingerprint image is matched with the two rectangular sub-blocks
The number of each rectangular block under the number accumulation condition is accumulated to obtain the number accumulation value associated with each fingerprint image
。
In this embodiment, the ratio sum is a weighted sum of the ratio of the number of face pixels to the total number of pixels in each remaining rectangular block obtained by calculating for each remaining rectangular block after each rectangular block in each face image is filtered by the rectangular block fully loaded and not carrying face information, and the third label, that is, the ratio sum is a numerical interval in which the ratio sum falls. In the present embodiment, the ratio and the value are expressed as
，
The calculating method comprises the following steps:
c1, shooting the face of each person under the same crowd classification at a fixed distance and a fixed angle to obtain a face image with the same size of each person;
c2, equally dividing each edge into a plurality of sections in an equally-spaced mode for the width and the height of each face image;
c3, connecting a line from each bisection point to the opposite bisection points of the opposite sides in a mode of being perpendicular to the side where the departure point is located, dispersing the face image into a plurality of rectangular blocks, taking the rectangular block at the left top corner of the image as an initial block of the labeling sequence, and labeling each rectangular block in a mode of rotating the labeling sequence anticlockwise inwards;
c4, filtering out rectangular blocks which do not carry face information and are fully loaded with the face information in the face image, wherein the fully loaded face information means that pixels representing the face information touch each edge of the rectangular block to which the pixels belong;
the human face image discretization method adopted in the steps C2-C4 is the same as the fingerprint image discretization method recorded in the steps A1-A3, and therefore, the description is omitted.
C5, searching the face pixels in each rectangular block filtered by the step C4, and calculating the number of the searched face pixels and the number of the face pixels in the face image to which the face pixels belong
The ratio of the number of pixels in a tile,is marked as
(ii) a For example, the number of pixels characterizing a face in a rectangular block is 100, to which
A total of 200 pixels in each rectangular block, then
；
C6, calculating the ratio and the value of all the rectangular blocks which are filtered and remained by the step C4 by the following formula (5)
：
In the formula (5), the first and second groups of the chemical reaction materials are selected from the group consisting of,
to represent
In the calculation of
The weight occupied by (c);
In this embodiment, the sum ratio is an area ratio of an iris region in each iris image to the entire image, and the fourth label, that is, the sum ratio interval is a ratio interval in which the area ratio falls. In the present embodiment, the sum ratio is expressed as
，
The method is calculated by the following method steps:
d1, shooting eye images of each person under the same crowd classification at a fixed distance and a fixed angle, and selecting an iris image from each eye image in a rectangular frame selection mode, wherein the selected iris image is shown in a figure 4, a circle in the figure 4 is an iris, and an external rectangle is a rectangular frame of the frame selected iris;
d2, halving each side of the width and the height of each iris image, and connecting the unequal points which are not opposite to each other to obtain a space quadrangle (indicated by a reference mark Q1 in figure 4);
d3, calculating the areas of the space quadrangle and the rectangular frame of the iris image, and respectively recording the areas as
、
；
D4, halving each side of the space quadrilateral (such as halving the side labeled "q1" shown in FIG. 4), and then from each bisecting point
Starting from a line perpendicular to the edge, the line is connected to the iris boundary (e.g., the iris boundary indicated by "R1" in FIG. 4) of the iris image, and the point of connection is denoted as a vertex
，
、
Respectively representing the first on a spatial quadrilateral
Bisector point of the sides, and from the bisector point
Vertex connecting to iris boundary;
d5, from the vertex
To the spatial quadrangle
The two endpoints of the edge are connected to obtain a triangle (such as the triangle denoted by the reference number "U1" in FIG. 4), which is marked as
Triangular shape
Will be provided with
The outer iris area is divided into two arc-shaped iris areas which are respectively marked as
(for example, as indicated by the reference numeral area1 in FIG. 4),
(indicated by reference numeral area2 in fig. 4);
d6, calculating the triangle
Area of (d) is marked as
；
D7, in a triangle
The two waists are
Dividing each waist equally, connecting lines from the equal division points to the iris boundary of the arc iris region in a mode of being vertical to the waist to obtain connection vertexes, and connecting the connection vertexes to the triangle
The two end points of the waist are connected to obtain a triangle (such as the triangle marked by the reference number "U11" in FIG. 4), which is marked as
Since there are two discrete arc iris regions per triangle, there is no need to provide a separate iris filter
；
D8, calculating the triangle
Area of (d) is marked as
；
D9, in a triangle
The two waists are
And (4) connecting the lines further by the method of the steps D7-D8 to obtain a triangle, calculating the area of the triangle until reaching the preset triangle construction times, and then calculating the area of the iris image by the following formula (6)
：
In the formula (6), the first and second groups of the compound,
an arbitrary equally dividing block (such as the region indicated by the reference symbol "V1" and selected by the bold solid line frame in FIG. 4) for equally dividing the iris image with the center point of the iris image as the origin of the XY axis coordinate system is shown as the second
Next to
Dividing the edges equally;
d10, calculating the iris image by the following formula (7)
：
The data exchange and processing method based on the distributed architecture provided by the embodiment of the invention is shown in figure 1 and comprises the following steps:
s1, a node is designated as a main node for data processing from a plurality of node groups of a distributed architecture, and the designation method specifically comprises the following steps:
s11, calculating the activity degree of each node group, and forming a node group sequence after sequencing from low to high according to the activity degree, wherein each element in the node group sequence carries a label of a corresponding node group (for example, the label is a distance value and a value interval);
s12, calculating the identity detail characteristic value of the person to be checked (such as calculating the distance value and value of the fingerprint image of the person to be checked)
）；
S13, extracting elements which have label corresponding relation with the calculated identity detail characteristic value from the node group sequence in an extraction mode of reserving a sorting sequence to form an extraction sequence;
for example, the node group sequence is (a first node group 1, a first node group 2, a second node group 3, a third node group 1, a fourth node group 1, a third node group 2, a first node group 3, a fourth node group 3, a third node group 4, a second node group 1, a third node group 3, and a fourth node group 2), and if the identity detail characteristic values calculated in step S12 are the distance value and the value
If the extracted extraction sequence is (the first node group 1, the first node group 2, the first node group 3);
and S14, taking the node with the lowest activity degree in the node group corresponding to the first element in the extraction sequence as a main node. For example, a node having the lowest activity level in the first node group 1 (first node group 1, first node group 2, and first node group 3) at the top of the extraction sequence is used as the master node.
The method for calculating the activity of the node group comprises the following steps:
s111, calculating a weighted summation value of the activity of each node in the node group;
and S112, calculating the product of the weighted summation value and the activity correction coefficient corresponding to the node group as the activity calculated for the node group.
Each node in the distributed architecture has data processing capabilities and data storage capabilities. In this embodiment, the node that is required for the master node to request data exchange sends the data stored in the node to the master node in the form of a data packet according to the data exchange request of the master node, and the master node processes the requested data and outputs a data processing result. Ideally, only one node is included in each node group in the distributed architecture. The distributed architecture is assumed to have first to fourth node groups, only one node in the first node group is used for storing a first label, namely a fingerprint image data set corresponding to a distance value and a value interval, only one node in the second node group is used for storing a second label, namely a fingerprint image data set corresponding to a quantity interval, only one node in the third node group is used for storing a third label, namely a face image data set corresponding to a ratio value and a value interval, and only one node in the fourth node group is used for storing a fourth label, namely an iris image data set corresponding to a value ratio value interval. When the master node performs data processing, assuming that image data associated with each node in the 4 node groups having tag association needs to be acquired simultaneously, the master node may directly acquire all image data from the 4 nodes. However, only 4 node groups with tag association relationships are set in the distributed architecture, and only one node is set in each node group, which simplifies the data processing process of the master node, but seriously affects the distributed computing capability of the system, and since the master node must be specified from the only 4 nodes and each node is a necessary node for the master node to process data, when an external third-party service calls the 4 nodes at high frequency simultaneously, the data processing performance of the master node is greatly affected. Therefore, it is necessary to improve the distributed data processing capability of the data exchange system provided in this embodiment.
The most direct and effective means for improving the distributed processing capability is to increase the number of distributed nodes, but in this embodiment, the type and number of the tags of the node group are determined, for example, if there are 3 different distance values and value intervals, there are 3 different first tags respectively corresponding to the 3 different distance values and value intervals, and therefore, the number of the node group cannot be extended by increasing the tags. Therefore, in order to solve the problem, the present embodiment adopts a scheme that the number of nodes is increased in each node group with a fixed number, for example, one original node group includes 1 node, and now the node group is expanded into 1 node group including 3 nodes, and tags corresponding to the 3 nodes after the expansion are the same as tags corresponding to nodes before the expansion, that is, for example, original distance values and original distance values are used
Corresponding fingerprint images are uniformly added to
The fingerprint image data sets associated with only the corresponding node 1 of the falling distance value and value interval will now be described
Corresponding fingerprint images being randomly added to
And the first fingerprint image data set, the second fingerprint image data set and the third fingerprint image data set are respectively associated with the nodes 1, 2 and 3 corresponding to the falling distance value and value interval. When the master node needs to process data, the first fingerprint image data set, the second fingerprint image data set and the third fingerprint image data set stored in 3 nodes in the node group are obtained. The method for increasing the number of nodes in each node group well improves the distributed processing capacity of the system, but creates new technical problems: because the main node needs to communicate with other nodes when processing data, the number of other nodes rises, the complexity of processing data by the main node is increased, if safety means such as a checking mechanism and the like are added in the communication,the complexity of the data processing will be higher. Therefore, how to simplify the data processing process of the master node as much as possible while considering the distributed processing capability becomes an urgent technical problem to be solved. In order to solve the technical problem, as shown in fig. 1, the data exchange and processing method based on the distributed architecture provided by this embodiment proceeds to the following steps:
s2, as shown in fig. 5, the service processing module forwards the data exchange request in the same poll received by the master node to a designated node in a designated node group in the distributed architecture, where the method for determining the designated node includes the following steps:
s21, matching label data pairs for representing a group of people from a database by taking labels corresponding to identity detail characteristic values calculated by a person to be checked as matching bases of a designated node group, and taking the node group corresponding to each type of label in the label data pairs as the matched designated node group;
for example, assume that the calculated identity detail feature value is
，
The corresponding of the falling distance value and the value interval is the first label, and then the first label is used
As a matching basis for specifying a node group, a first label is used for assuming the characteristics of a certain group of people
-a second label
-a third label
-a fourth label
The tag data pair is characterized according to the first tag
Matching the recorded tag data pair from the database and applying a second tag
The third label
And a fourth label
Respectively corresponding node groups are used as matched appointed node groups;
s22, extracting nodes with the highest activity degree left after the node filtering in each appointed node group as appointed nodes, wherein the method for filtering the appointed node group comprises the following steps:
s231, the master node performs individual identity characteristic comparison on the data packet sent by the node with the highest liveness in each designated node group, and judges whether the comparison is successful or not,
if yes, generating a termination signal for terminating the data exchange of the designated node group in the polling, and sending the termination signal to a service processing module, wherein the service processing module does not send a data exchange request to the designated node group in the polling after receiving the termination signal;
if not, generating a node filtering signal and sending the node filtering signal to the service processing module, and turning to the step S232;
s232, the service processing module filters out the node with the identity feature comparison in the current data exchange request in step S231 from the designated node group according to the node filtering signal, and does not include the node as the designated node in the subsequent data exchange in the polling.
After a node to be sent of a data exchange request is specified, as shown in fig. 1, the data exchange and processing method based on a distributed architecture provided in this embodiment proceeds to the following steps:
s3, each designated node receiving the data exchange request packs the data stored by the designated node and then sends the data to the service processing module;
in order to improve the communication security between the designated node and the service processing module, and between the service processing module and the main node, preferably, after each designated node receiving the data exchange request verifies the data exchange permission of the main node and passes the data exchange permission, the data stored by the designated node is packaged and sent to the service processing module, and the permission verification method specifically comprises the following steps:
s31, analyzing a main node access path, a label corresponding to the main node and a unique node number of the main node carried in the data exchange request forwarded by the service processing module;
s32, searching a node group to which the main node belongs according to the label corresponding to the main node;
s33, acquiring the access path of each other node in the node group from the database according to the binding relationship between the analyzed access path of the master node and the access positions of other nodes in the node group found in the step S32;
s34, accessing all nodes in the node group and reading the liveness recorded at each node;
s35, judging whether the read node with the lowest liveness is the main node carried in the data exchange request,
if so (the node numbers are successfully matched), judging that the authority verification is passed;
if not, the authority verification is judged to fail.
After the authority of the designated node to the master node is verified, the data exchange and processing method based on the distributed architecture provided by the embodiment proceeds to the following steps:
s4, the service processing module forms a data packet sent by the designated node in each designated node group into a data set requested by the current data exchange request and forwards the data set to the main node;
s5, the main node processes the data of the requested data set and judges whether the data processing result is successfully obtained (for example, whether the identity is successfully verified according to the data set),
if yes, feeding back a data processing result to each designated node of the corresponding data packet generated in the data set, finishing polling and sending a data exchange request, and accumulating the activity of each designated node by 1 after receiving the result feedback;
if not, initiating the next data exchange request in the polling, then returning to the step S2, and reassigning the node until the polling request is finished.
In conclusion, the invention has the following beneficial effects:
1. the liveness is introduced as a node designation basis of a data exchange request sending object, and in each data exchange request of the same polling, by re-determining a designated node, the pertinence of data exchange is improved, the distributed performance is improved by increasing the number of nodes, the data processing speed of the main node is considered, and the data processing process is simplified.
2. The liveness well represents rules and characteristics of data processing of the main node under different scenes and different periods, the problem of probability prediction of each node in the distributed architecture, which is needed, is converted into a relatively quantitative liveness calculation problem, and in each data exchange request, the main node can determine an object to be subjected to data exchange more quickly, simply and relatively accurately.
3. A communication verification mechanism is introduced, and each designated node forwards the data packet to the main node through the service processing module after the data exchange authority verification of the main node is passed, so that the safety of data exchange and processing is improved.
It should be understood that the above-described embodiments are merely preferred embodiments of the invention and the technical principles applied thereto. It will be understood by those skilled in the art that various modifications, equivalents, changes, and the like can be made to the present invention. However, such variations are within the scope of the invention as long as they do not depart from the spirit of the invention. In addition, certain terms used in the specification and claims of the present application are not limiting, but are used merely for convenience of description.
Claims (6)
1. A data exchange and processing method based on a distributed architecture is characterized by comprising the following steps:
s1, a node is designated from a node group in a distributed architecture as a main node for data processing;
s2, the service processing module forwards the data exchange request in the same polling received at the main node to a designated node in a designated node group in the distributed architecture;
s3, each appointed node receiving the data exchange request packs the data stored by the appointed node and then sends the packed data to the service processing module;
s4, the service processing module forms a data packet sent by the designated node in each designated node group into a data set requested by the current data exchange request and forwards the data set to the main node;
s5, the master node processes the data of the requested data set and judges whether the data processing result is successfully obtained or not,
if yes, feeding back a data processing result to each designated node corresponding to the data packet generated in the data set, finishing polling and sending the data exchange request, and accumulating the activity of each designated node by '1' after receiving result feedback;
if not, the next data exchange request is initiated under the polling, and then the step S2 is returned until the polling request is finished.
2. The method for data switching and processing based on distributed architecture of claim 1, wherein the distributed architecture comprises a plurality of first node groups respectively corresponding to different first tags, a plurality of second node groups respectively corresponding to different second tags, a plurality of third node groups respectively corresponding to different third tags, and a plurality of fourth node groups respectively corresponding to different fourth tags, each of the node groups comprises at least one of the nodes corresponding to the same tag;
in step S1, the method for designating the master node includes the steps of:
s11, calculating the activity degree of each node group, and forming a node group sequence after sequencing from low to high according to the activity degree, wherein each element in the node group sequence carries a label of the corresponding node group;
s12, calculating identity detail characteristic values of the personnel to be checked;
s13, extracting elements which have label corresponding relation with the calculated identity detail characteristic values from the node group sequence in an extraction mode of reserving a sorting sequence to form an extraction sequence;
s14, taking the node with the lowest activity degree in the node group corresponding to the first element in the extraction sequence as the main node.
3. The distributed architecture-based data exchange and processing method according to claim 2, wherein the method for calculating the activity level of each node group comprises the steps of:
s111, calculating a weighted summation value of the activity of each node in the node group;
and S112, calculating the product of the weighted summation value and the activity correction coefficient corresponding to the node group as the activity calculated for the node group.
4. The method according to claim 2, wherein in step S2, the method for determining the designated node in each designated node group as a forwarding object of the traffic processing to the data exchange request initiated by the master node comprises the steps of:
s21, matching label data pairs for representing a group of people from a database by taking labels corresponding to the identity detail characteristic values of the people to be checked as matching bases of a specified node group, and taking the node group corresponding to each type of label in the label data pairs as the matched specified node group;
and S22, extracting the nodes with the highest activity left in each appointed node group after node filtering as appointed nodes.
5. The data exchange and processing method based on distributed architecture of claim 4, wherein in step S23, the method for node filtering on the designated node group includes the steps of:
s231, the master node performs individual identity characteristic comparison on the data packet sent by the node with the highest liveness in each designated node group, and judges whether the comparison is successful or not,
if yes, generating a termination signal for terminating the data exchange of the specified node group in the polling, and sending the termination signal to the service processing module;
if not, generating a node filtering signal, sending the node filtering signal to the service processing module, and turning to the step S232;
s232, the service processing module filters out the nodes with identity feature comparison in the current data exchange request of step S231 from the designated node group according to the node filtering signal.
6. The data exchange and processing method based on the distributed architecture of claim 1, wherein in step S3, after each of the designated nodes that receive the data exchange request verifies the data exchange permission of the master node and passes the data exchange permission, the designated nodes package data stored by themselves and send the packaged data to the service processing module, and the permission verification method includes the steps of:
s31, analyzing a main node access path, a label corresponding to the main node and a unique node number of the main node carried in the data exchange request forwarded by the service processing module;
s32, searching the node group to which the main node belongs according to the label corresponding to the main node;
s33, acquiring the access path of each other node in the node group from a database according to the analyzed binding relationship between the access path of the main node and the access positions of other nodes in the node group found in the step S32;
s34, accessing all nodes in the node group and reading the liveness recorded at each node;
s35, judging whether the node with the lowest read liveness is the main node carried in the data exchange request or not,
if yes, judging that the authority verification is passed;
if not, the authority verification is judged to fail.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310090854.2A CN115811525B (en) | 2023-02-09 | 2023-02-09 | Data exchange and processing method based on distributed architecture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310090854.2A CN115811525B (en) | 2023-02-09 | 2023-02-09 | Data exchange and processing method based on distributed architecture |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115811525A true CN115811525A (en) | 2023-03-17 |
| CN115811525B CN115811525B (en) | 2023-06-16 |
Family
ID=85487872
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310090854.2A Active CN115811525B (en) | 2023-02-09 | 2023-02-09 | Data exchange and processing method based on distributed architecture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115811525B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140379896A1 (en) * | 2013-06-24 | 2014-12-25 | Cisco Technology, Inc. | Distributed liveness reporting in a computer network |
| US20170262519A1 (en) * | 2010-12-23 | 2017-09-14 | Eliot Horowitz | System and method for determining consensus within a distributed database |
| US20170286517A1 (en) * | 2010-12-23 | 2017-10-05 | Eliot Horowitz | Systems and methods for managing distributed database deployments |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
| CN112311596A (en) * | 2020-10-22 | 2021-02-02 | 深圳前海微众银行股份有限公司 | Data management method, device, equipment and computer storage medium |
| CN112532581A (en) * | 2020-10-26 | 2021-03-19 | 南京辰阔网络科技有限公司 | Improved PBFT consensus method based on consensus participation and transaction activity |
| CN112671932A (en) * | 2021-01-25 | 2021-04-16 | 梁志彬 | Data processing method based on big data and cloud computing node |
| CN113127267A (en) * | 2021-04-30 | 2021-07-16 | 郑州师范学院 | Strong-consistency multi-copy data access response method in distributed storage environment |
| CN113542398A (en) * | 2021-07-13 | 2021-10-22 | 广州云从凯风科技有限公司 | Control method, device, medium and equipment of distributed cluster system |
-
2023
- 2023-02-09 CN CN202310090854.2A patent/CN115811525B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170262519A1 (en) * | 2010-12-23 | 2017-09-14 | Eliot Horowitz | System and method for determining consensus within a distributed database |
| US20170286517A1 (en) * | 2010-12-23 | 2017-10-05 | Eliot Horowitz | Systems and methods for managing distributed database deployments |
| US20140379896A1 (en) * | 2013-06-24 | 2014-12-25 | Cisco Technology, Inc. | Distributed liveness reporting in a computer network |
| CN108829509A (en) * | 2018-05-03 | 2018-11-16 | 山东汇贸电子口岸有限公司 | Distributed container cluster framework resources management method based on domestic CPU and operating system |
| CN112311596A (en) * | 2020-10-22 | 2021-02-02 | 深圳前海微众银行股份有限公司 | Data management method, device, equipment and computer storage medium |
| WO2022083690A1 (en) * | 2020-10-22 | 2022-04-28 | 深圳前海微众银行股份有限公司 | Data management method and apparatus, and device, computer storage medium and program |
| CN112532581A (en) * | 2020-10-26 | 2021-03-19 | 南京辰阔网络科技有限公司 | Improved PBFT consensus method based on consensus participation and transaction activity |
| CN112671932A (en) * | 2021-01-25 | 2021-04-16 | 梁志彬 | Data processing method based on big data and cloud computing node |
| CN113127267A (en) * | 2021-04-30 | 2021-07-16 | 郑州师范学院 | Strong-consistency multi-copy data access response method in distributed storage environment |
| CN113542398A (en) * | 2021-07-13 | 2021-10-22 | 广州云从凯风科技有限公司 | Control method, device, medium and equipment of distributed cluster system |
Non-Patent Citations (1)
| Title |
|---|
| 侯桂云;陈桂英;卢志强;: "关于云计算中分布式数据存储仿真研究" * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115811525B (en) | 2023-06-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10650041B2 (en) | Image recognition verification | |
| CN109858371B (en) | Face recognition method and device | |
| CN110853033B (en) | Video detection method and device based on inter-frame similarity | |
| US8244002B2 (en) | System and method for performing rapid facial recognition | |
| JP4907621B2 (en) | Video processing apparatus and method and recording medium therefor | |
| JP7145224B2 (en) | Target object recognition method, apparatus and system | |
| CN112818149A (en) | Face clustering method and device based on space-time trajectory data and storage medium | |
| EP2605169A1 (en) | User detecting apparatus, user detecting method, and a user detecting program | |
| US8964126B2 (en) | Video processing system, video processing method, database for video processing and generating method thereof, video processing apparatus, control method thereof and control program thereof | |
| CN109447186A (en) | Clustering method and Related product | |
| CN105814582B (en) | Method and system for recognizing human face | |
| CN110378351A (en) | Seal discrimination method and device | |
| CN108647640A (en) | The method and electronic equipment of recognition of face | |
| CN109360205A (en) | Double record video quality detecting methods, device, medium and electronic equipment | |
| CN115661624A (en) | Digital method and device for goods shelf and electronic equipment | |
| CN109359689B (en) | Data identification method and device | |
| CN115811525A (en) | Data exchange and processing method based on distributed architecture | |
| WO2021217914A1 (en) | Damage evidence extraction method and apparatus, computer device, and storage medium | |
| CN112257666B (en) | Target image content aggregation method, device, equipment and readable storage medium | |
| CN113486852B (en) | Face and human body association method and device | |
| WO2013084731A1 (en) | Image identifying system | |
| CN115147633A (en) | Image clustering method, device, equipment and storage medium | |
| CN109145855A (en) | A kind of method for detecting human face and device | |
| CN113822280B (en) | Text recognition method, device, system and nonvolatile storage medium | |
| CN115131582A (en) | Target identification method, device and medium based on morphological identification template matching |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |