CN114238295A - Data sorting method and device based on grouping - Google Patents
Data sorting method and device based on grouping Download PDFInfo
- Publication number
- CN114238295A CN114238295A CN202111458699.2A CN202111458699A CN114238295A CN 114238295 A CN114238295 A CN 114238295A CN 202111458699 A CN202111458699 A CN 202111458699A CN 114238295 A CN114238295 A CN 114238295A
- Authority
- CN
- China
- Prior art keywords
- data
- sorting
- grouping
- packet
- sequencing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention discloses a data sorting method and a device based on grouping, wherein the method comprises the following steps: setting a post condition and a grouping strategy based on the current service scene and the characteristics of a target storage system; creating a plurality of groups according to the grouping strategy, and performing data grouping insertion according to insertion logics corresponding to the groups to obtain data of different groups; after the data packet insertion is finished, performing packet corresponding cleaning processing on the data of different packets according to a post condition; returning to the structural body according to the current service scene definition; integrating the cleaned data of the different groups into a preset data format, and sequencing the data according to a priority rule configured in the data format to obtain a sequencing result; and matching the sequencing result with the returned structure body, and verifying the validity of the sequencing result. The embodiment of the data sorting method and the device based on the grouping provided by the invention avoids the sorting process of repeated writing and can avoid generating excessive repeated useless data.
Description
Technical Field
The invention relates to the field of data automation processing, in particular to a data sorting method and device based on grouping.
Background
The current online mall system generally displays the commodity label information according to priority logic aiming at the commodity list and the detail part, thereby triggering the purchasing behavior of the user.
The current sorting method is generally: the method includes the following steps that script execution is compiled according to rules defined by requirements, and the following sequence is added to copy the existing execution method again (specifically, the existing version is defined according to the requirements for a sequencing or display verification scheme, then corresponding test data is prepared to execute sequencing, sequencing results are compared, when requirement priority adjustment is carried out, new priority display is added, or the sequencing rules are changed, related prepared data need to be copied again, then a return result of verification is modified, namely, data needs to be copied and structure verification needs to be executed once every time the rules are added or modified), and the sequencing method has the following defects: (1) no grouping is performed; (2) there is a lot of duplicated data that is useless. The data preparation process needs to use repeated preparation data, the only difference between the preparation data is that the sequencing tag ids needed by the preparation data are replaced, many similar data are the repeated data in different scenes of case execution, and in addition, the repeated change and check return result also has redundancy and repetition in the setting of the sequencing execution process.
Disclosure of Invention
The embodiment of the invention provides a data sorting method and device based on grouping, which are used for grouping all processes involved in a sorting execution flow (data initialization, rule setting, expected check rule, data cleaning and the like), removing a repeatedly written sorting flow and avoiding excessive repeated useless data generated in the sorting process.
To achieve the above object, a first aspect of the embodiments of the present application provides a packet-based data ordering method, including:
setting a post condition and a grouping strategy based on the current service scene and the characteristics of a target storage system;
creating a plurality of groups according to the grouping strategy, and performing data grouping insertion according to insertion logics corresponding to the groups to obtain data of different groups;
after the data packet insertion is finished, performing packet corresponding cleaning processing on the data of different packets according to a post condition;
returning to the structural body according to the current service scene definition;
integrating the cleaned data of the different groups into a preset data format, and sequencing the data according to a priority rule configured in the data format to obtain a sequencing result;
and matching the sequencing result with the returned structure body, and verifying the validity of the sequencing result.
In a possible implementation manner of the first aspect, the preset data format includes a priority rule, a value of a cause, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of each of the priority rules.
In a possible implementation manner of the first aspect, the performing data sorting according to a priority rule configured in the data format to obtain a sorting result specifically includes:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
In a possible implementation manner of the first aspect, the matching the ranking result and the returned structure, and verifying the validity of the ranking result specifically include:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
A second aspect of an embodiment of the present application provides a packet-based data sorting apparatus, including:
the initial setting module is used for setting a post condition and a grouping strategy based on the current service scene and the characteristics of a target storage system;
the grouping module is used for creating a plurality of groups according to the grouping strategy and inserting the data groups according to the inserting logic corresponding to the groups to obtain the data of different groups;
the cleaning module is used for executing cleaning processing corresponding to the grouping on the data of the different groups according to the post condition after the data grouping insertion is finished;
the definition module is used for defining a return structure body according to the current service scene;
the sorting module is used for integrating the cleaned data of the different groups into a preset data format and sorting the data according to a priority rule configured in the data format to obtain a sorting result;
and the matching module is used for matching the sequencing result with the returned structure body and verifying the validity of the sequencing result.
In a possible implementation manner of the second aspect, the preset data format includes a priority rule, a value of a cause, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of each of the priority rules.
In a possible implementation manner of the second aspect, the sorting the data according to the priority rule configured in the data format to obtain a sorting result specifically includes:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
In a possible implementation manner of the second aspect, the matching module is specifically configured to:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
Compared with the prior art, the packet-based data sorting method and device provided by the embodiment of the invention integrate the data of different packets into a preset data format, thereby avoiding repeatedly writing the data. And grouping processes (data initialization, rule setting, expected sorting rule, data cleaning and the like) involved in the sorting execution flow to obtain a sorting result. And finally, judging the correctness of the sequencing result by combining the returned structure.
In summary, by using the packet-based data sorting method and device provided by the embodiment of the invention, only data and predefined rules need to be formulated, other additional configuration is not needed, and a target system does not need to be modified, so that the generation of repeated data is effectively avoided, and the sorting efficiency is improved. In the sequencing and detection processes, manual intervention is not needed in the whole process, and the resource investment of testing personnel is reduced.
Drawings
Fig. 1 is a flowchart illustrating a packet-based data sorting method according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, an embodiment of the present invention provides a packet-based data sorting method, where the method includes:
and S10, setting a post condition and a grouping strategy based on the current service scene and the characteristics of the target storage system.
S11, creating a plurality of groups according to the grouping strategy, and inserting the data groups according to the inserting logic corresponding to the groups to obtain the data of different groups.
And S12, after the data packet insertion is finished, performing packet corresponding cleaning processing on the data of different packets according to a post condition.
And S13, returning the structural body according to the current business scene definition.
And S14, integrating the cleaned data of the different groups into a preset data format, and sequencing the data according to a priority rule configured in the data format to obtain a sequencing result.
And S15, matching the sorting result with the returned structure body, and verifying the validity of the sorting result.
S10 is a process of establishing a grouping policy in this embodiment of the method, the returned structure specified in S13 is mainly used to verify whether the sorting result is reliable, and is an ideal expected structure, S14 is a process of processing data according to the priority rule, S15 is a final sorting result determination process, and different types of analysis and determination are performed according to the defined priority.
In S10, based on the commonly used scenario of the current service, a post-condition needs to be established (when the script is executed, a behavior operation needs to be triggered, such as clearing data, and restoring the array in the execution process to the initialized state, so as to achieve data recycling). S11-S12 adopt similar data execution operation rules for grouping, and writing implementation methods in different grouping methods. For a typical data system, it is mainly written into different storage systems. If java is adopted to implement S11-S12, several method names X for batch writing need to be defined, and if groups are added in the execution scene, the method name X is called to execute batch insertion data.
Illustratively, the preset data format includes a priority rule, a value of a manufactured value, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of each of the priority rules.
Exemplarily, the sorting the data according to the priority rule configured in the data format to obtain the sorting result specifically includes:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
In practical applications, the packet data obtained from S11-S12 will generally be passed in a parameter format that sets "priority order/value/expected value", and different methods will be called to process when the packet data is received. Firstly, filtering and acquiring a required value by using Stream flow operation in java, calling basic service to issue configuration information to a current execution scene in priority sequencing, making a hit value according to a grouping strategy, and returning specific sequencing data according to service corresponding to a service scene.
It should be noted that, here, priority refers to a rule that data to be processed is sorted before or after data in the process of multiple pieces of data appearing at the same time according to the required function point of each service scenario, the priority rule is a format written according to the service function point, the format is similar to a | b, c, d > e.f, specifically expressed as initialization and cleaning data a, b, c, d, e, and then the display or sorting result of sorting priority is a, d, f.
Exemplarily, S15 specifically includes:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
The following description is given by using a certain commodity label system, where there are multiple labels A, B, C, D, E as functional points to be realized, the current configuration rule is a | B, C > D, E, and the function is finally displayed until the commodity sorting priority requires a, C, E to indicate normal or otherwise abnormal.
The first step is as follows: grouping is performed based on a tag current storage system, after a service scene is analyzed, it is confirmed that the current system needs to be stored in a cache and a database respectively, then a grouping strategy defines an operation cache (the cache comprises a local cache and a distributed cache) and the database, and generates two marks, namely a packet cache and a packet database, when the two marks exist, the execution process initialization and the execution are completed, then the cleaning data executes corresponding logic processing according to a method in the defined packet, and the data initialization and the data reuse are ensured.
The second step is that: and (3) based on a verification format formulated by a sequencing result needing verification, confirming that all the content of the label, including the label ID, the label name commodity, the display position, the corresponding picture information and the like, needs to be returned by the current service function point, adding corresponding structure content information aiming at the current structure, and ensuring that the structure content includes the function required information.
The third step: the current business function point configuration rule is A | B, C > D, E, 3 step examples can be split based on the rule, firstly, the information of the configuration rule is A | B, C > D, E is used for calling a corresponding configuration rule updating method to ensure that the current rule only takes effect for the current execution case, secondly, label information is generated according to the manufacturing data, the label information A, B, C, D and E are obtained here, the data manufacturing method is called by the information of the label A, B, C, D and E to generate the label data meeting the requirements, finally, priority display is obtained, and the final priority display sequence is A > C > E by calling priority obtaining logic processing through the configuration rule information.
The fourth step: based on the fact that the sequence of the label rules A, B, C, D and E is finally displayed clearly in the previous steps, the execution result information of this time can be obtained by calling a uniformly written interface calling mode for execution, then the execution result and the expected result in the step 2 are returned to call the consistency judging method for comparison, and if the data matching is successful, the current execution is failed.
Compared with the prior art, the data sorting method based on the grouping provided by the embodiment of the invention integrates the data of different groups into the preset data format, thereby avoiding repeatedly writing the data. And grouping processes (data initialization, rule setting, expected sorting rule, data cleaning and the like) involved in the sorting execution flow to obtain a sorting result. And finally, judging the correctness of the sequencing result by combining the returned structure.
In summary, by adopting the packet-based data sorting method provided by the embodiment of the invention, only data and predefined rules need to be formulated, other additional configuration is not needed, and a target system does not need to be modified, so that the generation of repeated data is effectively avoided, and the sorting efficiency is improved. In the sequencing and detection processes, manual intervention is not needed in the whole process, and the resource investment of testing personnel is reduced.
A second aspect of an embodiment of the present application provides a packet-based data sorting apparatus, including: the device comprises an initial setting module, a grouping module, a cleaning module, a defining module, a sequencing module and a matching module.
And the initial setting module is used for setting a post condition and a grouping strategy based on the current service scene and the characteristics of the target storage system.
And the grouping module is used for creating a plurality of groups according to the grouping strategy and inserting the data groups according to the inserting logic corresponding to the groups to obtain the data of different groups.
And the cleaning module is used for executing cleaning processing corresponding to the grouping on the data of the different groups according to a post condition after the data grouping insertion is finished.
And the definition module is used for defining a return structure body according to the current service scene.
And the sorting module is used for integrating the cleaned data of the different groups into a preset data format and sorting the data according to a priority rule configured in the data format to obtain a sorting result.
And the matching module is used for matching the sequencing result with the returned structure body and verifying the validity of the sequencing result.
Illustratively, the preset data format includes a priority rule, a value of a manufactured value, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of each of the priority rules.
Exemplarily, the sorting the data according to the priority rule configured in the data format to obtain the sorting result specifically includes:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
Exemplarily, the matching module is specifically configured to:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
It should be noted that the above modules are executed based on an internally stored software application during running, and the applications stored by the above modules may be applications written based on pure java. If the java writing is adopted, other redundant technical components (only depending on or a message sdk) are not needed, the system to be tested does not need to be modified, and the use is simple and convenient.
If the data is in a java-based testng data driving mode, filling priority information in the dataProvider according to the rule requirement. The priority here refers to the ordering of the processing rules according to the occurrence of multiple pieces of data at the same time by each service demand function point.
Compared with the prior art, the data sorting device based on the grouping provided by the embodiment of the invention integrates the data of different groups into the preset data format, thereby avoiding repeatedly writing the data. And grouping processes (data initialization, rule setting, expected sorting rule, data cleaning and the like) involved in the sorting execution flow to obtain a sorting result. And finally, judging the correctness of the sequencing result by combining the returned structure.
In summary, the packet-based data sorting device provided by the embodiment of the invention only needs to formulate data and predefined rules, does not need other additional configuration, and does not need to modify a target system, thereby effectively avoiding the generation of repeated data, and improving the sorting efficiency. In the sequencing and detection processes, manual intervention is not needed in the whole process, and the resource investment of testing personnel is reduced.
In summary, with the method and the device for automatically testing messages provided by the embodiment of the present invention, only the message sending personnel needs to prepare the required data, the message sending content and the service scene flow, and verify the logic, and does not need to pay attention to other basic service processing, so that the resource investment cost of the testing personnel is effectively reduced, and the message sending efficiency and the message checking efficiency are improved.
While the foregoing is directed to the preferred embodiment of the present invention, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention.
Claims (8)
1. A method for packet-based data sequencing, comprising:
setting a post condition and a grouping strategy based on the current service scene and the characteristics of a target storage system;
creating a plurality of groups according to the grouping strategy, and performing data grouping insertion according to insertion logics corresponding to the groups to obtain data of different groups;
after the data packet insertion is finished, performing packet corresponding cleaning processing on the data of different packets according to a post condition;
returning to the structural body according to the current service scene definition;
integrating the cleaned data of the different groups into a preset data format, and sequencing the data according to a priority rule configured in the data format to obtain a sequencing result;
and matching the sequencing result with the returned structure body, and verifying the validity of the sequencing result.
2. The packet-based data ordering method according to claim 1, wherein the predetermined data format includes a priority rule, a value of a cause, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of data to each of the priority rules.
3. The packet-based data sorting method according to claim 2, wherein the sorting of the data according to the priority rule configured in the data format to obtain a sorting result specifically comprises:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
4. The packet-based data sorting method according to claim 1, wherein the matching the sorting result and the returned structure to verify the validity of the sorting result specifically comprises:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
5. An apparatus for packet-based data sequencing, comprising:
the initial setting module is used for setting a post condition and a grouping strategy based on the current service scene and the characteristics of a target storage system;
the grouping module is used for creating a plurality of groups according to the grouping strategy and inserting the data groups according to the inserting logic corresponding to the groups to obtain the data of different groups;
the cleaning module is used for executing cleaning processing corresponding to the grouping on the data of the different groups according to the post condition after the data grouping insertion is finished;
the definition module is used for defining a return structure body according to the current service scene;
the sorting module is used for integrating the cleaned data of the different groups into a preset data format and sorting the data according to a priority rule configured in the data format to obtain a sorting result;
and the matching module is used for matching the sequencing result with the returned structure body and verifying the validity of the sequencing result.
6. The packet-based data sequencing apparatus of claim 5, wherein said predetermined data format comprises a priority rule, a value of a cause, and a value of a hit;
the priority rule is formulated based on the current business scene; the manufactured value is a reference value for performing initialization assignment on the data of the different groups; the hit value reflects the degree of matching of each of the priority rules.
7. The packet-based data sorting device of claim 6, wherein the sorting of the data according to the priority rule configured in the data format to obtain the sorting result specifically comprises:
filtering and acquiring needed packet data by using Stream streaming operation;
and generating data to be sorted according to the numerical values, and calling a priority rule matched with the hit value to sort the data to be sorted to obtain a sorting result.
8. The packet-based data sequencing of claim 5, wherein the matching module is specifically configured to:
if the sorting result is consistent with the data format of the returned structure body, the sorting result is valid;
and if the sorting result is inconsistent with the data format of the returned structure body, the sorting result is invalid.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111458699.2A CN114238295A (en) | 2021-12-01 | 2021-12-01 | Data sorting method and device based on grouping |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111458699.2A CN114238295A (en) | 2021-12-01 | 2021-12-01 | Data sorting method and device based on grouping |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114238295A true CN114238295A (en) | 2022-03-25 |
Family
ID=80752754
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111458699.2A Pending CN114238295A (en) | 2021-12-01 | 2021-12-01 | Data sorting method and device based on grouping |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114238295A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292202A (en) * | 2022-09-29 | 2022-11-04 | 立臻科技(昆山)有限公司 | Product test analysis method and device, electronic equipment and storage medium |
| CN116226296A (en) * | 2023-01-19 | 2023-06-06 | 广州海量数据库技术有限公司 | OpenGauss-based data packet aggregation method |
-
2021
- 2021-12-01 CN CN202111458699.2A patent/CN114238295A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292202A (en) * | 2022-09-29 | 2022-11-04 | 立臻科技(昆山)有限公司 | Product test analysis method and device, electronic equipment and storage medium |
| CN116226296A (en) * | 2023-01-19 | 2023-06-06 | 广州海量数据库技术有限公司 | OpenGauss-based data packet aggregation method |
| CN116226296B (en) * | 2023-01-19 | 2023-08-22 | 广州海量数据库技术有限公司 | OpenGauss-based data packet aggregation method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109189669A (en) | Test method, electronic device and the readable storage medium storing program for executing of business scenario | |
| CN114238295A (en) | Data sorting method and device based on grouping | |
| CN108874678B (en) | Automatic testing method and device for intelligent program | |
| CN107153646B (en) | Data processing method and equipment | |
| CN112446786A (en) | Abnormal transaction processing method and device, electronic equipment and readable storage medium | |
| CN111367782B (en) | Regression testing data automatic generation method and device | |
| CN111444101A (en) | Method and device for automatically creating product test defects | |
| CN108650123B (en) | Fault information recording method, device, equipment and storage medium | |
| CN112882957B (en) | Test task validity checking method and device | |
| CN112052157B (en) | Method, device and system for constructing test message | |
| WO2023124314A1 (en) | Micro-service test method and system for hybrid cpu architecture device, and related apparatus | |
| CN116431522A (en) | Automatic test method and system for low-code object storage gateway | |
| CN115080401A (en) | Automatic testing method and related device | |
| CN110618943B (en) | Security service test method and device, electronic equipment and readable storage medium | |
| CN112035287B (en) | Method, device, storage medium and equipment for testing data cleaning result | |
| CN110362464B (en) | Software analysis method and equipment | |
| CN113297149A (en) | Method and device for monitoring data processing request | |
| CN110781142A (en) | Data import method and device, server and storage medium | |
| CN110990278A (en) | Test method and device | |
| CN111241082A (en) | Data correction method and device | |
| CN117724986B (en) | Method and device for automatically operating kafka in use case execution | |
| CN117235058A (en) | Service data processing method, device, equipment and medium | |
| CN115511563A (en) | Service order verification method and device, electronic equipment and storage medium | |
| CN116974801A (en) | Transaction link abnormality analysis method and device, storage medium and computer equipment | |
| CN114518885A (en) | Information collection method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |