CN117156169A - Data processing method, device, equipment and storage medium - Google Patents

Data processing method, device, equipment and storage medium Download PDF

Info

Publication number
CN117156169A
CN117156169A CN202311024983.8A CN202311024983A CN117156169A CN 117156169 A CN117156169 A CN 117156169A CN 202311024983 A CN202311024983 A CN 202311024983A CN 117156169 A CN117156169 A CN 117156169A
Authority
CN
China
Prior art keywords
client
data
liveness
partition
sub
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311024983.8A
Other languages
Chinese (zh)
Inventor
郭桂成
程彬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Maojia Technology Guangdong Co ltd
Original Assignee
Maojia Technology Guangdong Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Maojia Technology Guangdong Co ltd filed Critical Maojia Technology Guangdong Co ltd
Priority to CN202311024983.8A priority Critical patent/CN117156169A/en
Publication of CN117156169A publication Critical patent/CN117156169A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H04N21/23106—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving caching operations
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/901—Indexing; Data structures therefor; Storage structures
    • G06F16/9017—Indexing; Data structures therefor; Storage structures using directory or table look-up
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G06F16/90335—Query processing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/231—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
    • H04N21/23113—Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion involving housekeeping operations for stored content, e.g. prioritizing content for deletion because of storage space restrictions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/25—Management operations performed by the server for facilitating the content distribution or administrating data related to end-users or client devices, e.g. end-user or client device authentication, learning user preferences for recommending movies
    • H04N21/258—Client or end-user data management, e.g. managing client capabilities, user preferences or demographics, processing of multiple end-users preferences to derive collaborative data
    • H04N21/25866—Management of end-user data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/27—Server based end-user applications
    • H04N21/274—Storing end-user multimedia data in response to end-user request, e.g. network recorder
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

The invention relates to the field of computers and discloses a data processing method, a device, equipment and a storage medium, wherein the method comprises the steps of obtaining the request times of each client in a preset time period, and determining an active reference value based on the request times; and determining a target liveness partition of the client according to the liveness reference value, and storing the data sent by the client to the target liveness partition corresponding to the client when the data sent by the client is obtained. Because the data of different clients are stored through a plurality of liveness partitions, the liveness reference value of the client is determined according to the request times of the client, and the storage resource allocation is carried out based on the liveness reference value of the client, the utilization rate of the database resource is improved, and the condition that resources are wasted due to blind stacking of high hardware performance is effectively avoided. Meanwhile, the information of the client can be updated to the cloud service in time, so that the problems of blockage of a database, untimely data updating and the like are avoided.

Description

Data processing method, device, equipment and storage medium
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and storage medium.
Background
With the continuous enhancement of the role of the intelligent television in each home environment, different from the prior art of only watching television programs, a plurality of internet of things devices can be displayed on the intelligent television by virtue of the large-screen visual advantage, and the intelligent television gradually trends to a visual center of the home internet of things.
Many smart television products in the market are designed to be in a standby low-power consumption state when the screen is turned off, and internet data can be still sent and received at the bottom layer instead of actually powering off and off. Thus, the cloud service interacting with the television device needs to take on a more heavy processing task, manage data sent by tens of millions of intelligent television terminals every day at random, process and analyze business, distribute data, and store the latest real-time data of each device. How to solve the problem that the database table is too huge due to the read-write of high-frequency data when a large number of terminals of intelligent televisions are connected with the network is an urgent problem to be solved. In the prior art, only a simple database list table is used for processing read-write tasks, so that the problems of blocking and untimely updating of real-time data are easily caused.
The foregoing is provided merely for the purpose of facilitating understanding of the technical solutions of the present invention and is not intended to represent an admission that the foregoing is prior art.
Disclosure of Invention
The invention mainly aims to provide a data processing method, a device, equipment and a storage medium, and aims to solve the technical problems that in the prior art, a read-write task is processed through a simple database list table, and in the Internet of things equipment with high real-time requirements, the data processing mode is easy to cause blocking, real-time data updating is not timely and the like.
To achieve the above object, the present invention provides a data processing method, including the steps of:
acquiring the request times of each client in a preset time period, and determining an active reference value based on the request times;
determining a target liveness partition of each client according to the liveness reference value;
and when the data sent by the client is obtained, storing the data sent by the client to a target activity partition corresponding to the client.
Optionally, before the step of determining the target liveness partition of each client according to the liveness reference value, the method further includes:
dividing a database into M initial liveness partitions;
and obtaining the client proportion of each initial liveness partition, and dividing each client into M initial liveness partitions according to the client proportion.
Optionally, the step of obtaining the number of requests of each client in a preset time period and determining the active reference value based on the number of requests includes:
acquiring the total number of accessed clients and the request times of each client in a preset time period;
the proportion of the clients is adjusted according to the total number of the clients and the request times of the clients in a preset time period;
and determining an active reference value of each client based on the adjusted client proportion.
Optionally, when obtaining the data sent by the client, the step of storing the data sent by the client to a target activity partition corresponding to the client includes:
when the data sent by the client is obtained, obtaining the history data received in a history time period, and performing data change verification on the data sent by the client based on the history data;
and if the data sent by the client is not changed relative to the historical data, discarding the data sent by the client until the data sent by the client is changed, and storing the changed data into a target activity partition corresponding to the client.
Optionally, a client sub-table is arranged in the liveness partition, and the client sub-table is used for storing client data;
the step of storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained includes:
acquiring a client identifier of the client;
searching a client sub-table in the target liveness partition according to the client identifier;
and when the client sub-table corresponding to the client identifier is found, storing the data sent by the client into the client sub-table.
Optionally, before the step of obtaining the client identifier of the client, the method further includes:
when a new client is accessed, dividing the new client into corresponding client liveness partitions according to a preset table;
and establishing a new client sub-table corresponding to the client in the client liveness partition.
Optionally, after the step of determining the target liveness partition of each client according to the liveness reference value, the method further includes:
if the target liveness partition corresponding to the client changes, acquiring client sub-table data in the target liveness partition before the change;
Establishing a new client sub-table in the target activity partition after the change, and transplanting the client sub-table data into the new client sub-table;
and obtaining the new sub-table information of the client sub-table, and sending the sub-table information to a device data control module for information synchronization.
In order to achieve the above object, the present invention also provides a data processing apparatus including:
the system comprises an active standard control module, a reference value generation module and a reference value generation module, wherein the active standard control module is used for acquiring the request times of each client in a preset time period and determining an active reference value based on the request times;
the activity partition determining module is used for determining a target activity partition of each client according to the activity reference value;
and the data storage module is used for storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained.
In order to achieve the above object, the present invention also proposes a data processing apparatus comprising: a memory, a processor and a data processing program stored on said memory and executable on said processor, said data processing program being configured to implement the steps of data processing as described.
In addition, in order to achieve the above object, the present invention also proposes a storage medium having stored thereon a data processing program which, when executed by a processor, implements the steps of the data processing method as described above.
The invention provides a data processing method, which comprises the steps of obtaining the request times of each client in a preset time period, and determining an active reference value based on the request times; and determining a target liveness partition of the client according to the liveness reference value, and storing the data sent by the client to the target liveness partition corresponding to the client when the data sent by the client is obtained. Because the data of different clients are stored through a plurality of liveness partitions, the liveness reference value of the client is determined according to the request times of the client, and the storage resource allocation is carried out based on the liveness reference value of the client, the utilization rate of the database resource is improved, and the condition that resources are wasted due to blind stacking of high hardware performance is effectively avoided. Meanwhile, the information of the client can be updated to the cloud service in time, so that the problems of blockage of a database, untimely data updating and the like are avoided.
Drawings
FIG. 1 is a schematic diagram of a data processing apparatus of a hardware operating environment in which embodiments of the present invention are directed;
FIG. 2 is a flow chart of a first embodiment of a data processing method according to the present invention;
FIG. 3 is a flow chart of a second embodiment of the data processing method of the present invention;
FIG. 4 is a flowchart of an application scenario of the data processing method of the present invention;
fig. 5 is a block diagram showing the structure of a first embodiment of the data processing apparatus of the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
With reference to fig. 1, fig. 1 is a schematic diagram of a data processing device structure of a hardware running environment according to an embodiment of the present invention.
As shown in fig. 1, the data processing apparatus may include: a processor 1001, such as a central processing unit (Central Processing Unit, CPU), a communication bus 1002, a user interface 1003, a network interface 1004, a memory 1005. Wherein the communication bus 1002 is used to enable connected communication between these components. The user interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and the optional user interface 1003 may further include a standard wired interface, a wireless interface. The network interface 1004 may optionally include a standard wired interface, a Wireless interface (e.g., a Wireless-Fidelity (Wi-Fi) interface). The Memory 1005 may be a high-speed random access Memory (Random Access Memory, RAM) or a stable nonvolatile Memory (NVM), such as a disk Memory. The memory 1005 may also optionally be a storage device separate from the processor 1001 described above.
It will be appreciated by those skilled in the art that the structure shown in fig. 1 does not constitute a limitation of the data processing apparatus and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a data processing program may be included in the memory 1005 as one type of storage medium.
In the data processing apparatus shown in fig. 1, the network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the data processing apparatus of the present invention may be provided in a data processing apparatus that calls a data processing program stored in the memory 1005 through the processor 1001 and executes the data processing method provided by the embodiment of the present invention.
An embodiment of the present invention provides a data processing method, referring to fig. 2, fig. 2 is a schematic flow chart of a first embodiment of the data processing method of the present invention.
In this embodiment, the data processing method includes the following steps:
step S10: and acquiring the request times of each client in a preset time period, and determining an active reference value based on the request times.
It should be noted that, the execution body of the method of this embodiment may be a terminal device having functions of data processing and program running, such as a computer, a server, a cloud service application, or an electronic device having the same or similar functions, such as the above-mentioned data processing device. The present embodiment and the following embodiments will be described below by taking a data processing apparatus (hereinafter referred to as a processing apparatus) as an example.
It can be understood that the client may be a smart tv, an internet of things device terminal, a personal computer, or other devices, and the active reference value of each client may be determined by obtaining the number of requests of the client in a preset time period.
It should be noted that the above-mentioned preset period may be the last 24 hours, 12 hours or 6 hours, which is not limited in this embodiment.
It will be appreciated that the number of requests may be the total number of requests sent by the client to the processing device or the number of times the client data is processed, and the requests may include a data acquisition request, a data storage request, etc., which is not limited in this embodiment.
It should be explained that the above-mentioned client data, i.e. the data sent by the client stored in the liveness partition.
It should be noted that, the data in this embodiment and the embodiments described below may be real-time data, that is, real-time data collected or acquired by each internet of things device connected to the client.
In one implementation, the smart television may send data to the cloud service application background, and the cloud service application may record the number of requests of the smart television for the last 24 hours in a cache manner.
It should be noted that, the processing device of this embodiment may include an activity level standard module, where the activity level standard module may obtain a large amount of real-time data and the total number of actual clients accessed by the processing device, so as to perform correction and division standard adjustment, and provide an activity level reference value for each client.
It should be explained that the above-mentioned activity level reference value is a reference value for partitioning the activity level of the client, and the activity level partition of the client may be determined by using the activity level reference value.
In a specific implementation, the processing device obtains the number of requests of each client in a preset time period, and determines an active reference value based on the number of requests.
Step S20: and determining the target liveness partition of each client according to the liveness reference value.
It should be noted that, the target liveness partition is the liveness partition corresponding to the client. When the processing device receives the data sent by the client, the data may be stored to a corresponding liveness partition, that is, a target liveness partition. The target activity partition corresponding to the client is not invariable, the target activity partition can be changed according to the actual activity, and the ascending area or the descending area of the activity partition is performed, namely, the target activity partition of the client is determined according to the activity reference value.
It can be understood that the reference for performing the activity partition change is an activity level reference value, and the activity level reference value of the client can be determined according to the number of requests in the preset time period of the client, so as to determine whether to perform the target activity partition change.
In one implementation manner, the background service of the smart television uses a cache to record the actual real-time request times of each device in the latest day, and divides each client into 5 active partitions with relative activities of high A, high B, low C, low D and low E according to the different request times.
It may be appreciated that, in order to generate the liveness partition, before the step of determining the target liveness partition of each client according to the liveness reference value, the method further includes:
The database is partitioned into M initial liveness partitions.
It is understood that the database is a database in the processing device that stores the client data. The database may be partitioned into liveness partitions to obtain each initial liveness partition.
Before the data processing method is performed, the database may be divided into M initial liveness partitions, and the scheme of the present application is described by taking 5 liveness partitions as an example in this embodiment and the following embodiments, but this does not limit the number of liveness partitions in this embodiment. In practical applications, M may be 2, 3, 4 or other numbers.
And obtaining the client proportion of each initial liveness partition, and dividing each client into M initial liveness partitions according to the client proportion.
It should be noted that, the above-mentioned client proportion is the proportion of the number of clients in the liveness partition to the number of all clients. The processing device of the present embodiment may be connected with a plurality of clients, and in order to determine the client data stored in each liveness partition, the present embodiment divides the clients connected with the processing device into corresponding liveness partitions through the client proportion.
It should be explained that, each client is divided into M initial liveness partitions, that is, storage client data corresponding to the M liveness partitions is determined, so that when data sent by a client is received, the data is stored into an initial liveness partition corresponding to the client, that is, a target liveness partition. In one application, a mapping relationship between the client and the liveness partition may be established, which is not limited in this embodiment.
In one implementation, when the number of initial liveness partitions is 5, client partitioning is performed on each liveness partition according to the client proportion of 10%, 15%, 20%, 25% and 30%, so as to determine a target liveness partition of each client.
It should be explained that the above-mentioned client proportion of the liveness partition indicates the number of clients involved in the data preservation in the liveness partition. For example, when the number of the accessed actual clients is 1000, the database is divided into 5 liveness partitions, the number of the clients occupied by the liveness partition A is 100, the number of the clients occupied by the liveness partition B is 150, the number of the clients occupied by the liveness partition C is 200, the number of the clients occupied by the liveness partition D is 250, and the number of the clients occupied by the liveness partition E is 300.
It should be understood that, in practical applications, the number of requests sent by each client is different, and there are cases where the number of requests sent by some clients is greater and the number of requests sent by some clients is less. And the client with more sending request times is arranged in the liveness partition B, and the client with less request times is arranged in the liveness partition D, so that the request processing capacity of each liveness partition is kept in a relatively balanced state.
It should be noted that, in order to further maintain performance balance of each liveness partition, the step of obtaining the number of requests of each client in a preset time period and determining an activity reference value based on the number of requests includes:
step S11: and obtaining the total number of accessed clients and the request times of the clients in a preset time period.
It should be appreciated that the total number of clients described above is the number of clients actually accessed in the processing device.
In a specific implementation, the processing device obtains the total number of clients and records the number of requests of each client in a preset time period.
Step S12: and adjusting the client proportion according to the total number of the clients and the request times of the clients in a preset time period.
It can be appreciated that, since the number of requests of each client is not constant, the client scaling needs to be performed according to the number of requests of each client and the total number of clients, so that approximately equal amounts of read-write tasks are processed in each liveness partition.
It should be appreciated that when a client performs data reading or data storage, a corresponding data request is sent to start the read/write task. The approximately equal number of read-write tasks in this embodiment may refer to approximately equal data amount, or may refer to approximately equal number of read-write tasks, which may be adjusted according to specific applications, which is not limited in this embodiment.
In an application scenario, the number of processing and reading tasks of each liveness partition in this embodiment is approximately equal, and the liveness partition a contains 103 pieces of data of clients, and the total processing request number is 100010 within 24 hours; the activity partition B contains 152 pieces of data of clients, and the total processing request times is 100005 pieces within 24 hours; the activity partition C contains 199 pieces of data of clients, and the total processing request times is 100003 pieces within 24 hours; the data of 248 clients are contained in the liveness partition D, the total processing request times are 100007 in 24 hours, the data of 298 clients are contained in the liveness partition E, and the total processing request times are 100001 in 24 hours. And adjusting the proportion of the clients of each liveness partition according to the total number of the clients and the request times of the clients in a preset time period, so that the request times processed by each liveness partition are similar.
Step S13: and determining an active reference value of each client based on the adjusted client proportion.
It can be understood that after the proportion of the clients is determined, the active reference value of each client can be determined according to the number of requests of each client, and the target activity partition corresponding to each client can be determined according to the active reference value of each client.
In one implementation, the active reference value of the client corresponds to the number of requests of the client, and each client is divided into 5 active partitions with relative activities of high a, high B, medium C, low D and low E according to the active reference value of the client, so that the total number of requests processed in each active partition is similar.
Step S30: and when the data sent by the client is obtained, storing the data sent by the client to a target activity partition corresponding to the client.
It can be understood that, when determining the target activity partition corresponding to the client, if the data and the data storage command sent by the client are received, the data sent by the client is stored to the target activity partition corresponding to the client. And if a data reading command sent by the client is received, acquiring corresponding data from the target activity partition corresponding to the client according to the data reading command.
In a specific implementation, when the processing device obtains data sent by the client, the processing device stores the data sent by the client to a target activity partition corresponding to the client.
The embodiment provides a data processing method, which comprises the steps of obtaining the request times of each client in a preset time period, and determining an active reference value based on the request times; and determining a target liveness partition of the client according to the liveness reference value, and storing the data sent by the client to the target liveness partition corresponding to the client when the data sent by the client is obtained. Because the data of different clients are stored through a plurality of liveness partitions, the liveness reference value of the client is determined according to the request times of the client, and the storage resource allocation is carried out based on the liveness reference value of the client, the utilization rate of the database resource is improved, and the condition that resources are wasted due to blind stacking of high hardware performance is effectively avoided. Meanwhile, the information of the client can be updated to the cloud service in time, so that the problems of blockage of a database, untimely data updating and the like are avoided.
Based on the above-mentioned first embodiment of the data processing method of the present invention, in order to verify the data sent by the client, a second embodiment of the data processing method of the present invention is provided, and referring to fig. 4, fig. 4 is a schematic flow chart of the second embodiment of the data processing method of the present invention.
As shown in fig. 2, in this embodiment, when obtaining data sent by the client, the step of storing the data sent by the client to a target activity partition corresponding to the client includes:
step S31: when the data sent by the client is obtained, the historical data received in the historical time period is obtained, and data change verification is carried out on the data sent by the client based on the historical data.
In practical applications, the client may send a plurality of pieces of identical data to the processing device due to network fluctuations, device delays, and the like. In this case, in order to avoid excessive redundant data, a real-time data change verification module may be included in the processing apparatus. The real-time data change verification module can perform change verification on the data uploaded by the client.
It should be explained that the above-mentioned change verification is a step of verifying whether the data uploaded by the client is changed. The data uploaded by the client can be changed and verified through the real-time data change verification module, so that the data uploaded by the client is processed.
It should be noted that the above-mentioned historical time period may be a time period of the device according to the user's requirement, for example, one second, two seconds or three seconds, which is not limited in this embodiment.
It should be noted that, the history data is data sent by the client and received by the processing device in the history period.
It should be understood that the history data may include one or more pieces of data, and when the data change verification is performed, if there is the same data as the data sent by the client in the history data, it is determined that the data is not changed, otherwise it is determined that the data is changed.
In one implementation, a processing device may include a cache region in which historical data received over a historical time period may be stored. The processing equipment can compare the data according to the data identification of the historical data and the data identification of the data sent by the client, so as to judge whether the historical data has the same data as the data sent by the client, and complete the data change verification.
In a specific implementation, when data sent by a client is obtained, historical data received in a historical time period is obtained, and data change verification is performed on the data sent by the client based on the historical data.
Step S32: and if the data sent by the client is not changed relative to the historical data, discarding the data sent by the client until the data sent by the client is changed, and storing the changed data into a target activity partition corresponding to the client.
It can be understood that when uploading data of a client, a plurality of pieces of same data may be uploaded, by performing change verification on the data uploaded by the client, if the data sent by the client is unchanged relative to the historical data, the data is discarded until the data sent by the client is changed, and the changed data is stored in a target activity partition corresponding to the client.
In a specific implementation, if the data sent by the client is not changed relative to the historical data, discarding the data sent by the client until the data sent by the client is changed, and storing the changed data to a target activity partition corresponding to the client.
Further, in order to store data sent by the client, a client sub-table corresponding to the client is built in the liveness partition, and the client sub-table is used for storing the client data. Specifically, the step of storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained includes: acquiring a client identifier of the client; according to the client identification, searching a client sub-table in the target liveness partition according to the client identification; and when the client sub-table corresponding to the client identifier is found, storing the data sent by the client into the client sub-table.
It will be appreciated that a client identifier may be included in the client, which may be used to identify the unique client. And the client sub-table comprises a client sub-table identification corresponding to the client identification. Specifically, a corresponding client sub-table identifier can be obtained through the client identifier, a client sub-table in the target liveness partition is determined according to the client sub-table identifier, and data sent by the client are stored in the client sub-table.
It should be noted that, the processing device includes a device data control module for controlling the data of the client, where the device data control module may obtain the mapping relationship between the client and the client sub-table through the sub-table information of the client sub-table, and store the data sent by the client into the client sub-table based on the mapping relationship.
In one implementation manner, the internet of things device connected with the client is also provided with a device identifier capable of being uniquely identified, and the device identifier is sent to the client together when the internet of things device sends data. When the client transmits data, the data such as the equipment identifier, the client identifier and the like are transmitted to the target liveness partition, the processing equipment acquires a client sub-table corresponding to the client according to the client identifier, and the data acquired by the Internet of things equipment is stored in a field corresponding to the client sub-table according to the equipment identifier.
It should be noted that, the number of the client sub-tables is related to the total number of the active partitions responsible for clients where the client sub-tables are located, and the number of devices recorded by a single client sub-table is not more than 50 ten thousand.
In one implementation manner, before the step of obtaining the client identifier of the client, in order to store the data of the newly added client, the method further includes:
when a new client is accessed, dividing the new client into corresponding client liveness partitions according to a preset table;
and establishing a new client sub-table corresponding to the client in the client liveness partition.
It will be appreciated that when accessing a new client, the liveness partition of the new client needs to be partitioned to store the data sent by the new client to the corresponding liveness partition.
It should be understood that the preset table is a table of the corresponding liveness partition of the new client set by the user. Since the new client has no historical request times, the liveness partition of the new client needs to be divided by a preset table. For example, the number of liveness partitions is 5, and when a new client is accessed, the newly accessed client is divided into a default liveness partition, namely, liveness partition D according to a preset table.
It can be appreciated that by identifying the client identifier, it can be determined whether the client is a new client or an accessed client.
It should be understood that, if the client is accessed, the target activity partition corresponding to the client can be obtained, and the steps of data storage, inquiry, update, modification, deletion and the like are performed.
It can be appreciated that when determining the client liveness partition corresponding to the new client, a client sub-table corresponding to the new client can be established.
It should be noted that, since the target liveness partition is determined based on the liveness reference value, the liveness reference value is determined based on the number of requests in the preset time period of the client, and since there may be a large difference in the number of requests sent by the client, there may be a case where the target liveness partition of the client is changed, in order to cope with this, the step of determining the target liveness partition of each client according to the liveness reference value further includes:
if the target liveness partition corresponding to the client changes, acquiring client sub-table data in the target liveness partition before the change;
Establishing a new client sub-table in the target activity partition after the change, and transplanting the client sub-table data into the new client sub-table;
and obtaining the new sub-table information of the client sub-table, and sending the sub-table information to a device data control module for information synchronization.
It can be understood that if the number of request changes in the preset period of time of the client is greatly increased, the target liveness partition corresponding to the client needs to be lifted, and the client sub-table corresponding to the client in the old target liveness partition is transplanted to the target liveness partition after lifting, and vice versa.
It should be understood that, because naming rules of the client sub-tables in different target liveness partitions may be different, when the client sub-table is transplanted, the client sub-tables are named according to the naming rules of the sub-tables in the changed target liveness partition and then stored in the target liveness sub-table.
Specifically, the client sub-table data in the target liveness partition before the change can be obtained, a new client sub-table is built according to the naming rule in the target liveness partition after the change, and the client sub-table data is copied to the new client sub-table, so that the migration of the client sub-table data is realized.
It should be appreciated that in order to maintain data brevity and reduce database redundancy, when the client sub-table data migration is successful, the client sub-table corresponding to the client in the target liveness partition before the change may be deleted.
It should be noted that, the processing device is also provided with a device data control module, and the device data control module may be used for performing control such as data storage, data reading, data modification, and data deletion. When the newly built client sub-table or the client sub-table changes in position, the information of the client sub-table can be synchronized to the equipment data control module, so that the equipment data control module can update the information of the client sub-table in time, and the data operation is more accurate.
It should be understood that the information of the client sub-table may include information of a client to which the client sub-table belongs, a sub-table related device, a sub-table size, a sub-table name, and the like, which is not limited in this embodiment.
When the data sent by the client is obtained, the embodiment performs data change verification on the data sent by the client; and discarding the data if the data sent by the client is not changed until the data sent by the client is changed, and storing the changed data to the target activity partition corresponding to the client. By judging the occurrence of data of the client, triggering the data storage task when the data is changed substantially, the interference of repeated data to the database is avoided, and the storage of useless data is reduced.
Referring to fig. 4, fig. 4 is a schematic diagram of an application scenario of the data processing method of the present invention.
As shown in fig. 4, in this application scenario, the processing device includes three liveness partitions, namely, liveness partition a, liveness partition B, and liveness partition C. Each liveness partition comprises a client sub-table 1, a client sub-table 2, a client sub-table 3 and a client sub-table N.
The processing equipment can record the request times of the intelligent televisions and the number of the intelligent televisions through the recording buffer module, determine an activity reference value based on the request times and the number of the intelligent televisions through the activity standard control module, and determine the activity partition corresponding to each intelligent television based on the activity reference value.
The processing equipment can also acquire the data sent by the intelligent television through the record caching module, carry out change verification on the data through the data change verification module, generate a data storage task if the data are changed, and send the data storage task to the equipment data control module. The device data control module may store data of the data storage task to a client partition table of the corresponding liveness partition according to the liveness reference value.
When the newly built client sub-table or the client sub-table changes in position, the information of the client sub-table can be transmitted back to the equipment data control module, so that the equipment data control module can update the information in time.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium stores a data processing program, and the data processing program realizes the steps of the data processing method when being executed by a processor.
Based on the first embodiment of the data processing method of the present invention, a first embodiment of the data processing apparatus of the present invention is provided, and referring to fig. 5, fig. 5 is a block diagram of the first embodiment of the data processing apparatus of the present invention.
As shown in fig. 5, a data processing apparatus according to an embodiment of the present invention includes:
the active standard control module 501 is configured to obtain a number of requests of each client in a preset time period, and determine an active reference value based on the number of requests;
an activity partition determining module 502, configured to determine a target activity partition of each client according to the activity reference value;
and the data storage module 503 is configured to store, when obtaining data sent by the client, the data sent by the client to a target activity partition corresponding to the client.
The embodiment provides a data processing method, which comprises the steps of obtaining the request times of each client in a preset time period, and determining an active reference value based on the request times; and determining a target liveness partition of the client according to the liveness reference value, and storing the data sent by the client to the target liveness partition corresponding to the client when the data sent by the client is obtained. Because the data of different clients are stored through a plurality of liveness partitions, the liveness reference value of the client is determined according to the request times of the client, and the storage resource allocation is carried out based on the liveness reference value of the client, the utilization rate of the database resource is improved, and the condition that resources are wasted due to blind stacking of high hardware performance is effectively avoided. Meanwhile, the information of the client can be updated to the cloud service in time, so that the problems of blockage of a database, untimely data updating and the like are avoided.
Further, the liveness partition determination module 502 is further configured to divide the database into a preset number of initial liveness partitions; and setting the proportion of the clients of each initial liveness partition to obtain a target liveness partition.
Further, the active standard control module 501 is further configured to obtain a total number of accessed clients, and record a number of requests of each client in a preset time period; the proportion of the clients is adjusted according to the total number of the clients and the request times of the clients in a preset time period; and determining an active reference value of each client based on the adjusted client proportion.
Based on the first embodiment of the data processing device of the present invention, a second embodiment of the data processing device of the present invention is presented.
In this embodiment, the data storage module 503 is further configured to, when acquiring data sent by the client, acquire historical data received in a historical time period, and perform data change verification on the data sent by the client based on the historical data; and if the data sent by the client is not changed relative to the historical data, discarding the data sent by the client until the data sent by the client is changed, and storing the changed data into a target activity partition corresponding to the client.
Further, the liveness partition determination module 502 is further configured to obtain a client identifier of the client; and storing the data sent by the client into a corresponding client sub-table of the liveness partition according to the client identifier.
Further, the liveness partition determining module 502 is further configured to divide, when a new client is accessed, the new client into corresponding client liveness partitions according to a preset table; and establishing a client sub-table corresponding to the new client in the client liveness partition.
Further, the liveness partition determining module 502 is further configured to, if a target liveness partition corresponding to the client changes, migrate the client partition table in the target liveness partition before the change to the target liveness partition after the change; and synchronizing the information of the client sub-table to the equipment data control module when the position of the client sub-table corresponding to the client is changed or the client sub-table is newly built.
When the data sent by the client is obtained, the embodiment performs data change verification on the data sent by the client; and discarding the data if the data sent by the client is not changed until the data sent by the client is changed, and storing the changed data to the target activity partition corresponding to the client. By judging the occurrence of data of the client, triggering the data storage task when the data is changed substantially, the interference of repeated data to the database is avoided, and the storage of useless data is reduced.
Other embodiments or specific implementations of the data processing apparatus of the present invention may refer to the above method embodiments, and are not described herein.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. read-only memory/random-access memory, magnetic disk, optical disk), comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.

Claims (10)

1. A method of data processing, the method comprising:
acquiring the request times of each client in a preset time period, and determining an active reference value based on the request times;
determining a target liveness partition of each client according to the liveness reference value;
and when the data sent by the client is obtained, storing the data sent by the client to a target activity partition corresponding to the client.
2. The data processing method of claim 1, wherein prior to the step of determining a target liveness partition for each of the clients based on the liveness reference values, further comprising:
dividing a database into M initial liveness partitions;
and obtaining the client proportion of each initial liveness partition, and dividing each client into M initial liveness partitions according to the client proportion.
3. The data processing method as claimed in claim 2, wherein the step of obtaining the number of requests of each client in a preset period of time and determining the active reference value based on the number of requests comprises:
acquiring the total number of accessed clients and the request times of each client in a preset time period;
the proportion of the clients is adjusted according to the total number of the clients and the request times of the clients in a preset time period;
and determining an active reference value of each client based on the adjusted client proportion.
4. The data processing method as claimed in claim 1, wherein the step of storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained includes:
when the data sent by the client is obtained, obtaining the history data received in a history time period, and performing data change verification on the data sent by the client based on the history data;
and if the data sent by the client is not changed relative to the historical data, discarding the data sent by the client until the data sent by the client is changed, and storing the changed data into a target activity partition corresponding to the client.
5. The data processing method according to any one of claims 1 to 4, wherein a client sub-table is provided in the liveness partition, and the client sub-table is used for storing client data;
the step of storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained includes:
acquiring a client identifier of the client;
searching a client sub-table in the target liveness partition according to the client identifier;
and when the client sub-table corresponding to the client identifier is found, storing the data sent by the client into the client sub-table.
6. The data processing method of claim 5, wherein prior to the step of obtaining the client identification of the client, further comprising:
when a new client is accessed, dividing the new client into corresponding client liveness partitions according to a preset table;
and establishing a new client sub-table corresponding to the client in the client liveness partition.
7. The data processing method of claim 6, wherein after the step of determining the target liveness partition for each of the clients based on the liveness reference values, further comprising:
If the target liveness partition corresponding to the client changes, acquiring client sub-table data in the target liveness partition before the change;
establishing a new client sub-table in the target activity partition after the change, and transplanting the client sub-table data into the new client sub-table;
and obtaining the new sub-table information of the client sub-table, and sending the sub-table information to a device data control module for information synchronization.
8. A data processing apparatus, characterized in that the data processing apparatus comprises:
the system comprises an active standard control module, a reference value generation module and a reference value generation module, wherein the active standard control module is used for acquiring the request times of each client in a preset time period and determining an active reference value based on the request times;
the activity partition determining module is used for determining a target activity partition of each client according to the activity reference value;
and the data storage module is used for storing the data sent by the client to the target activity partition corresponding to the client when the data sent by the client is obtained.
9. A data processing apparatus, the apparatus comprising: a memory, a processor and a data processing program stored on the memory and executable on the processor, the data processing program being configured to implement the steps of data processing according to any one of claims 1 to 7.
10. A storage medium having stored thereon a data processing program which, when executed by a processor, implements the steps of the data processing method according to any one of claims 1 to 7.
CN202311024983.8A 2023-08-14 2023-08-14 Data processing method, device, equipment and storage medium Pending CN117156169A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311024983.8A CN117156169A (en) 2023-08-14 2023-08-14 Data processing method, device, equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311024983.8A CN117156169A (en) 2023-08-14 2023-08-14 Data processing method, device, equipment and storage medium

Publications (1)

Publication Number Publication Date
CN117156169A true CN117156169A (en) 2023-12-01

Family

ID=88909121

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311024983.8A Pending CN117156169A (en) 2023-08-14 2023-08-14 Data processing method, device, equipment and storage medium

Country Status (1)

Country Link
CN (1) CN117156169A (en)

Similar Documents

Publication Publication Date Title
US10853329B2 (en) Distributed file system
US9984013B2 (en) Method, controller, and system for service flow control in object-based storage system
US11005717B2 (en) Storage capacity evaluation method based on content delivery network application and device thereof
US20070276838A1 (en) Distributed storage
CN107357896A (en) Expansion method, device, system and the data base cluster system of data-base cluster
US20140237024A1 (en) Network communication devices and file tracking methods thereof
US20180373615A1 (en) Tunable, efficient monitoring of capacity usage in distributed storage systems
CN101635734A (en) Method and device for storing and managing downloaded data on non-volatile storage medium
CN110310198B (en) Enterprise quota information management method, device, equipment and readable storage medium
CN111597213B (en) Caching method, software server and storage medium
CN104158858A (en) Method and system for accessing distributed cluster file system
US20050234966A1 (en) System and method for managing supply of digital content
CN104050216A (en) File system manager for customized resource allocation
CN106933868A (en) A kind of method and data server for adjusting data fragmentation distribution
JP2019047257A (en) Distribution device and distribution method
CN109684270A (en) Database filing method, apparatus, system, equipment and readable storage medium storing program for executing
CN114020444B (en) Calling system and method for resource service application in enterprise digital middle station
US20220103500A1 (en) Method and device for managing group member, and method for processing group message
CN113342746A (en) File management system, file management method, electronic device, and storage medium
CN111404828A (en) Method and device for realizing global flow control
CN117156169A (en) Data processing method, device, equipment and storage medium
CN113626404B (en) Control method, device and medium for nested directory quota of distributed file system
CN111064700B (en) Downloading method, device and system of cloud game
CN107196962B (en) User access control method and system
US10671311B1 (en) System and method for dynamic data migration

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication