CN110519319A - A kind of method and device dividing subregion - Google Patents
A kind of method and device dividing subregion Download PDFInfo
- Publication number
- CN110519319A CN110519319A CN201810494401.5A CN201810494401A CN110519319A CN 110519319 A CN110519319 A CN 110519319A CN 201810494401 A CN201810494401 A CN 201810494401A CN 110519319 A CN110519319 A CN 110519319A
- Authority
- CN
- China
- Prior art keywords
- unit time
- time
- subregion
- rnn
- hbase table
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application is to belong to the communications field about a kind of method and device for dividing subregion.The described method includes: stored data record, prediction need to occupy in the target time section the first number of the subregion in the Hbase table in Hbase table before being started according to target time section;When first number is greater than the second number, it is more than to preset the subregion of filling rate threshold value as target partition that filling rate is obtained from the second number subregion, and the second number subregion is the pre-assigned subregion for needing to occupy in the target time section in the Hbase table;When the number of the target partition of acquisition is less than or equal to third number, the target partition is divided in the corresponding splitting time of the target partition, the splitting time is later than the acquisition time for obtaining the target partition, and the third number is equal to first number and subtracts second number.The application can reduce impacted number of clients.
Description
Technical field
This application involves the communications field, in particular to a kind of method and device for dividing subregion.
Background technique
Hbase (Hadoop Database) is a kind of high reliability, high-performance, towards column, telescopic distributed storage
System.Only one Region in Hbase table when default one Hbase table of creation, Region is the subregion in Hbase table, is used
In storing data.
The capacity of Region is fixed, it is assumed that the capacity of Region is N, when the data for needing to store are more and more,
When leading to the insufficient space of Region, need to split into the Region Region that two capacity are N, to store more numbers
According to.Hbase has the function of dividing Region automatically at present, can be automatically by the Region points when Region is filled with data
Two new Region are cleaved into, the data in the Region are possibly stored in this two new Region.
During realizing the application, discovery aforesaid way at least has following defects that inventor
Client can not access the Region during Region is split into two new Region.Industry is being provided
It is engaged in the more busy busy period, can store or access a large amount of data into Region, Region is caused to be easy busy
Be filled in period it is concurrent it is estranged split, and have a large amount of client access Region within the busy period, request real-time industry
Business causes to provide real time business without a large amount of client of normal direction within the busy period in this way, generates shadow to a large amount of client
It rings.
Summary of the invention
In order to reduce the number of clients by shadow, the embodiment of the present application provides a kind of method and device for dividing subregion.
The technical solution is as follows:
In a first aspect, the application provides a kind of method for dividing subregion, which comprises
Stored data record in Hbase table, is predicted in the target time section before being started according to target time section
Need to occupy the first number of the subregion in the Hbase table;
When first number is greater than the second number, it is more than default that filling rate is obtained from the second number subregion
The subregion of filling rate threshold value is pre-assigned in the target time section as target partition, the second number subregion
Need to occupy the subregion in the Hbase table;
When the number of the target partition of acquisition is less than or equal to third number, in the corresponding division of the target partition
Between divide the target partition, the splitting time is later than the acquisition time for obtaining the target partition, the third number etc.
Second number is subtracted in first number.
Optionally, described that stored data record in preceding Hbase table is started according to target time section, it predicts in the mesh
Need to occupy the first number of the subregion in the Hbase table in the mark period, comprising:
According to used spatial content in the stored data record total amount of Hbase table and the Hbase table, institute is calculated
State the average amount of every data record in Hbase table;
Pass through the number for the data to be stored record that prediction model prediction generates in the target time section;
According to number, the average amount and subregion capacity that the data to be stored records, calculate in the target
Need to occupy the first number of the subregion in the Hbase table in period.
Optionally, the prediction needs to occupy in the target time section the first number of the subregion in the Hbase table
Before mesh, further includes:
The data record stored in the Hbase table before being started according to the target time section, generates the prediction model.
Optionally, the data record for being started to store in the preceding Hbase table according to the target time section, generates institute
State prediction model, comprising:
Corresponding timestamp is recorded according to the pieces of data in the Hbase table, obtains the first unit time set and the
Two unit time set, the first unit time set include in each unit time stabbed between the second timestamp at the first time
The the first data record number generated, the second unit time set includes every between the second timestamp and third timestamp
The the first data record number generated in a unit time, stamp is the pieces of data record pair in the Hbase table at the first time
From current earliest timestamp in the timestamp answered, it is corresponding that third timestamp is that the pieces of data in the Hbase table records
From current nearest timestamp in timestamp, the second timestamp is located between first time stamp and the third timestamp;
The first parameter value at least one Recognition with Recurrent Neural Network RNN parameter is obtained, according at least one described RNN parameter
The first parameter value be arranged the first RNN RNN parameter, obtain the 2nd RNN;
It is generated according to first unit time set, second unit time set and the 2nd RNN described pre-
Survey model.
Optionally, described according to first unit time set, second unit time set and the 2nd RNN
Generate the prediction model, comprising:
According to the first data record number of each unit time in the first unit time set, pass through described second
RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction
The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre-
Survey model.
Optionally, the method also includes:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter
The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third
RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre-
Survey model.
Optionally, described that the subregion work that filling rate is more than default filling rate threshold value is obtained from the second number subregion
After target partition, further includes:
It determines the date on the same day where the acquisition time for obtaining the target partition, one is selected in the current date
The time point is simultaneously determined as the corresponding splitting time of the target partition by time point, and the splitting time is later than the same day
Preset time point in date.
Optionally, the method also includes:
The data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related
Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field
Race.
Optionally, point that the data packet in the spatial cache of message system is stored in Hbase according to configuration file
Qu Zhong, comprising:
According to the corresponding subject identification of the spatial cache of the message system, obtaining from configuration file includes the theme
The subject correlation message of mark further includes the mark of Hbase table and the set identification of object set in the subject information;
From being obtained in the data packet in the spatial cache of the message system in the corresponding object set of the set identification
The corresponding field contents of each field name;
Each field contents of the acquisition are formed into data record, according to belonging to each field in the object set
The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table by column family.
Second aspect, this application provides a kind of device for dividing subregion, described device includes:
Prediction module is predicted for stored data record in Hbase table before being started according to target time section described
Need to occupy the first number of the subregion in the Hbase table in target time section;
Module is obtained, for being obtained from the second number subregion when first number is greater than the second number
Filling rate is more than the subregion of default filling rate threshold value as target partition, and the second number subregion is pre-assigned in institute
State the subregion for needing to occupy in target time section in the Hbase table;
Module is divided, for dividing in the target when the number of the target partition of acquisition is less than or equal to third number
The corresponding splitting time in area divides the target partition, and the splitting time is later than the acquisition time for obtaining the target partition,
The third number is equal to first number and subtracts second number.
Optionally, the prediction module includes:
First computing unit, for having been used according in the stored data record total amount of Hbase table and the Hbase table
Spatial content, calculate in the Hbase table every data record average amount;
Predicting unit, what the data to be stored for being generated in the target time section by prediction model prediction recorded
Number;
Second computing unit, number, the average amount and subregion for being recorded according to the data to be stored are held
Amount calculates the first number for needing to occupy the subregion in the Hbase table in the target time section.
Optionally, described device further include:
Generation module, the data record for storing in the Hbase table before being started according to the target time section, generates
The prediction model.
Optionally, the generation module includes:
First acquisition unit obtains first for recording corresponding timestamp according to the pieces of data in the Hbase table
Unit time set and the second unit time set, the first unit time set include at the first time between stamp and the second timestamp
Each unit time in the first data record number for generating, second unit time set includes the second timestamp and the
The the first data record number generated in each unit time between three timestamps, stamp is in the Hbase table at the first time
Pieces of data record from current earliest timestamp in corresponding timestamp, third timestamp is each in the Hbase table
Data is recorded from current nearest timestamp in corresponding timestamp, the second timestamp be located at first time stamp with it is described
Between third timestamp;
Second acquisition unit, for obtaining the first parameter value at least one Recognition with Recurrent Neural Network RNN parameter, according to institute
The RNN parameter of the first RNN is arranged in the first parameter value for stating at least one RNN parameter, obtains the 2nd RNN;
Generation unit, for according to first unit time set, second unit time set and described second
RNN generates the prediction model.
Optionally, the generation unit, is used for:
According to the first data record number of each unit time in the first unit time set, pass through described second
RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction
The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre-
Survey model.
Optionally, the generation unit, is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter
The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third
RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre-
Survey model.
Optionally, described device further include:
Determining module, the date on the same day where acquisition time for determining the acquisition target partition, described current
One time point of selection and the time point is determined as the corresponding splitting time of the target partition in date, when the division
Between be later than preset time point in the date on the same day.
Optionally, described device further include:
Memory module, for the data packet in the spatial cache of message system to be stored in Hbase's according to configuration file
In subregion;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related
Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field
Race.
Optionally, the memory module includes:
Third acquiring unit, for the corresponding subject identification of spatial cache according to the message system, from configuration file
It is middle to obtain the subject correlation message including the subject identification, it further include the mark and object of Hbase table in the subject information
The set identification of set;
4th acquiring unit, for obtaining the set identification from the data packet in the spatial cache of the message system
The corresponding field contents of each field name in corresponding object set;
Storage unit, for each field contents of the acquisition to be formed data record, according to the object set
In each field affiliated column family the data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table.
6th aspect, the application provides a kind of non-volatile computer readable storage medium storing program for executing, for storing computer program,
The computer program is loaded and is executed by processor, to realize any optional method of first aspect or first aspect
Instruction.
Technical solution provided by the embodiments of the present application can include the following benefits:
By predicting the first number of subregion in the target time period, according to the first number it can be concluded that in target time section
The third number of the interior subregion for needing to divide.So in the target time period when detecting that filling rate is more than default filling rate threshold value
Target partition when, can be in the more idle sky of the business of offer if the target partition number obtained is less than or equal to third number
The splitting time of not busy period divides target partition, can reduce affected number of clients, while also logical
The target partition number for crossing the control division of third number, avoids dividing excessive subregion, and causes a large amount of subregion
Storage resource waste.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
The application can be limited.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows the implementation for meeting the application
Example, and together with specification it is used to explain the principle of the application.
Fig. 1 is network architecture schematic diagram provided by the embodiments of the present application;
Fig. 2 is a kind of method flow diagram for dividing subregion provided by the embodiments of the present application;
Fig. 3-1 is a kind of method flow diagram for dividing subregion provided by the embodiments of the present application;
Fig. 3-2 is a kind of method flow diagram for generating prediction module provided by the embodiments of the present application;
Fig. 3-3 is the waveform diagram of the record count of unit time provided by the embodiments of the present application;
Fig. 4 is a kind of apparatus structure schematic diagram for dividing subregion provided by the embodiments of the present application;
Fig. 5 is terminal structure schematic diagram provided by the embodiments of the present application.
Through the above attached drawings, it has been shown that the specific embodiment of the application will be hereinafter described in more detail.These attached drawings
It is not intended to limit the range of the application design in any manner with verbal description, but is by referring to specific embodiments
Those skilled in the art illustrate the concept of the application.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to
When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment
Described in embodiment do not represent all embodiments consistent with the application.On the contrary, they be only with it is such as appended
The example of the consistent device and method of some aspects be described in detail in claims, the application.
Referring to Fig. 1, the embodiment of the present application provides a kind of network architecture, comprising:
Data collection terminal 1, message system 2 and distributed memory system 3.It include that at least one caching is empty in message system 2
Between, the corresponding theme of each spatial cache at least one spatial cache.
Message system can be KAFKA, and the spatial cache in message system can be the message queue in KAFKA.Distribution
Formula storage system includes total node and multiple back end, and total node is for managing multiple back end, multiple data section
Point is for storing data.
For each spatial cache in message system, which belongs to this for buffered message system 2 to be received
The data packet of the corresponding theme of spatial cache.
It include configuration file in distributed memory system, which includes at least one subject correlation message and at least
One object set, subject correlation message include at least the collection of subject identification, the mark of Hbase table and object set (Schema)
Close mark;It can also be including cluster belonging to message system cluster information belonging to rowkey create-rule, theme and Hbase table
At least one of information etc., rowkey are the unique identifications for identifying data record.
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field
Race;It can also include field type, whether the field is optional and whether the field indexes at least one of information etc..
Optionally, data collection terminal 1 can be used for acquiring data sequence, include in chronological sequence generating in the data sequence
At least one data packet, in the packet header of each data packet include timestamp and the subject identification of theme that the data packet belongs to,
Data collection terminal 1 is also used to each data packet of acquisition being sent to message system 2.
Optionally, data collection terminal 1 can also dress up each data envelope of acquisition before sending each data packet
Then the data packet of PB (PetaByte, petabyte) protocol format sends the data packet after encapsulation to message system 2 again.
Since each data envelope of acquisition is dressed up the data packet of PB protocol format, PB protocol format by data collection terminal 1
Data packet data volume it is small, and the speed parsed is faster and simpler.
The data packet that collection terminal 1 is sent for receiving data of message system 2, according to the theme mark for including in the data packet
Know, the corresponding theme of the subject identification is determined, by the data pack buffer in the corresponding spatial cache of the theme.
Optionally, it is cached in each spatial cache that distributed memory system can be included according to the configuration file
Data packet is stored in distributed memory system, can be executed the storage by total node in distributed memory system and be operated, can
With are as follows:
For each spatial cache, according to the corresponding subject identification of the spatial cache, obtaining from configuration file includes being somebody's turn to do
The subject correlation message of subject identification further includes the mark and object set of Hbase table in the subject correlation message;From the caching
The corresponding field contents of each field name in the object set are obtained in data packet in space;In each field that will acquire
Hold composition one data record, which is stored in by the Hbase according to the affiliated column family of each field in the object set
In the Region of the corresponding Hbase table of mark of table.
Referring to fig. 2, the embodiment of the present application provides a kind of method for dividing subregion, and the method can be applied to such as Fig. 1
The network architecture that illustrated embodiment provides, and the executing subject of this method can be total node in distributed memory system, packet
It includes:
Step 201: stored data record in Hbase table before being started according to target time section was predicted in the object time
Need to occupy the first number of the Region in Hbase table in section.
Region is subregion, can be the subregion in Hbase table, Region occurs in the other embodiments of the application
Meaning may refer to herein, just no longer introduce one by one.
Optionally, the embodiment of the present application provides the first optional implementation to realize this step, the first is realized
Mode includes following 2011 to 2013 operation, is respectively as follows:
2011: according to used spatial content in the stored data record total amount of Hbase table and Hbase table, calculating
The average amount of every data record in Hbase table.
2012: passing through the number for the data to be stored record that prediction model prediction generates in the target time period.
2013: number, the average amount and the subregion capacity recorded according to data to be stored is calculated in target time section
Inside need to occupy the first number of the subregion in Hbase table.
Optionally, the embodiment of the present application provides second of optional implementation, and second of optional implementation exists
The operation of following steps 200 can also be performed before executing step 201.
Step 200: the data record stored in the Hbase table before being started according to target time section generates prediction model.
Optionally, the first optional implementation and second of optional implementation may be combined to form a kind of division
The method of subregion.
Optionally, in conjunction with above-mentioned second optional implementation, it is optional real that the embodiment of the present application provides the third
Existing mode, in the third optional implementation, above-mentioned steps 210 may include following 2001 to 2003 operation, respectively
Are as follows:
2001: corresponding timestamp being recorded according to the pieces of data in the Hbase table, obtains the first unit time set
Gather with the second unit time, the first unit time set includes when stabbing each unit between the second timestamp at the first time
First data record number of interior generation, the second unit time set include every between the second timestamp and third timestamp
The the first data record number generated in a unit time, it is corresponding to be that pieces of data in Hbase table records for stamp at the first time
From current earliest timestamp in timestamp, third timestamp is that the pieces of data in Hbase table records in corresponding timestamp
From current nearest timestamp, the second timestamp is located at the first time between stamp and third timestamp.
2002: obtaining at least one Recognition with Recurrent Neural Network (Recurrent Neural Networks, RNN) parameter
The RNN parameter of the first RNN is arranged according to the first parameter value of at least one RNN parameter, obtains the 2nd RNN for first parameter value.
2003: prediction model is generated according to the first unit time set, the second unit time set and the 2nd RNN.
Optionally, the first above-mentioned optional implementation, second of optional implementation and the third optional reality
Existing mode may be combined to form a kind of method for dividing subregion.
Optionally, in conjunction with the third above-mentioned optional implementation, the embodiment of the present application provides the 4th kind optionally in fact
Existing mode, the operation of above-mentioned steps 2003 can be in the 4th kind of optional implementation are as follows:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set
At the first model;
Generated in the unit time between the second timestamp and third timestamp by the first model prediction
Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre-
Survey model.
Optionally, the first above-mentioned optional implementation, second of optional implementation, the third optional realization
Mode and the 4th kind of optional implementation may be combined to form a kind of method for dividing subregion.
Optionally, in conjunction in above-mentioned 4th kind optional implementation, the embodiment of the present application provides the 5th kind optionally
Implementation, in the 5th kind of optional implementation, the method for dividing subregion can also include following operation:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter
The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN;Root
Prediction model is generated according to the first unit time set, the second unit time set and the 3rd RNN.
Optionally, the first above-mentioned optional implementation, second of optional implementation, the third optional realization
Mode, the 4th kind of optional implementation and the 5th kind of optional implementation may be combined to form a kind of side for dividing subregion
Method.
Step 202: when the first number is greater than the second number, it is more than pre- that filling rate is obtained from the second number Region
If the Region of filling rate threshold value as target Region, the second number Region be it is pre-assigned in the target time period
Need to occupy the Region in Hbase table.
Optionally, the embodiment of the present application provides the 6th kind of optional implementation, in the 6th kind of optional implementation
In after executing the step 202 operation, the method for dividing subregion can also include following operation:
It determines the date on the same day where the acquisition time for obtaining target partition, selects a time point simultaneously in current date
The time point is determined as the corresponding splitting time of target partition, which is later than the preset time point in the date on the same day.
Optionally, the 6th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional
Implementation, the third optional implementation, in the 4th kind of optional implementation and the 5th kind of optional implementation
Any implementation combine formed division subregion method.
Step 203: corresponding in target Region when the number of the target Region of acquisition is less than or equal to third number
Splitting time divide target Region, which is later than the acquisition time for obtaining target Region, and third number is equal to
First number subtracts the second number.
Wherein, presetting filling rate threshold value is numerical value less than 1, such as can be the numerical value such as 0.8,0.7 or 0.9, when division
Between can be daily offer business it is more idle free time section in sometime.
Optionally, the embodiment of the present application provides the 7th kind of optional implementation, in the 7th kind of optional implementation
In, the method for dividing subregion can also include following 204 operation:
Step 204: the data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file
In;Wherein, which includes at least one subject correlation message and at least one object set, and subject correlation message is at least
Set identification including subject identification, the mark of Hbase table and object set;Object set includes at least one field domain letter
Breath, field domain information include at least field name and the affiliated column family of field.
Optionally, the 7th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional
Implementation, the third optional implementation, in the 4th kind of optional implementation and the 6th kind of optional implementation
Any implementation combine formed division subregion method.
Optionally, the embodiment of the present application provides the 8th kind of optional implementation, in the 8th kind of optional implementation
In above-mentioned 204 operation can be with are as follows:
According to the corresponding subject identification of the spatial cache of message system, obtaining from configuration file includes the subject identification
Subject correlation message further includes the mark of Hbase table and the set identification of object set in the subject information.
Each of corresponding object set of the set identification is obtained from the data packet in the spatial cache of message system
The corresponding field contents of field name.
Each field contents that will acquire form data record, will according to the affiliated column family of each field in the object set
The data record is stored in the subregion of the corresponding Hbase table of mark of Hbase table.
Optionally, the 8th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional
Implementation, the third optional implementation, in the 4th kind of optional implementation and the 6th kind of optional implementation
Any implementation combine formed division subregion method.
In the embodiment of the present application, by predicting the first number of Region in the target time period, according to the first number
It can be concluded that the third number for the Region for needing to divide in the target time period.In the target time period when detecting filling rate
More than the Region of default filling rate threshold value, using the Region as target Region to be divided, wherein default filling rate threshold
Value is less than 1 value, so target Region is also not stored full, can not divide at this time to target Region, but
Offer business it is more idle free time section splitting time target Region is divided, in this way can provide business compared with
Target Region continues to provide real time business in the busy busy period, avoids largely objective without normal direction within the busy period
Family end provides real time business.Target Region is divided in section during idle time, due to free time section access target Region's
Number of clients is less, can reduce affected number of clients.In addition, only in the number of the target Region of acquisition
Mesh just divides target Region when being less than or equal to third number, so being not in a large amount of It is not necessary to the Region divided
It is divided, reduces the waste of resource.
For embodiment shown in Fig. 2, the embodiment of the present application provides a kind of method example for dividing subregion, in this method
At least one target time section can be defined in example in advance, and configured in advance needs to occupy in each target time section
The second number of Region in Hbase table.
Wherein, Hbase table has the function of dividing Region automatically, and the function is for automatic when some Region is filled with
The Region is split into two new Region, but in the embodiment of the present application, it can be by the automatic division Region of the Hbase table
Function close.
Optionally, target time section can be week age, one month or two months etc..Each object time
The duration of section can be equal, can also be unequal.The second number for the Region of each target time section configuration can be equal,
It can also be unequal.
It is assumed that by taking target time section is one month as an example, five target time sections of predefined configure each mesh
The Region number for needing to occupy in the mark period is 6.I.e. in next five months, Hbase is can be used in every month
Six Region in table carry out storing data record.
For any one target time section, if the data record generated in the target time section is more, cause for the mesh
The Region for the second number that the mark period distributes in advance is insufficient, then in the second number Region in the target time section
Some or all of Region divided, to divide out more Region.Referring to Fig. 3-1, the method for division Region
Include:
Step 301: being held according to space used in the stored data record total number of Hbase table and the Hbase table
Amount calculates the average amount of every data record in the Hbase table.
It include at least one Region in Hbase table, the capacity of each Region is equal, for storing at least one data
Record, each data record be in the received data packet of message system include some or all of field contents composition.
The step of this step can be by following 3011 to 3013 calculate being averaged for every data record in Hbase table
Data volume, comprising:
3011: stored data record number in each Region in Hbase table is obtained, according in each Region
Stored data record number calculates the stored data record total number of the Hbase table.
Optionally, directly the data record stored in the Region can be counted, obtains having deposited in the Region
The data record number of storage.
3012: obtaining used spatial content in each Region in Hbase table, used according to each Region
Spatial content calculate used spatial content in the Hbase table.
For each Region in Hbase table, data record stores item by item in the Region.The Region
Attribute information in include the current used spatial content of the Region, can also include the space of current Region free time
Capacity.
For used spatial content in each Region in the Hbase table, can believe from the attribute of the Region
It directly reads to obtain in breath.
3013: according to used spatial content in the stored data record total number of Hbase table and the Hbase table,
Calculate the average amount of every data record in the Hbase table.
It can be obtained in the Hbase table by used spatial content in the Hbase table divided by the data record total number
Every data record average amount.
Step 302: passing through the number for the data to be stored record that prediction model prediction generates in the target time period.
Target time section in this step is any one target time section of definition, and above-mentioned steps 301 and 302 are in the target
Period executes before starting.Optionally, the previous target that above-mentioned steps 301 and 302 can be adjacent in the target time section
The last day of period executes.
In this step, the initial time of objective time interval and end time can be input to prediction model, passes through prediction
The number for the data to be stored record that model prediction generates in the target time period.
Optionally, before executing this step, prediction model can also be generated.The operation of the generation prediction model can be with
Are as follows: the data record stored in Hbase table before being started according to target time section generates the prediction model.Referring to Fig. 3-2, the instruction
Drill make detailed process can be with are as follows:
3021: corresponding timestamp being recorded according to pieces of data stored in Hbase table, obtains the first unit time collection
It closes and the second unit time gathered.
Wherein, the first unit time set includes producing in each unit time stabbed between the second timestamp at the first time
The first raw data record number, the second unit time set include each unit between the second timestamp and third timestamp
The the first data record number generated in time, stamp is that the pieces of data in Hbase table records corresponding timestamp at the first time
In from current earliest timestamp, third timestamp is that the pieces of data in Hbase table records in corresponding timestamp from current
Nearest timestamp, the second timestamp are located at the first time between stamp and third timestamp.
This step can be with are as follows: can be obtained in the corresponding timestamp of stored each data record from the Hbase table from
Current time earliest first time stamp and the third timestamp nearest from current time, according to the time span of unit time,
Period between first time stamp and the second timestamp is divided to obtain S unit time;According in the Hbase table
The corresponding timestamp of each data record of storage determines each data generated in each unit time in the S unit time
Record counts the first data record number of the data record generated in each unit time;The S unit time is divided
At two parts, a part is to gather the first unit time, and another part is the second unit time set.
First unit time set includes M unit time, and the second unit time set includes N number of unit time, S=M+
N, and the ratio between M and S is default ratio.
Optionally, can also by the first unit time gather in each unit time in the first data record number and
The first data record number in each unit time in second unit time set is zoomed in unit range and is carried out just
Then change.
3022: the parameter value of at least one RNN parameter is obtained, according to the parameter value of at least one RNN parameter setting the
The RNN parameter of one RNN, obtains the 2nd RNN.
At least one RNN parameter include random seed number, learning algorithm, algorithm iteration number, weights initialisation method,
At least one in direction of optimization method, learning rate, input layer, output layer and Recursive Networks algorithm etc..
Optionally, when generating prediction model, user can input corresponding first parameter value of each RNN parameter.Accordingly
, in this step, corresponding first parameter value of each RNN parameter of available user's input.
For example, it is 40, learning algorithm corresponding first that user, which can input corresponding first parameter value of random seed number,
Parameter value is SGD, corresponding first parameter value of algorithm iteration number is 1, corresponding first parameter value of weights initialisation method is
Corresponding first parameter value of Xavier, optimization method is Adadelta, corresponding first parameter value of learning rate is 0.0004, input
Corresponding first parameter value of layer is 1, corresponding first parameter value of output layer is 10 corresponding with the direction of Recursive Networks algorithm the
One parameter value be it is preceding to.Then, each first parameter value for obtaining user's input is arranged the first RNN's according to each first parameter value
Each parameter.
3023: according to the first data record number of each unit time in the first unit time set, passing through second
RNN generates the first model.
Optionally, the first data record number of each unit time in the first unit time being gathered is input to
In 2nd RNN, the output of the 2nd RNN is obtained as a result, the output result is a function, using the function as the first model.
After generating the first model, it can be gathered according to the first model, the first unit time set and the second unit time, be pressed
Following 3024 and 3025 operation generates prediction model.
3024: being generated in unit time between the second timestamp and third timestamp by the first model prediction
The second data record number, obtain the third unit time set.
Optionally, the second timestamp and third timestamp are input in the first model, the first model prediction simultaneously exports the
The the second data record number generated in unit time between two timestamps and third timestamp.
Wherein, when the unit time number and the second unit time that third unit time set includes gather the unit for including
Between number it is equal, i.e., the third unit time set include N number of unit time.
3025: when the second unit time set and third unit time set meet preset condition, the first model is true
It is set to prediction model.
For the same unit time in the second unit time set and third unit time set, the unit time is calculated
Difference between interior the first data record number and the second data record number, obtains the difference of the unit time;By above-mentioned
The difference of each unit time in N number of unit time is calculated in mode, obtains from the difference of N number of unit time super
Cross each difference of preset difference value threshold value;If the number of each difference obtained and the ratio of N are more than default fractional threshold, it is determined that
Second unit time set and the third unit time set meets preset condition, if obtain each difference number and ratio
It is less than default fractional threshold, it is determined that the second unit time set and third unit time set are unsatisfactory for preset condition.
3026: when the second unit time set and third unit time set are unsatisfactory for preset condition, obtaining at least one
The RNN parameter of the 2nd RNN is arranged according to the second parameter value of at least one RNN parameter, obtains for second parameter value of a RNN parameter
To the 3rd RNN.
It optionally, can also be in a coordinate system according to of each unit time in S unit time referring to Fig. 3-3
One data record number draws out the first waveform figure line changed with the unit time, and, according to the third unit of prediction
The second data record number of each unit time in N number of unit time in time set draws out one with the unit time
Second waveform figure line of variation;Then the first waveform figure line and the second waveform figure line of drafting are shown.
User can compare first waveform figure line and the second waveform figure line in this way, adjust at least one according to comparing result
The parameter value of RNN parameter to get arriving the second parameter value of at least one RNN parameter, and input it is adjusted this at least one
Second parameter value of RNN parameter.Correspondingly, obtaining the second parameter value of at least one RNN parameter.
Optionally, referring to Fig. 3-3, the horizontal axis of above-mentioned coordinate system is time shaft, and the scale on time shaft is the unit time, is erected
Scale on axis is data record number.
Next, can be gathered according to the first unit time, the second unit time set and the 3rd RNN generate prediction mould
Type.It is accomplished by detail
3027: according to the first data record number of each unit time in the first unit time set, passing through third
RNN generates the second model.
Optionally, the first data record number of each unit time in the first unit time being gathered is input to
In 3rd RNN, the output of the 3rd RNN is obtained as a result, the output result is a function, using the function as the second model.
3028: being generated in unit time between the second timestamp and third timestamp by the second model prediction
The second data record number, obtain the third unit time set.
Optionally, the second timestamp and third timestamp are input in the second model, the second model prediction simultaneously exports the
The the second data record number generated in unit time between two timestamps and third timestamp.
Wherein, when the unit time number and the second unit time that third unit time set includes gather the unit for including
Between number it is equal, i.e., the third unit time set include N number of unit time.
3029: when the second unit time set and third unit time set meet preset condition, the second model is true
It is set to prediction model.
When the second unit time set and third unit time set are unsatisfactory for preset condition, user's adjustment is reacquired
At least one RNN parameter parameter value, then press above-mentioned 3026 to 3029 operation, until when obtaining prediction model.
Step 303: number, the average amount and the Region capacity recorded according to data to be stored is calculated in target
Need to occupy the first number of the Region in Hbase table in period.
This step can be with are as follows: according to the number and the average amount of data to be stored record, calculates in the target
Between the total amount of data of data to be stored record that generates in section;According to the total amount of data and Region capacity, calculate in target
Need to occupy the first number of the Region in Hbase table in period.
Step 304: when the first number is greater than the second number, it is more than pre- that filling rate is obtained from the second number Region
If the Region of filling rate threshold value is as target Region.
Second number Region is to need to occupy the Region in Hbase table in pre-assigned target time section.
When the first number be greater than the second number, show the data record generated in the target time period may it is more, in advance
Second number Region of distribution possibly can not store all data records generated in target time section, it is also necessary to divide
Third number Region out, third number are the differences between the first number and the second number.Thus in application embodiment
The Region that default filling rate threshold value can be reached at first to filling rate in target time section is divided.
Optionally, when message system stores data record in a Region into Hbase table, increasing should
It include used spatial content in the configuration information of Region.It also obtains in the second number Region in real time in this step
Each Region filling rate, acquisition process can be with are as follows: for each Region, reads from the configuration information of the Region
The used spatial content of the Region calculates the Region according to the used spatial content and the Region capacity
Filling rate.
In this step, the filling rate for obtaining each Region in the second number Region in real time, when some
The filling rate of Region is more than default filling rate threshold value, then executes the operation of following steps 305.
Step 305: when acquisition target Region number be less than or equal to third number, determine Region pairs of the target
The splitting time answered, the splitting time are later than the acquisition time for obtaining target Region.
Specifically, it is determined that the date on the same day where the acquisition time in the domain target Region is obtained, choosing in the date in this prior
It selects a time point and the time point is determined as the corresponding splitting time of target Region, which is later than the same day
Preset time point in date.
In this step, daily preset time point can be defined on and divide Region, daily preset time point later
Time later is usually the period more idle in the business of offer.For example, being usually to provide after 10 points of daily night
The business more idle period, so preset time point can be 10 points of night, 10 thirty or 11 points etc..It may be implemented in this way
Region is provided in the offer business more idle period, affected number of clients can be reduced to the greatest extent.
Step 306: divide target Region in the corresponding splitting time of target Region, obtain two it is new
Region。
Division obtain two new Region after, data record stored in target Region can be stored in this two
In a new Region.
Optionally, data record stored in target Region can be averagely stored in two new Region
In.
Optionally, if in the target time period, the number of the target Region obtained is greater than third number, show
Third number Region is divided out, current Region stores the data record generated in the target time period enough, so
Stop the target Region that division obtains.
Wherein, target Region also corresponds to an identification section, which includes that origin identification and end are got to know, to mesh
When marking Region and storing data record, also according to the create-rule of rowkey, the unique identification data record is generated
Rowkey, rowkey is the unique identification of data record, which is located in the identification section.
When target Region is split into two new Region, the available mark positioned at the identification section, at this
A suffix is added on the basis of mark to obtain separating mark.Optionally, the mark of acquisition can be the median of the identification section
Or other numerical value.
The identification section of one of them new Region can be set as to the mark from the origin identification to the separation between mark
Know section, the identification section of another new Region is set as the identification section identified to the end of identification from the separation.
Wherein, it should be understood that the suffix added in the present embodiment is shorter, which only has one or a few units
The characters such as word and/or letter can make separation sign know shorter in this way, keep the origin identification of Region and end of identification shorter.
In this way, existing after obtaining data record and generating the rowkey of data record by the data record storage
When Region, can this according to the rowkey match Region identification section, by the data record storage with this rowkey
In the Region matched, since the starting of the identification section is got to know shorter with end of identification, matched speed can be improved in this way, improve
Storage speed.
In the embodiment of the present application, by predicting the first number of Region in the target time period, according to the first number
It can be concluded that the third number for the Region for needing to divide in the target time period.When third number is not 0, in the object time
When detecting that filling rate is more than the Region of default filling rate threshold value in section, using the Region as target to be divided
Region, wherein default filling rate threshold value is less than 1 value, so target Region is also not stored full, it at this time can not be to mesh
Mark Region is divided, but the splitting time in the more idle free time section of offer business divides target Region
It splits, target Region can continue to provide real time business within the offer business more busy busy period in this way, avoid busy
Real time business is provided without a large amount of client of normal direction in the commonplace period.Target Region is divided in section during idle time, due to sky
The number of clients of not busy period access target Region is less, can reduce affected number of clients.In addition, only
Have and just divide target Region when the number of the target Region of acquisition is less than or equal to third number, so being not in pair
A large amount of Region for being not necessarily to division are divided, and the memory space for having many Region in the target time period is caused not have
Have and used, reduces the waste of storage resource.
Following is the application Installation practice, can be used for executing the application embodiment of the method.It is real for the application device
Undisclosed details in example is applied, the application embodiment of the method is please referred to.
Referring to fig. 4, the embodiment of the present application provides the first optional implementation, provides in the first optional implementation
A kind of device 400 dividing subregion, described device 400 include:
Prediction module 401, for stored data record in Hbase table before being started according to target time section, prediction exists
Need to occupy the first number of the Region in the Hbase table in the target time section;
Module 402 is obtained, is used for when first number is greater than the second number, from the second number Region
Obtaining filling rate is more than to preset the Region of filling rate threshold value as target Region, and the second number Region is preparatory
The Region for needing to occupy in the target time section in the Hbase table of distribution;
Divide module 403, for when the number of the target Region of acquisition be less than or equal to third number when, in the mesh
It marks the corresponding splitting time of Region and divides the target Region, the splitting time, which is later than, obtains the target Region's
Acquisition time, the third number are equal to first number and subtract second number.
Optionally, optional at second of the embodiment of the present application in conjunction with the first optional implementation of the embodiment of the present application
Implementation in, the prediction module 401 includes:
First computing unit, for having been used according in the stored data record total amount of Hbase table and the Hbase table
Spatial content, calculate in the Hbase table every data record average amount;
Predicting unit, what the data to be stored for being generated in the target time section by prediction model prediction recorded
Number;
Second computing unit, number, the average amount and Region for being recorded according to the data to be stored
Capacity calculates the first number for needing to occupy the Region in the Hbase table in the target time section.
Optionally, in conjunction with the first optional implementation of the embodiment of the present application or second of optional implementation, In
In the third optional implementation of the embodiment of the present application, described device 400 further include:
Generation module, the data record for storing in the Hbase table before being started according to the target time section, generates
The prediction model.
Optionally, in conjunction with the third optional implementation of the embodiment of the present application, the 4th kind in the embodiment of the present application can
It selects in implementation, the generation module includes:
First acquisition unit obtains first for recording corresponding timestamp according to the pieces of data in the Hbase table
Unit time set and the second unit time set, the first unit time set include at the first time between stamp and the second timestamp
Each unit time in the first data record number for generating, second unit time set includes the second timestamp and the
The the first data record number generated in each unit time between three timestamps, stamp is in the Hbase table at the first time
Pieces of data record from current earliest timestamp in corresponding timestamp, third timestamp is each in the Hbase table
Data is recorded from current nearest timestamp in corresponding timestamp, the second timestamp be located at first time stamp with it is described
Between third timestamp;
Second acquisition unit, for obtaining the first parameter value at least one RNN parameter, according to it is described at least one
The RNN parameter of the first RNN is arranged in first parameter value of RNN parameter, obtains the 2nd RNN;
Generation unit, for according to first unit time set, second unit time set and described second
RNN generates the prediction model.
Optionally, in conjunction with the 4th kind of optional implementation of the embodiment of the present application, the 5th kind in the embodiment of the present application can
It selects in implementation, the generation unit is used for:
According to the first data record number of each unit time in the first unit time set, pass through described second
RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction
The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre-
Survey model.
Optionally, in conjunction with the 5th kind of optional implementation of the embodiment of the present application, the 6th kind in the embodiment of the present application can
It selects in implementation, the generation unit is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter
The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third
RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre-
Survey model.
Optionally, in conjunction with any optional realization side in first to the 6th kind of optional implementation of the embodiment of the present application
Formula, in the 7th kind of optional implementation of the embodiment of the present application, described device 400 further include:
Determining module on the date on the same day where acquisition time for determining the acquisition target Region, is worked as described
One time point of selection and the time point is determined as the corresponding splitting time of the target Region in the preceding date, described point
Split the preset time point that the time is later than in the date on the same day.
Optionally, in conjunction with any optional realization side in first to the 7th kind of optional implementation of the embodiment of the present application
Formula, in the 8th kind of optional implementation of the embodiment of the present application, described device 400 further include:
Memory module, for the data packet in the spatial cache of message system to be stored in Hbase's according to configuration file
In Region;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related
Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field
Race.
Optionally, in conjunction with the 8th kind of optional implementation of the embodiment of the present application, the 9th kind in the embodiment of the present application can
It selects in implementation, the memory module includes:
Third acquiring unit, for the corresponding subject identification of spatial cache according to the message system, from configuration file
It is middle to obtain the subject correlation message including the subject identification, it further include the mark and object of Hbase table in the subject information
The set identification of set;
4th acquiring unit, for obtaining the set identification from the data packet in the spatial cache of the message system
The corresponding field contents of each field name in corresponding object set;
Storage unit, for each field contents of the acquisition to be formed data record, according to the object set
In each field affiliated column family the data record is stored in the Hbase table the corresponding Hbase table of mark Region
In.
In the embodiment of the present application, the first number that Region in the target time period is predicted by prediction module, according to
First number it can be concluded that the Region for needing to divide in the target time period third number.So in the target time period when
When detecting that filling rate is more than the target Region of default filling rate threshold value, if the target Region number obtained is less than or waits
In third number, target Region can be divided in the splitting time of the more idle free time section of offer business by dividing module
It splits, can reduce affected number of clients, while the target Region number also divided by the control of third number,
It avoids dividing excessive Region, and the storage resource of a large amount of Region is caused to waste.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method
Embodiment in be described in detail, no detailed explanation will be given here.
Fig. 5 shows the structural block diagram of the terminal 500 of an illustrative embodiment of the invention offer, which can be with
For total node in distributed memory system.The terminal 500 can be portable mobile termianl, such as: smart phone, plate electricity
(Moving Picture Experts Group Audio Layer III, dynamic image expert compress mark for brain, MP3 player
Quasi- audio level 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression
Standard audio level 4) player, laptop or desktop computer.Terminal 500 is also possible to referred to as user equipment, portable
Other titles such as terminal, laptop terminal, terminal console.
In general, terminal 500 includes: processor 501 and memory 502.
Processor 501 may include one or more processing cores, such as 4 core processors, 8 core processors etc..Place
Reason device 501 can use DSP (Digital Signal Processing, Digital Signal Processing), FPGA (Field-
Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, may be programmed
Logic array) at least one of example, in hardware realize.Processor 501 also may include primary processor and coprocessor, master
Processor is the processor for being handled data in the awake state, also referred to as CPU (Central Processing
Unit, central processing unit);Coprocessor is the low power processor for being handled data in the standby state.In
In some embodiments, processor 501 can be integrated with GPU (Graphics Processing Unit, image processor),
GPU is used to be responsible for the rendering and drafting of content to be shown needed for display screen.In some embodiments, processor 501 can also be wrapped
AI (Artificial Intelligence, artificial intelligence) processor is included, the AI processor is for handling related machine learning
Calculating operation.
Memory 502 may include one or more computer readable storage mediums, which can
To be non-transient.Memory 502 may also include high-speed random access memory and nonvolatile memory, such as one
Or multiple disk storage equipments, flash memory device.In some embodiments, the non-transient computer in memory 502 can
Storage medium is read for storing at least one instruction, at least one instruction performed by processor 501 for realizing this Shen
Please in embodiment of the method provide a kind of division Region method.
In some embodiments, terminal 500 is also optional includes: peripheral device interface 503 and at least one peripheral equipment.
It can be connected by bus or signal wire between processor 501, memory 502 and peripheral device interface 503.Each peripheral equipment
It can be connected by bus, signal wire or circuit board with peripheral device interface 503.Specifically, peripheral equipment includes: radio circuit
504, at least one of touch display screen 505, camera 506, voicefrequency circuit 507, positioning component 508 and power supply 509.
Peripheral device interface 503 can be used for I/O (Input/Output, input/output) is relevant outside at least one
Peripheral equipment is connected to processor 501 and memory 502.In some embodiments, processor 501, memory 502 and peripheral equipment
Interface 503 is integrated on same chip or circuit board;In some other embodiments, processor 501, memory 502 and outer
Any one or two in peripheral equipment interface 503 can realize on individual chip or circuit board, the present embodiment to this not
It is limited.
Radio circuit 504 is for receiving and emitting RF (Radio Frequency, radio frequency) signal, also referred to as electromagnetic signal.It penetrates
Frequency circuit 504 is communicated by electromagnetic signal with communication network and other communication equipments.Radio circuit 504 turns electric signal
It is changed to electromagnetic signal to be sent, alternatively, the electromagnetic signal received is converted to electric signal.Optionally, radio circuit 504 wraps
It includes: antenna system, RF transceiver, one or more amplifiers, tuner, oscillator, digital signal processor, codec chip
Group, user identity module card etc..Radio circuit 504 can be carried out by least one wireless communication protocol with other terminals
Communication.The wireless communication protocol includes but is not limited to: WWW, Metropolitan Area Network (MAN), Intranet, each third generation mobile communication network (2G, 3G,
4G and 5G), WLAN and/or WiFi (Wireless Fidelity, Wireless Fidelity) network.In some embodiments, it penetrates
Frequency circuit 504 can also include NFC (Near Field Communication, wireless near field communication) related circuit, this
Application is not limited this.
Display screen 505 is for showing UI (User Interface, user interface).The UI may include figure, text, figure
Mark, video and its their any combination.When display screen 505 is touch display screen, display screen 505 also there is acquisition to show
The ability of the touch signal on the surface or surface of screen 505.The touch signal can be used as control signal and be input to processor
501 are handled.At this point, display screen 505 can be also used for providing virtual push button and/or dummy keyboard, also referred to as soft button and/or
Soft keyboard.In some embodiments, display screen 505 can be one, and the front panel of terminal 500 is arranged;In other embodiments
In, display screen 505 can be at least two, be separately positioned on the different surfaces of terminal 500 or in foldover design;In still other reality
It applies in example, display screen 505 can be flexible display screen, be arranged on the curved surface of terminal 500 or on fold plane.Even, it shows
Display screen 505 can also be arranged to non-rectangle irregular figure, namely abnormity screen.Display screen 505 can use LCD (Liquid
Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, Organic Light Emitting Diode)
Etc. materials preparation.
CCD camera assembly 506 is for acquiring image or video.Optionally, CCD camera assembly 506 include front camera and
Rear camera.In general, the front panel of terminal is arranged in front camera, the back side of terminal is arranged in rear camera.One
In a little embodiments, rear camera at least two is main camera, depth of field camera, wide-angle camera, focal length camera shooting respectively
Any one in head, to realize that main camera and the fusion of depth of field camera realize background blurring function, main camera and wide-angle
Camera fusion realizes that pan-shot and VR (Virtual Reality, virtual reality) shooting function or other fusions are clapped
Camera shooting function.In some embodiments, CCD camera assembly 506 can also include flash lamp.Flash lamp can be monochromatic warm flash lamp,
It is also possible to double-colored temperature flash lamp.Double-colored temperature flash lamp refers to the combination of warm light flash lamp and cold light flash lamp, can be used for not
With the light compensation under colour temperature.
Voicefrequency circuit 507 may include microphone and loudspeaker.Microphone is used to acquire the sound wave of user and environment, and will
Sound wave, which is converted to electric signal and is input to processor 501, to be handled, or is input to radio circuit 504 to realize voice communication.
For stereo acquisition or the purpose of noise reduction, microphone can be separately positioned on the different parts of terminal 500 to be multiple.Mike
Wind can also be array microphone or omnidirectional's acquisition type microphone.Loudspeaker is then used to that processor 501 or radio circuit will to be come from
504 electric signal is converted to sound wave.Loudspeaker can be traditional wafer speaker, be also possible to piezoelectric ceramic loudspeaker.When
When loudspeaker is piezoelectric ceramic loudspeaker, the audible sound wave of the mankind can be not only converted electrical signals to, it can also be by telecommunications
Number the sound wave that the mankind do not hear is converted to carry out the purposes such as ranging.In some embodiments, voicefrequency circuit 507 can also include
Earphone jack.
Positioning component 508 is used for the current geographic position of positioning terminal 500, to realize navigation or LBS (Location
Based Service, location based service).Positioning component 508 can be the GPS (Global based on the U.S.
Positioning System, global positioning system), China dipper system or Russia Galileo system positioning group
Part.
Power supply 509 is used to be powered for the various components in terminal 500.Power supply 509 can be alternating current, direct current,
Disposable battery or rechargeable battery.When power supply 509 includes rechargeable battery, which can be wired charging electricity
Pond or wireless charging battery.Wired charging battery is the battery to be charged by Wireline, and wireless charging battery is by wireless
The battery of coil charges.The rechargeable battery can be also used for supporting fast charge technology.
In some embodiments, terminal 500 further includes having one or more sensors 510.The one or more sensors
510 include but is not limited to: acceleration transducer 511, gyro sensor 512, pressure sensor 513, fingerprint sensor 514,
Optical sensor 515 and proximity sensor 516.
The acceleration that acceleration transducer 511 can detecte in three reference axis of the coordinate system established with terminal 500 is big
It is small.For example, acceleration transducer 511 can be used for detecting component of the acceleration of gravity in three reference axis.Processor 501 can
With the acceleration of gravity signal acquired according to acceleration transducer 511, touch display screen 505 is controlled with transverse views or longitudinal view
Figure carries out the display of user interface.Acceleration transducer 511 can be also used for the acquisition of game or the exercise data of user.
Gyro sensor 512 can detecte body direction and the rotational angle of terminal 500, and gyro sensor 512 can
To cooperate with acquisition user to act the 3D of terminal 500 with acceleration transducer 511.Processor 501 is according to gyro sensor 512
Following function may be implemented in the data of acquisition: when action induction (for example changing UI according to the tilt operation of user), shooting
Image stabilization, game control and inertial navigation.
The lower layer of side frame and/or touch display screen 505 in terminal 500 can be set in pressure sensor 513.Work as pressure
When the side frame of terminal 500 is arranged in sensor 513, user can detecte to the gripping signal of terminal 500, by processor 501
Right-hand man's identification or prompt operation are carried out according to the gripping signal that pressure sensor 513 acquires.When the setting of pressure sensor 513 exists
When the lower layer of touch display screen 505, the pressure operation of touch display screen 505 is realized to UI circle according to user by processor 501
Operability control on face is controlled.Operability control includes button control, scroll bar control, icon control, menu
At least one of control.
Fingerprint sensor 514 is used to acquire the fingerprint of user, collected according to fingerprint sensor 514 by processor 501
The identity of fingerprint recognition user, alternatively, by fingerprint sensor 514 according to the identity of collected fingerprint recognition user.It is identifying
When the identity of user is trusted identity out, the user is authorized to execute relevant sensitive operation, the sensitive operation packet by processor 501
Include solution lock screen, check encryption information, downloading software, payment and change setting etc..Terminal can be set in fingerprint sensor 514
500 front, the back side or side.When being provided with physical button or manufacturer Logo in terminal 500, fingerprint sensor 514 can be with
It is integrated with physical button or manufacturer Logo.
Optical sensor 515 is for acquiring ambient light intensity.In one embodiment, processor 501 can be according to optics
The ambient light intensity that sensor 515 acquires controls the display brightness of touch display screen 505.Specifically, when ambient light intensity is higher
When, the display brightness of touch display screen 505 is turned up;When ambient light intensity is lower, the display for turning down touch display screen 505 is bright
Degree.In another embodiment, the ambient light intensity that processor 501 can also be acquired according to optical sensor 515, dynamic adjust
The acquisition parameters of CCD camera assembly 506.
Proximity sensor 516, also referred to as range sensor are generally arranged at the front panel of terminal 500.Proximity sensor 516
For acquiring the distance between the front of user Yu terminal 500.In one embodiment, when proximity sensor 516 detects use
When family and the distance between the front of terminal 500 gradually become smaller, touch display screen 505 is controlled from bright screen state by processor 501
It is switched to breath screen state;When proximity sensor 516 detects user and the distance between the front of terminal 500 becomes larger,
Touch display screen 505 is controlled by processor 501 and is switched to bright screen state from breath screen state.
It will be understood by those skilled in the art that the restriction of the not structure paired terminal 500 of structure shown in Fig. 5, can wrap
It includes than illustrating more or fewer components, perhaps combine certain components or is arranged using different components.
Those skilled in the art will readily occur to its of the application after considering specification and practicing application disclosed herein
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the application, these modifications, purposes or
Person's adaptive change follows the general principle of the application and including the undocumented common knowledge in the art of the application
Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the application are by following
Claim is pointed out.
It should be understood that the application is not limited to the precise structure that has been described above and shown in the drawings, and
And various modifications and changes may be made without departing from the scope thereof.Scope of the present application is only limited by the accompanying claims.
Claims (18)
1. a method of division subregion, which is characterized in that the described method includes:
Stored data record in Hbase table, prediction need in the target time section before being started according to target time section
Occupy the first number of the subregion in the Hbase table;
When first number is greater than the second number, it is more than default storage that filling rate is obtained from the second number subregion
For the subregion of rate threshold value as target partition, the second number subregion is pre-assigned to need in the target time section
Occupy the subregion in the Hbase table;
When the number of the target partition of acquisition is less than or equal to third number, in the corresponding splitting time point of the target partition
The target partition is split, the splitting time is later than the acquisition time for obtaining the target partition, and the third number is equal to institute
It states the first number and subtracts second number.
2. the method as described in claim 1, which is characterized in that described to start to have deposited in preceding Hbase table according to target time section
The data record of storage, prediction need to occupy in the target time section the first number of the subregion in the Hbase table, packet
It includes:
According to used spatial content in the stored data record total amount of Hbase table and the Hbase table, described in calculating
The average amount of every data record in Hbase table;
Pass through the number for the data to be stored record that prediction model prediction generates in the target time section;
According to number, the average amount and subregion capacity that the data to be stored records, calculate in the object time
Need to occupy the first number of the subregion in the Hbase table in section.
3. method according to claim 2, which is characterized in that the prediction needs to occupy in the target time section described
Before first number of the subregion in Hbase table, further includes:
The data record stored in the Hbase table before being started according to the target time section, generates the prediction model.
4. method as claimed in claim 3, which is characterized in that described to start the preceding Hbase according to the target time section
The data record stored in table generates the prediction model, comprising:
Corresponding timestamp is recorded according to the pieces of data in the Hbase table, obtains the first unit time set and the second list
Position time set, the first unit time set in each unit time at the first time between stamp and the second timestamp including generating
The first data record number, second unit time set includes each list between the second timestamp and third timestamp
The the first data record number generated in the time of position, it is corresponding to be that pieces of data in the Hbase table records for stamp at the first time
From current earliest timestamp in timestamp, third timestamp is that the pieces of data in the Hbase table records the corresponding time
From current nearest timestamp in stamp, the second timestamp is located between first time stamp and the third timestamp;
The first parameter value at least one Recognition with Recurrent Neural Network RNN parameter is obtained, according to the of at least one RNN parameter
The RNN parameter of the first RNN is arranged in one parameter value, obtains the 2nd RNN;
The prediction mould is generated according to first unit time set, second unit time set and the 2nd RNN
Type.
5. method as claimed in claim 4, which is characterized in that described according to first unit time set, described second
Unit time set and the 2nd RNN generate the prediction model, comprising:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set
At the first model;
Generated in the unit time between the second timestamp and third timestamp by first model prediction
Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as to predict mould
Type.
6. method as claimed in claim 5, which is characterized in that the method also includes:
When the second unit time set and third unit time set are unsatisfactory for preset condition, RNN parameter corresponding the is obtained
The RNN parameter of the 2nd RNN is arranged according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN for two parameter values;
The prediction mould is generated according to first unit time set, second unit time set and the 3rd RNN
Type.
7. the method as described in claim 1, which is characterized in that the filling rate that obtains from the second number subregion surpasses
After the subregion for presetting filling rate threshold value is crossed as target partition, further includes:
It determines the date on the same day where the acquisition time for obtaining the target partition, a time is selected in the current date
The time point is simultaneously determined as the corresponding splitting time of the target partition by point, and the splitting time is later than the date on the same day
Interior preset time point.
8. method as described in any one of claim 1 to 7, which is characterized in that the method also includes:
The data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, subject correlation message
Including at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least field name and the affiliated column family of field.
9. method according to claim 8, which is characterized in that it is described will be in the spatial cache of message system according to configuration file
Data packet be stored in the subregion of Hbase, comprising:
According to the corresponding subject identification of the spatial cache of the message system, obtaining from configuration file includes the subject identification
Subject correlation message, further include the mark of Hbase table and the set identification of object set in the subject information;
It is every in the corresponding object set of the set identification from being obtained in the data packet in the spatial cache of the message system
The corresponding field contents of a field name;
Each field contents of the acquisition are formed into data record, according to the affiliated column family of each field in the object set
The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table.
10. a kind of device for dividing subregion, which is characterized in that described device includes:
Prediction module is predicted for stored data record in Hbase table before being started according to target time section in the target
Need to occupy the first number of the subregion in the Hbase table in period;
Module is obtained, for obtaining storage from the second number subregion when first number is greater than the second number
Rate is more than the subregion of default filling rate threshold value as target partition, and the second number subregion is pre-assigned in the mesh
Need to occupy the subregion in the Hbase table in the mark period;
Divide module, for when the number of the target partition of acquisition be less than or equal to third number when, in the target partition pair
The splitting time answered divides the target partition, and the splitting time is later than the acquisition time for obtaining the target partition, described
Third number is equal to first number and subtracts second number.
11. device as claimed in claim 10, which is characterized in that the prediction module includes:
First computing unit, for according to used sky in the stored data record total amount of Hbase table and the Hbase table
Between capacity, calculate in the Hbase table every data record average amount;
Predicting unit, the number of the data to be stored record for being generated in the target time section by prediction model prediction
Mesh;
Second computing unit, number, the average amount and subregion capacity for being recorded according to the data to be stored, meter
Calculate the first number for needing to occupy the subregion in the Hbase table in the target time section.
12. device as claimed in claim 11, which is characterized in that described device further include:
Generation module, the data record for being stored in the Hbase table before being started according to the target time section, described in generation
Prediction model.
13. device as claimed in claim 12, which is characterized in that the generation module includes:
First acquisition unit obtains the first unit for recording corresponding timestamp according to the pieces of data in the Hbase table
Time set and the second unit time set, the first unit time set include every between stamp and the second timestamp at the first time
The the first data record number generated in a unit time, when the second unit time set includes the second timestamp and third
Between stamp between each unit time in generate the first data record number, at the first time stamp be each in the Hbase table
Data is recorded from current earliest timestamp in corresponding timestamp, and third timestamp is each item number in the Hbase table
According to recording from current nearest timestamp in corresponding timestamp, the second timestamp is located at first time stamp and the third
Between timestamp;
Second acquisition unit, for obtaining the first parameter value at least one Recognition with Recurrent Neural Network RNN parameter, according to it is described extremely
The RNN parameter of the first RNN is arranged in first parameter value of a few RNN parameter, obtains the 2nd RNN;
Generation unit, for raw according to first unit time set, second unit time set and the 2nd RNN
At the prediction model.
14. device as claimed in claim 13, which is characterized in that the generation unit is used for:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set
At the first model;
Generated in the unit time between the second timestamp and third timestamp by first model prediction
Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as to predict mould
Type.
15. device as claimed in claim 14, which is characterized in that the generation unit is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, RNN parameter corresponding the is obtained
The RNN parameter of the 2nd RNN is arranged according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN for two parameter values;
The prediction mould is generated according to first unit time set, second unit time set and the 3rd RNN
Type.
16. device as claimed in claim 10, which is characterized in that described device further include:
Determining module, the date on the same day where acquisition time for determining the acquisition target partition, in the current date
The time point is simultaneously determined as the corresponding splitting time of the target partition, the splitting time evening by one time point of interior selection
In the preset time point in the date on the same day.
17. such as the described in any item devices of claim 10 to 16, which is characterized in that described device further include:
Memory module, for the data packet in the spatial cache of message system to be stored in the subregion of Hbase according to configuration file
In;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, subject correlation message
Including at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least field name and the affiliated column family of field.
18. device as claimed in claim 17, which is characterized in that the memory module includes:
Third acquiring unit is obtained from configuration file for the corresponding subject identification of spatial cache according to the message system
The subject correlation message including the subject identification is taken, further includes the mark and object set of Hbase table in the subject information
Set identification;
4th acquiring unit, it is corresponding for obtaining the set identification from the data packet in the spatial cache of the message system
Object set in the corresponding field contents of each field name;
Storage unit, for each field contents of the acquisition to be formed data record, according in the object set
The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table by each affiliated column family of field.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810494401.5A CN110519319B (en) | 2018-05-22 | 2018-05-22 | Method and device for splitting partitions |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201810494401.5A CN110519319B (en) | 2018-05-22 | 2018-05-22 | Method and device for splitting partitions |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110519319A true CN110519319A (en) | 2019-11-29 |
CN110519319B CN110519319B (en) | 2022-02-11 |
Family
ID=68621791
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201810494401.5A Active CN110519319B (en) | 2018-05-22 | 2018-05-22 | Method and device for splitting partitions |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN110519319B (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110888846A (en) * | 2019-12-10 | 2020-03-17 | 北京北龙云海网络数据科技有限责任公司 | Data memory management method and system |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6023706A (en) * | 1997-07-11 | 2000-02-08 | International Business Machines Corporation | Parallel file system and method for multiple node file access |
US20160241890A1 (en) * | 2013-08-20 | 2016-08-18 | Lg Electronics Inc. | Apparatus for transmitting media data via streaming service, apparatus for receiving media data via streaming service, method for transmitting media data via streaming service and method for receiving media data via streaming service |
CN105988995A (en) * | 2015-01-27 | 2016-10-05 | 杭州海康威视数字技术股份有限公司 | HFile based data batch loading method |
CN107169009A (en) * | 2017-03-31 | 2017-09-15 | 北京奇艺世纪科技有限公司 | A kind of data splitting method and device of distributed memory system |
CN107480205A (en) * | 2017-07-24 | 2017-12-15 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for carrying out data partition |
CN107943412A (en) * | 2016-10-12 | 2018-04-20 | 阿里巴巴集团控股有限公司 | A kind of subregion division, the method, apparatus and system for deleting data file in subregion |
-
2018
- 2018-05-22 CN CN201810494401.5A patent/CN110519319B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6023706A (en) * | 1997-07-11 | 2000-02-08 | International Business Machines Corporation | Parallel file system and method for multiple node file access |
US20160241890A1 (en) * | 2013-08-20 | 2016-08-18 | Lg Electronics Inc. | Apparatus for transmitting media data via streaming service, apparatus for receiving media data via streaming service, method for transmitting media data via streaming service and method for receiving media data via streaming service |
CN105988995A (en) * | 2015-01-27 | 2016-10-05 | 杭州海康威视数字技术股份有限公司 | HFile based data batch loading method |
CN107943412A (en) * | 2016-10-12 | 2018-04-20 | 阿里巴巴集团控股有限公司 | A kind of subregion division, the method, apparatus and system for deleting data file in subregion |
CN107169009A (en) * | 2017-03-31 | 2017-09-15 | 北京奇艺世纪科技有限公司 | A kind of data splitting method and device of distributed memory system |
CN107480205A (en) * | 2017-07-24 | 2017-12-15 | 北京京东尚科信息技术有限公司 | A kind of method and apparatus for carrying out data partition |
Non-Patent Citations (2)
Title |
---|
XIAOFENG LEI: ""BESIII Physics Data Storing and Processing on HBase and MapReduce"", 《21ST INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS》 * |
王卓: ""基于增量式分区策略的MapReduce数据均衡方法"", 《计算机学报》 * |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110888846A (en) * | 2019-12-10 | 2020-03-17 | 北京北龙云海网络数据科技有限责任公司 | Data memory management method and system |
Also Published As
Publication number | Publication date |
---|---|
CN110519319B (en) | 2022-02-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110110263B (en) | Webpage display method, device, terminal and storage medium | |
CN110290421A (en) | Frame per second method of adjustment, device, computer equipment and storage medium | |
CN109600678A (en) | Information displaying method, apparatus and system, server, terminal, storage medium | |
CN110213616A (en) | Video providing method, acquisition methods, device and equipment | |
CN109656930A (en) | Data query method, apparatus and system | |
CN109816042B (en) | Data classification model training method and device, electronic equipment and storage medium | |
CN109618212A (en) | Information display method, device, terminal and storage medium | |
CN110019929A (en) | Processing method, device and the computer readable storage medium of web page contents | |
CN110166786A (en) | Virtual objects transfer method and device | |
CN109302385A (en) | Multimedia resource sharing method, device and storage medium | |
CN108449651A (en) | Subtitle adding method and device | |
CN110278464A (en) | The method and apparatus for showing list | |
CN110147503B (en) | Information issuing method and device, computer equipment and storage medium | |
CN109922356A (en) | Video recommendation method, device and computer readable storage medium | |
CN110380904A (en) | Bandwidth allocation methods, device, electronic equipment and storage medium | |
CN108900925A (en) | The method and apparatus of live streaming template are set | |
CN110418152A (en) | It is broadcast live the method and device of prompt | |
CN111031391A (en) | Video dubbing method, device, server, terminal and storage medium | |
CN109218751A (en) | The method, apparatus and system of recommendation of audio | |
CN110535890A (en) | The method and apparatus that file uploads | |
CN113032587A (en) | Multimedia information recommendation method, system, device, terminal and server | |
CN110045958A (en) | Data texturing generation method, device, storage medium and equipment | |
CN109889858A (en) | Information processing method, device and the computer readable storage medium of virtual objects | |
CN109600301A (en) | Message treatment method and device | |
CN110471614A (en) | A kind of method of storing data, the method and device for detecting terminal |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |