CN110519319A - A kind of method and device dividing subregion - Google Patents

A kind of method and device dividing subregion Download PDF

Info

Publication number
CN110519319A
CN110519319A CN201810494401.5A CN201810494401A CN110519319A CN 110519319 A CN110519319 A CN 110519319A CN 201810494401 A CN201810494401 A CN 201810494401A CN 110519319 A CN110519319 A CN 110519319A
Authority
CN
China
Prior art keywords
unit time
time
subregion
rnn
hbase table
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201810494401.5A
Other languages
Chinese (zh)
Other versions
CN110519319B (en
Inventor
王玉华
王鹏宇
董明
李林森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Hikvision Digital Technology Co Ltd
Original Assignee
Hangzhou Hikvision Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Hikvision Digital Technology Co Ltd filed Critical Hangzhou Hikvision Digital Technology Co Ltd
Priority to CN201810494401.5A priority Critical patent/CN110519319B/en
Publication of CN110519319A publication Critical patent/CN110519319A/en
Application granted granted Critical
Publication of CN110519319B publication Critical patent/CN110519319B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/10Protocols in which an application is distributed across nodes in the network
    • H04L67/1097Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application is to belong to the communications field about a kind of method and device for dividing subregion.The described method includes: stored data record, prediction need to occupy in the target time section the first number of the subregion in the Hbase table in Hbase table before being started according to target time section;When first number is greater than the second number, it is more than to preset the subregion of filling rate threshold value as target partition that filling rate is obtained from the second number subregion, and the second number subregion is the pre-assigned subregion for needing to occupy in the target time section in the Hbase table;When the number of the target partition of acquisition is less than or equal to third number, the target partition is divided in the corresponding splitting time of the target partition, the splitting time is later than the acquisition time for obtaining the target partition, and the third number is equal to first number and subtracts second number.The application can reduce impacted number of clients.

Description

A kind of method and device dividing subregion
Technical field
This application involves the communications field, in particular to a kind of method and device for dividing subregion.
Background technique
Hbase (Hadoop Database) is a kind of high reliability, high-performance, towards column, telescopic distributed storage System.Only one Region in Hbase table when default one Hbase table of creation, Region is the subregion in Hbase table, is used In storing data.
The capacity of Region is fixed, it is assumed that the capacity of Region is N, when the data for needing to store are more and more, When leading to the insufficient space of Region, need to split into the Region Region that two capacity are N, to store more numbers According to.Hbase has the function of dividing Region automatically at present, can be automatically by the Region points when Region is filled with data Two new Region are cleaved into, the data in the Region are possibly stored in this two new Region.
During realizing the application, discovery aforesaid way at least has following defects that inventor
Client can not access the Region during Region is split into two new Region.Industry is being provided It is engaged in the more busy busy period, can store or access a large amount of data into Region, Region is caused to be easy busy Be filled in period it is concurrent it is estranged split, and have a large amount of client access Region within the busy period, request real-time industry Business causes to provide real time business without a large amount of client of normal direction within the busy period in this way, generates shadow to a large amount of client It rings.
Summary of the invention
In order to reduce the number of clients by shadow, the embodiment of the present application provides a kind of method and device for dividing subregion. The technical solution is as follows:
In a first aspect, the application provides a kind of method for dividing subregion, which comprises
Stored data record in Hbase table, is predicted in the target time section before being started according to target time section Need to occupy the first number of the subregion in the Hbase table;
When first number is greater than the second number, it is more than default that filling rate is obtained from the second number subregion The subregion of filling rate threshold value is pre-assigned in the target time section as target partition, the second number subregion Need to occupy the subregion in the Hbase table;
When the number of the target partition of acquisition is less than or equal to third number, in the corresponding division of the target partition Between divide the target partition, the splitting time is later than the acquisition time for obtaining the target partition, the third number etc. Second number is subtracted in first number.
Optionally, described that stored data record in preceding Hbase table is started according to target time section, it predicts in the mesh Need to occupy the first number of the subregion in the Hbase table in the mark period, comprising:
According to used spatial content in the stored data record total amount of Hbase table and the Hbase table, institute is calculated State the average amount of every data record in Hbase table;
Pass through the number for the data to be stored record that prediction model prediction generates in the target time section;
According to number, the average amount and subregion capacity that the data to be stored records, calculate in the target Need to occupy the first number of the subregion in the Hbase table in period.
Optionally, the prediction needs to occupy in the target time section the first number of the subregion in the Hbase table Before mesh, further includes:
The data record stored in the Hbase table before being started according to the target time section, generates the prediction model.
Optionally, the data record for being started to store in the preceding Hbase table according to the target time section, generates institute State prediction model, comprising:
Corresponding timestamp is recorded according to the pieces of data in the Hbase table, obtains the first unit time set and the Two unit time set, the first unit time set include in each unit time stabbed between the second timestamp at the first time The the first data record number generated, the second unit time set includes every between the second timestamp and third timestamp The the first data record number generated in a unit time, stamp is the pieces of data record pair in the Hbase table at the first time From current earliest timestamp in the timestamp answered, it is corresponding that third timestamp is that the pieces of data in the Hbase table records From current nearest timestamp in timestamp, the second timestamp is located between first time stamp and the third timestamp;
The first parameter value at least one Recognition with Recurrent Neural Network RNN parameter is obtained, according at least one described RNN parameter The first parameter value be arranged the first RNN RNN parameter, obtain the 2nd RNN;
It is generated according to first unit time set, second unit time set and the 2nd RNN described pre- Survey model.
Optionally, described according to first unit time set, second unit time set and the 2nd RNN Generate the prediction model, comprising:
According to the first data record number of each unit time in the first unit time set, pass through described second RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre- Survey model.
Optionally, the method also includes:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre- Survey model.
Optionally, described that the subregion work that filling rate is more than default filling rate threshold value is obtained from the second number subregion After target partition, further includes:
It determines the date on the same day where the acquisition time for obtaining the target partition, one is selected in the current date The time point is simultaneously determined as the corresponding splitting time of the target partition by time point, and the splitting time is later than the same day Preset time point in date.
Optionally, the method also includes:
The data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field Race.
Optionally, point that the data packet in the spatial cache of message system is stored in Hbase according to configuration file Qu Zhong, comprising:
According to the corresponding subject identification of the spatial cache of the message system, obtaining from configuration file includes the theme The subject correlation message of mark further includes the mark of Hbase table and the set identification of object set in the subject information;
From being obtained in the data packet in the spatial cache of the message system in the corresponding object set of the set identification The corresponding field contents of each field name;
Each field contents of the acquisition are formed into data record, according to belonging to each field in the object set The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table by column family.
Second aspect, this application provides a kind of device for dividing subregion, described device includes:
Prediction module is predicted for stored data record in Hbase table before being started according to target time section described Need to occupy the first number of the subregion in the Hbase table in target time section;
Module is obtained, for being obtained from the second number subregion when first number is greater than the second number Filling rate is more than the subregion of default filling rate threshold value as target partition, and the second number subregion is pre-assigned in institute State the subregion for needing to occupy in target time section in the Hbase table;
Module is divided, for dividing in the target when the number of the target partition of acquisition is less than or equal to third number The corresponding splitting time in area divides the target partition, and the splitting time is later than the acquisition time for obtaining the target partition, The third number is equal to first number and subtracts second number.
Optionally, the prediction module includes:
First computing unit, for having been used according in the stored data record total amount of Hbase table and the Hbase table Spatial content, calculate in the Hbase table every data record average amount;
Predicting unit, what the data to be stored for being generated in the target time section by prediction model prediction recorded Number;
Second computing unit, number, the average amount and subregion for being recorded according to the data to be stored are held Amount calculates the first number for needing to occupy the subregion in the Hbase table in the target time section.
Optionally, described device further include:
Generation module, the data record for storing in the Hbase table before being started according to the target time section, generates The prediction model.
Optionally, the generation module includes:
First acquisition unit obtains first for recording corresponding timestamp according to the pieces of data in the Hbase table Unit time set and the second unit time set, the first unit time set include at the first time between stamp and the second timestamp Each unit time in the first data record number for generating, second unit time set includes the second timestamp and the The the first data record number generated in each unit time between three timestamps, stamp is in the Hbase table at the first time Pieces of data record from current earliest timestamp in corresponding timestamp, third timestamp is each in the Hbase table Data is recorded from current nearest timestamp in corresponding timestamp, the second timestamp be located at first time stamp with it is described Between third timestamp;
Second acquisition unit, for obtaining the first parameter value at least one Recognition with Recurrent Neural Network RNN parameter, according to institute The RNN parameter of the first RNN is arranged in the first parameter value for stating at least one RNN parameter, obtains the 2nd RNN;
Generation unit, for according to first unit time set, second unit time set and described second RNN generates the prediction model.
Optionally, the generation unit, is used for:
According to the first data record number of each unit time in the first unit time set, pass through described second RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre- Survey model.
Optionally, the generation unit, is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre- Survey model.
Optionally, described device further include:
Determining module, the date on the same day where acquisition time for determining the acquisition target partition, described current One time point of selection and the time point is determined as the corresponding splitting time of the target partition in date, when the division Between be later than preset time point in the date on the same day.
Optionally, described device further include:
Memory module, for the data packet in the spatial cache of message system to be stored in Hbase's according to configuration file In subregion;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field Race.
Optionally, the memory module includes:
Third acquiring unit, for the corresponding subject identification of spatial cache according to the message system, from configuration file It is middle to obtain the subject correlation message including the subject identification, it further include the mark and object of Hbase table in the subject information The set identification of set;
4th acquiring unit, for obtaining the set identification from the data packet in the spatial cache of the message system The corresponding field contents of each field name in corresponding object set;
Storage unit, for each field contents of the acquisition to be formed data record, according to the object set In each field affiliated column family the data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table.
6th aspect, the application provides a kind of non-volatile computer readable storage medium storing program for executing, for storing computer program, The computer program is loaded and is executed by processor, to realize any optional method of first aspect or first aspect Instruction.
Technical solution provided by the embodiments of the present application can include the following benefits:
By predicting the first number of subregion in the target time period, according to the first number it can be concluded that in target time section The third number of the interior subregion for needing to divide.So in the target time period when detecting that filling rate is more than default filling rate threshold value Target partition when, can be in the more idle sky of the business of offer if the target partition number obtained is less than or equal to third number The splitting time of not busy period divides target partition, can reduce affected number of clients, while also logical The target partition number for crossing the control division of third number, avoids dividing excessive subregion, and causes a large amount of subregion Storage resource waste.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not The application can be limited.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows the implementation for meeting the application Example, and together with specification it is used to explain the principle of the application.
Fig. 1 is network architecture schematic diagram provided by the embodiments of the present application;
Fig. 2 is a kind of method flow diagram for dividing subregion provided by the embodiments of the present application;
Fig. 3-1 is a kind of method flow diagram for dividing subregion provided by the embodiments of the present application;
Fig. 3-2 is a kind of method flow diagram for generating prediction module provided by the embodiments of the present application;
Fig. 3-3 is the waveform diagram of the record count of unit time provided by the embodiments of the present application;
Fig. 4 is a kind of apparatus structure schematic diagram for dividing subregion provided by the embodiments of the present application;
Fig. 5 is terminal structure schematic diagram provided by the embodiments of the present application.
Through the above attached drawings, it has been shown that the specific embodiment of the application will be hereinafter described in more detail.These attached drawings It is not intended to limit the range of the application design in any manner with verbal description, but is by referring to specific embodiments Those skilled in the art illustrate the concept of the application.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related to When attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodiment Described in embodiment do not represent all embodiments consistent with the application.On the contrary, they be only with it is such as appended The example of the consistent device and method of some aspects be described in detail in claims, the application.
Referring to Fig. 1, the embodiment of the present application provides a kind of network architecture, comprising:
Data collection terminal 1, message system 2 and distributed memory system 3.It include that at least one caching is empty in message system 2 Between, the corresponding theme of each spatial cache at least one spatial cache.
Message system can be KAFKA, and the spatial cache in message system can be the message queue in KAFKA.Distribution Formula storage system includes total node and multiple back end, and total node is for managing multiple back end, multiple data section Point is for storing data.
For each spatial cache in message system, which belongs to this for buffered message system 2 to be received The data packet of the corresponding theme of spatial cache.
It include configuration file in distributed memory system, which includes at least one subject correlation message and at least One object set, subject correlation message include at least the collection of subject identification, the mark of Hbase table and object set (Schema) Close mark;It can also be including cluster belonging to message system cluster information belonging to rowkey create-rule, theme and Hbase table At least one of information etc., rowkey are the unique identifications for identifying data record.
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field Race;It can also include field type, whether the field is optional and whether the field indexes at least one of information etc..
Optionally, data collection terminal 1 can be used for acquiring data sequence, include in chronological sequence generating in the data sequence At least one data packet, in the packet header of each data packet include timestamp and the subject identification of theme that the data packet belongs to, Data collection terminal 1 is also used to each data packet of acquisition being sent to message system 2.
Optionally, data collection terminal 1 can also dress up each data envelope of acquisition before sending each data packet Then the data packet of PB (PetaByte, petabyte) protocol format sends the data packet after encapsulation to message system 2 again.
Since each data envelope of acquisition is dressed up the data packet of PB protocol format, PB protocol format by data collection terminal 1 Data packet data volume it is small, and the speed parsed is faster and simpler.
The data packet that collection terminal 1 is sent for receiving data of message system 2, according to the theme mark for including in the data packet Know, the corresponding theme of the subject identification is determined, by the data pack buffer in the corresponding spatial cache of the theme.
Optionally, it is cached in each spatial cache that distributed memory system can be included according to the configuration file Data packet is stored in distributed memory system, can be executed the storage by total node in distributed memory system and be operated, can With are as follows:
For each spatial cache, according to the corresponding subject identification of the spatial cache, obtaining from configuration file includes being somebody's turn to do The subject correlation message of subject identification further includes the mark and object set of Hbase table in the subject correlation message;From the caching The corresponding field contents of each field name in the object set are obtained in data packet in space;In each field that will acquire Hold composition one data record, which is stored in by the Hbase according to the affiliated column family of each field in the object set In the Region of the corresponding Hbase table of mark of table.
Referring to fig. 2, the embodiment of the present application provides a kind of method for dividing subregion, and the method can be applied to such as Fig. 1 The network architecture that illustrated embodiment provides, and the executing subject of this method can be total node in distributed memory system, packet It includes:
Step 201: stored data record in Hbase table before being started according to target time section was predicted in the object time Need to occupy the first number of the Region in Hbase table in section.
Region is subregion, can be the subregion in Hbase table, Region occurs in the other embodiments of the application Meaning may refer to herein, just no longer introduce one by one.
Optionally, the embodiment of the present application provides the first optional implementation to realize this step, the first is realized Mode includes following 2011 to 2013 operation, is respectively as follows:
2011: according to used spatial content in the stored data record total amount of Hbase table and Hbase table, calculating The average amount of every data record in Hbase table.
2012: passing through the number for the data to be stored record that prediction model prediction generates in the target time period.
2013: number, the average amount and the subregion capacity recorded according to data to be stored is calculated in target time section Inside need to occupy the first number of the subregion in Hbase table.
Optionally, the embodiment of the present application provides second of optional implementation, and second of optional implementation exists The operation of following steps 200 can also be performed before executing step 201.
Step 200: the data record stored in the Hbase table before being started according to target time section generates prediction model.
Optionally, the first optional implementation and second of optional implementation may be combined to form a kind of division The method of subregion.
Optionally, in conjunction with above-mentioned second optional implementation, it is optional real that the embodiment of the present application provides the third Existing mode, in the third optional implementation, above-mentioned steps 210 may include following 2001 to 2003 operation, respectively Are as follows:
2001: corresponding timestamp being recorded according to the pieces of data in the Hbase table, obtains the first unit time set Gather with the second unit time, the first unit time set includes when stabbing each unit between the second timestamp at the first time First data record number of interior generation, the second unit time set include every between the second timestamp and third timestamp The the first data record number generated in a unit time, it is corresponding to be that pieces of data in Hbase table records for stamp at the first time From current earliest timestamp in timestamp, third timestamp is that the pieces of data in Hbase table records in corresponding timestamp From current nearest timestamp, the second timestamp is located at the first time between stamp and third timestamp.
2002: obtaining at least one Recognition with Recurrent Neural Network (Recurrent Neural Networks, RNN) parameter The RNN parameter of the first RNN is arranged according to the first parameter value of at least one RNN parameter, obtains the 2nd RNN for first parameter value.
2003: prediction model is generated according to the first unit time set, the second unit time set and the 2nd RNN.
Optionally, the first above-mentioned optional implementation, second of optional implementation and the third optional reality Existing mode may be combined to form a kind of method for dividing subregion.
Optionally, in conjunction with the third above-mentioned optional implementation, the embodiment of the present application provides the 4th kind optionally in fact Existing mode, the operation of above-mentioned steps 2003 can be in the 4th kind of optional implementation are as follows:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set At the first model;
Generated in the unit time between the second timestamp and third timestamp by the first model prediction Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre- Survey model.
Optionally, the first above-mentioned optional implementation, second of optional implementation, the third optional realization Mode and the 4th kind of optional implementation may be combined to form a kind of method for dividing subregion.
Optionally, in conjunction in above-mentioned 4th kind optional implementation, the embodiment of the present application provides the 5th kind optionally Implementation, in the 5th kind of optional implementation, the method for dividing subregion can also include following operation:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN;Root Prediction model is generated according to the first unit time set, the second unit time set and the 3rd RNN.
Optionally, the first above-mentioned optional implementation, second of optional implementation, the third optional realization Mode, the 4th kind of optional implementation and the 5th kind of optional implementation may be combined to form a kind of side for dividing subregion Method.
Step 202: when the first number is greater than the second number, it is more than pre- that filling rate is obtained from the second number Region If the Region of filling rate threshold value as target Region, the second number Region be it is pre-assigned in the target time period Need to occupy the Region in Hbase table.
Optionally, the embodiment of the present application provides the 6th kind of optional implementation, in the 6th kind of optional implementation In after executing the step 202 operation, the method for dividing subregion can also include following operation:
It determines the date on the same day where the acquisition time for obtaining target partition, selects a time point simultaneously in current date The time point is determined as the corresponding splitting time of target partition, which is later than the preset time point in the date on the same day.
Optionally, the 6th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional Implementation, the third optional implementation, in the 4th kind of optional implementation and the 5th kind of optional implementation Any implementation combine formed division subregion method.
Step 203: corresponding in target Region when the number of the target Region of acquisition is less than or equal to third number Splitting time divide target Region, which is later than the acquisition time for obtaining target Region, and third number is equal to First number subtracts the second number.
Wherein, presetting filling rate threshold value is numerical value less than 1, such as can be the numerical value such as 0.8,0.7 or 0.9, when division Between can be daily offer business it is more idle free time section in sometime.
Optionally, the embodiment of the present application provides the 7th kind of optional implementation, in the 7th kind of optional implementation In, the method for dividing subregion can also include following 204 operation:
Step 204: the data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file In;Wherein, which includes at least one subject correlation message and at least one object set, and subject correlation message is at least Set identification including subject identification, the mark of Hbase table and object set;Object set includes at least one field domain letter Breath, field domain information include at least field name and the affiliated column family of field.
Optionally, the 7th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional Implementation, the third optional implementation, in the 4th kind of optional implementation and the 6th kind of optional implementation Any implementation combine formed division subregion method.
Optionally, the embodiment of the present application provides the 8th kind of optional implementation, in the 8th kind of optional implementation In above-mentioned 204 operation can be with are as follows:
According to the corresponding subject identification of the spatial cache of message system, obtaining from configuration file includes the subject identification Subject correlation message further includes the mark of Hbase table and the set identification of object set in the subject information.
Each of corresponding object set of the set identification is obtained from the data packet in the spatial cache of message system The corresponding field contents of field name.
Each field contents that will acquire form data record, will according to the affiliated column family of each field in the object set The data record is stored in the subregion of the corresponding Hbase table of mark of Hbase table.
Optionally, the 8th kind of optional implementation, can with the first above-mentioned optional implementation, second it is optional Implementation, the third optional implementation, in the 4th kind of optional implementation and the 6th kind of optional implementation Any implementation combine formed division subregion method.
In the embodiment of the present application, by predicting the first number of Region in the target time period, according to the first number It can be concluded that the third number for the Region for needing to divide in the target time period.In the target time period when detecting filling rate More than the Region of default filling rate threshold value, using the Region as target Region to be divided, wherein default filling rate threshold Value is less than 1 value, so target Region is also not stored full, can not divide at this time to target Region, but Offer business it is more idle free time section splitting time target Region is divided, in this way can provide business compared with Target Region continues to provide real time business in the busy busy period, avoids largely objective without normal direction within the busy period Family end provides real time business.Target Region is divided in section during idle time, due to free time section access target Region's Number of clients is less, can reduce affected number of clients.In addition, only in the number of the target Region of acquisition Mesh just divides target Region when being less than or equal to third number, so being not in a large amount of It is not necessary to the Region divided It is divided, reduces the waste of resource.
For embodiment shown in Fig. 2, the embodiment of the present application provides a kind of method example for dividing subregion, in this method At least one target time section can be defined in example in advance, and configured in advance needs to occupy in each target time section The second number of Region in Hbase table.
Wherein, Hbase table has the function of dividing Region automatically, and the function is for automatic when some Region is filled with The Region is split into two new Region, but in the embodiment of the present application, it can be by the automatic division Region of the Hbase table Function close.
Optionally, target time section can be week age, one month or two months etc..Each object time The duration of section can be equal, can also be unequal.The second number for the Region of each target time section configuration can be equal, It can also be unequal.
It is assumed that by taking target time section is one month as an example, five target time sections of predefined configure each mesh The Region number for needing to occupy in the mark period is 6.I.e. in next five months, Hbase is can be used in every month Six Region in table carry out storing data record.
For any one target time section, if the data record generated in the target time section is more, cause for the mesh The Region for the second number that the mark period distributes in advance is insufficient, then in the second number Region in the target time section Some or all of Region divided, to divide out more Region.Referring to Fig. 3-1, the method for division Region Include:
Step 301: being held according to space used in the stored data record total number of Hbase table and the Hbase table Amount calculates the average amount of every data record in the Hbase table.
It include at least one Region in Hbase table, the capacity of each Region is equal, for storing at least one data Record, each data record be in the received data packet of message system include some or all of field contents composition.
The step of this step can be by following 3011 to 3013 calculate being averaged for every data record in Hbase table Data volume, comprising:
3011: stored data record number in each Region in Hbase table is obtained, according in each Region Stored data record number calculates the stored data record total number of the Hbase table.
Optionally, directly the data record stored in the Region can be counted, obtains having deposited in the Region The data record number of storage.
3012: obtaining used spatial content in each Region in Hbase table, used according to each Region Spatial content calculate used spatial content in the Hbase table.
For each Region in Hbase table, data record stores item by item in the Region.The Region Attribute information in include the current used spatial content of the Region, can also include the space of current Region free time Capacity.
For used spatial content in each Region in the Hbase table, can believe from the attribute of the Region It directly reads to obtain in breath.
3013: according to used spatial content in the stored data record total number of Hbase table and the Hbase table, Calculate the average amount of every data record in the Hbase table.
It can be obtained in the Hbase table by used spatial content in the Hbase table divided by the data record total number Every data record average amount.
Step 302: passing through the number for the data to be stored record that prediction model prediction generates in the target time period.
Target time section in this step is any one target time section of definition, and above-mentioned steps 301 and 302 are in the target Period executes before starting.Optionally, the previous target that above-mentioned steps 301 and 302 can be adjacent in the target time section The last day of period executes.
In this step, the initial time of objective time interval and end time can be input to prediction model, passes through prediction The number for the data to be stored record that model prediction generates in the target time period.
Optionally, before executing this step, prediction model can also be generated.The operation of the generation prediction model can be with Are as follows: the data record stored in Hbase table before being started according to target time section generates the prediction model.Referring to Fig. 3-2, the instruction Drill make detailed process can be with are as follows:
3021: corresponding timestamp being recorded according to pieces of data stored in Hbase table, obtains the first unit time collection It closes and the second unit time gathered.
Wherein, the first unit time set includes producing in each unit time stabbed between the second timestamp at the first time The first raw data record number, the second unit time set include each unit between the second timestamp and third timestamp The the first data record number generated in time, stamp is that the pieces of data in Hbase table records corresponding timestamp at the first time In from current earliest timestamp, third timestamp is that the pieces of data in Hbase table records in corresponding timestamp from current Nearest timestamp, the second timestamp are located at the first time between stamp and third timestamp.
This step can be with are as follows: can be obtained in the corresponding timestamp of stored each data record from the Hbase table from Current time earliest first time stamp and the third timestamp nearest from current time, according to the time span of unit time, Period between first time stamp and the second timestamp is divided to obtain S unit time;According in the Hbase table The corresponding timestamp of each data record of storage determines each data generated in each unit time in the S unit time Record counts the first data record number of the data record generated in each unit time;The S unit time is divided At two parts, a part is to gather the first unit time, and another part is the second unit time set.
First unit time set includes M unit time, and the second unit time set includes N number of unit time, S=M+ N, and the ratio between M and S is default ratio.
Optionally, can also by the first unit time gather in each unit time in the first data record number and The first data record number in each unit time in second unit time set is zoomed in unit range and is carried out just Then change.
3022: the parameter value of at least one RNN parameter is obtained, according to the parameter value of at least one RNN parameter setting the The RNN parameter of one RNN, obtains the 2nd RNN.
At least one RNN parameter include random seed number, learning algorithm, algorithm iteration number, weights initialisation method, At least one in direction of optimization method, learning rate, input layer, output layer and Recursive Networks algorithm etc..
Optionally, when generating prediction model, user can input corresponding first parameter value of each RNN parameter.Accordingly , in this step, corresponding first parameter value of each RNN parameter of available user's input.
For example, it is 40, learning algorithm corresponding first that user, which can input corresponding first parameter value of random seed number, Parameter value is SGD, corresponding first parameter value of algorithm iteration number is 1, corresponding first parameter value of weights initialisation method is Corresponding first parameter value of Xavier, optimization method is Adadelta, corresponding first parameter value of learning rate is 0.0004, input Corresponding first parameter value of layer is 1, corresponding first parameter value of output layer is 10 corresponding with the direction of Recursive Networks algorithm the One parameter value be it is preceding to.Then, each first parameter value for obtaining user's input is arranged the first RNN's according to each first parameter value Each parameter.
3023: according to the first data record number of each unit time in the first unit time set, passing through second RNN generates the first model.
Optionally, the first data record number of each unit time in the first unit time being gathered is input to In 2nd RNN, the output of the 2nd RNN is obtained as a result, the output result is a function, using the function as the first model.
After generating the first model, it can be gathered according to the first model, the first unit time set and the second unit time, be pressed Following 3024 and 3025 operation generates prediction model.
3024: being generated in unit time between the second timestamp and third timestamp by the first model prediction The second data record number, obtain the third unit time set.
Optionally, the second timestamp and third timestamp are input in the first model, the first model prediction simultaneously exports the The the second data record number generated in unit time between two timestamps and third timestamp.
Wherein, when the unit time number and the second unit time that third unit time set includes gather the unit for including Between number it is equal, i.e., the third unit time set include N number of unit time.
3025: when the second unit time set and third unit time set meet preset condition, the first model is true It is set to prediction model.
For the same unit time in the second unit time set and third unit time set, the unit time is calculated Difference between interior the first data record number and the second data record number, obtains the difference of the unit time;By above-mentioned The difference of each unit time in N number of unit time is calculated in mode, obtains from the difference of N number of unit time super Cross each difference of preset difference value threshold value;If the number of each difference obtained and the ratio of N are more than default fractional threshold, it is determined that Second unit time set and the third unit time set meets preset condition, if obtain each difference number and ratio It is less than default fractional threshold, it is determined that the second unit time set and third unit time set are unsatisfactory for preset condition.
3026: when the second unit time set and third unit time set are unsatisfactory for preset condition, obtaining at least one The RNN parameter of the 2nd RNN is arranged according to the second parameter value of at least one RNN parameter, obtains for second parameter value of a RNN parameter To the 3rd RNN.
It optionally, can also be in a coordinate system according to of each unit time in S unit time referring to Fig. 3-3 One data record number draws out the first waveform figure line changed with the unit time, and, according to the third unit of prediction The second data record number of each unit time in N number of unit time in time set draws out one with the unit time Second waveform figure line of variation;Then the first waveform figure line and the second waveform figure line of drafting are shown.
User can compare first waveform figure line and the second waveform figure line in this way, adjust at least one according to comparing result The parameter value of RNN parameter to get arriving the second parameter value of at least one RNN parameter, and input it is adjusted this at least one Second parameter value of RNN parameter.Correspondingly, obtaining the second parameter value of at least one RNN parameter.
Optionally, referring to Fig. 3-3, the horizontal axis of above-mentioned coordinate system is time shaft, and the scale on time shaft is the unit time, is erected Scale on axis is data record number.
Next, can be gathered according to the first unit time, the second unit time set and the 3rd RNN generate prediction mould Type.It is accomplished by detail
3027: according to the first data record number of each unit time in the first unit time set, passing through third RNN generates the second model.
Optionally, the first data record number of each unit time in the first unit time being gathered is input to In 3rd RNN, the output of the 3rd RNN is obtained as a result, the output result is a function, using the function as the second model.
3028: being generated in unit time between the second timestamp and third timestamp by the second model prediction The second data record number, obtain the third unit time set.
Optionally, the second timestamp and third timestamp are input in the second model, the second model prediction simultaneously exports the The the second data record number generated in unit time between two timestamps and third timestamp.
Wherein, when the unit time number and the second unit time that third unit time set includes gather the unit for including Between number it is equal, i.e., the third unit time set include N number of unit time.
3029: when the second unit time set and third unit time set meet preset condition, the second model is true It is set to prediction model.
When the second unit time set and third unit time set are unsatisfactory for preset condition, user's adjustment is reacquired At least one RNN parameter parameter value, then press above-mentioned 3026 to 3029 operation, until when obtaining prediction model.
Step 303: number, the average amount and the Region capacity recorded according to data to be stored is calculated in target Need to occupy the first number of the Region in Hbase table in period.
This step can be with are as follows: according to the number and the average amount of data to be stored record, calculates in the target Between the total amount of data of data to be stored record that generates in section;According to the total amount of data and Region capacity, calculate in target Need to occupy the first number of the Region in Hbase table in period.
Step 304: when the first number is greater than the second number, it is more than pre- that filling rate is obtained from the second number Region If the Region of filling rate threshold value is as target Region.
Second number Region is to need to occupy the Region in Hbase table in pre-assigned target time section.
When the first number be greater than the second number, show the data record generated in the target time period may it is more, in advance Second number Region of distribution possibly can not store all data records generated in target time section, it is also necessary to divide Third number Region out, third number are the differences between the first number and the second number.Thus in application embodiment The Region that default filling rate threshold value can be reached at first to filling rate in target time section is divided.
Optionally, when message system stores data record in a Region into Hbase table, increasing should It include used spatial content in the configuration information of Region.It also obtains in the second number Region in real time in this step Each Region filling rate, acquisition process can be with are as follows: for each Region, reads from the configuration information of the Region The used spatial content of the Region calculates the Region according to the used spatial content and the Region capacity Filling rate.
In this step, the filling rate for obtaining each Region in the second number Region in real time, when some The filling rate of Region is more than default filling rate threshold value, then executes the operation of following steps 305.
Step 305: when acquisition target Region number be less than or equal to third number, determine Region pairs of the target The splitting time answered, the splitting time are later than the acquisition time for obtaining target Region.
Specifically, it is determined that the date on the same day where the acquisition time in the domain target Region is obtained, choosing in the date in this prior It selects a time point and the time point is determined as the corresponding splitting time of target Region, which is later than the same day Preset time point in date.
In this step, daily preset time point can be defined on and divide Region, daily preset time point later Time later is usually the period more idle in the business of offer.For example, being usually to provide after 10 points of daily night The business more idle period, so preset time point can be 10 points of night, 10 thirty or 11 points etc..It may be implemented in this way Region is provided in the offer business more idle period, affected number of clients can be reduced to the greatest extent.
Step 306: divide target Region in the corresponding splitting time of target Region, obtain two it is new Region。
Division obtain two new Region after, data record stored in target Region can be stored in this two In a new Region.
Optionally, data record stored in target Region can be averagely stored in two new Region In.
Optionally, if in the target time period, the number of the target Region obtained is greater than third number, show Third number Region is divided out, current Region stores the data record generated in the target time period enough, so Stop the target Region that division obtains.
Wherein, target Region also corresponds to an identification section, which includes that origin identification and end are got to know, to mesh When marking Region and storing data record, also according to the create-rule of rowkey, the unique identification data record is generated Rowkey, rowkey is the unique identification of data record, which is located in the identification section.
When target Region is split into two new Region, the available mark positioned at the identification section, at this A suffix is added on the basis of mark to obtain separating mark.Optionally, the mark of acquisition can be the median of the identification section Or other numerical value.
The identification section of one of them new Region can be set as to the mark from the origin identification to the separation between mark Know section, the identification section of another new Region is set as the identification section identified to the end of identification from the separation.
Wherein, it should be understood that the suffix added in the present embodiment is shorter, which only has one or a few units The characters such as word and/or letter can make separation sign know shorter in this way, keep the origin identification of Region and end of identification shorter. In this way, existing after obtaining data record and generating the rowkey of data record by the data record storage When Region, can this according to the rowkey match Region identification section, by the data record storage with this rowkey In the Region matched, since the starting of the identification section is got to know shorter with end of identification, matched speed can be improved in this way, improve Storage speed.
In the embodiment of the present application, by predicting the first number of Region in the target time period, according to the first number It can be concluded that the third number for the Region for needing to divide in the target time period.When third number is not 0, in the object time When detecting that filling rate is more than the Region of default filling rate threshold value in section, using the Region as target to be divided Region, wherein default filling rate threshold value is less than 1 value, so target Region is also not stored full, it at this time can not be to mesh Mark Region is divided, but the splitting time in the more idle free time section of offer business divides target Region It splits, target Region can continue to provide real time business within the offer business more busy busy period in this way, avoid busy Real time business is provided without a large amount of client of normal direction in the commonplace period.Target Region is divided in section during idle time, due to sky The number of clients of not busy period access target Region is less, can reduce affected number of clients.In addition, only Have and just divide target Region when the number of the target Region of acquisition is less than or equal to third number, so being not in pair A large amount of Region for being not necessarily to division are divided, and the memory space for having many Region in the target time period is caused not have Have and used, reduces the waste of storage resource.
Following is the application Installation practice, can be used for executing the application embodiment of the method.It is real for the application device Undisclosed details in example is applied, the application embodiment of the method is please referred to.
Referring to fig. 4, the embodiment of the present application provides the first optional implementation, provides in the first optional implementation A kind of device 400 dividing subregion, described device 400 include:
Prediction module 401, for stored data record in Hbase table before being started according to target time section, prediction exists Need to occupy the first number of the Region in the Hbase table in the target time section;
Module 402 is obtained, is used for when first number is greater than the second number, from the second number Region Obtaining filling rate is more than to preset the Region of filling rate threshold value as target Region, and the second number Region is preparatory The Region for needing to occupy in the target time section in the Hbase table of distribution;
Divide module 403, for when the number of the target Region of acquisition be less than or equal to third number when, in the mesh It marks the corresponding splitting time of Region and divides the target Region, the splitting time, which is later than, obtains the target Region's Acquisition time, the third number are equal to first number and subtract second number.
Optionally, optional at second of the embodiment of the present application in conjunction with the first optional implementation of the embodiment of the present application Implementation in, the prediction module 401 includes:
First computing unit, for having been used according in the stored data record total amount of Hbase table and the Hbase table Spatial content, calculate in the Hbase table every data record average amount;
Predicting unit, what the data to be stored for being generated in the target time section by prediction model prediction recorded Number;
Second computing unit, number, the average amount and Region for being recorded according to the data to be stored Capacity calculates the first number for needing to occupy the Region in the Hbase table in the target time section.
Optionally, in conjunction with the first optional implementation of the embodiment of the present application or second of optional implementation, In In the third optional implementation of the embodiment of the present application, described device 400 further include:
Generation module, the data record for storing in the Hbase table before being started according to the target time section, generates The prediction model.
Optionally, in conjunction with the third optional implementation of the embodiment of the present application, the 4th kind in the embodiment of the present application can It selects in implementation, the generation module includes:
First acquisition unit obtains first for recording corresponding timestamp according to the pieces of data in the Hbase table Unit time set and the second unit time set, the first unit time set include at the first time between stamp and the second timestamp Each unit time in the first data record number for generating, second unit time set includes the second timestamp and the The the first data record number generated in each unit time between three timestamps, stamp is in the Hbase table at the first time Pieces of data record from current earliest timestamp in corresponding timestamp, third timestamp is each in the Hbase table Data is recorded from current nearest timestamp in corresponding timestamp, the second timestamp be located at first time stamp with it is described Between third timestamp;
Second acquisition unit, for obtaining the first parameter value at least one RNN parameter, according to it is described at least one The RNN parameter of the first RNN is arranged in first parameter value of RNN parameter, obtains the 2nd RNN;
Generation unit, for according to first unit time set, second unit time set and described second RNN generates the prediction model.
Optionally, in conjunction with the 4th kind of optional implementation of the embodiment of the present application, the 5th kind in the embodiment of the present application can It selects in implementation, the generation unit is used for:
According to the first data record number of each unit time in the first unit time set, pass through described second RNN generates the first model;
It is generated in the unit time between the second timestamp and third timestamp by first model prediction The second data record number, obtain the third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as pre- Survey model.
Optionally, in conjunction with the 5th kind of optional implementation of the embodiment of the present application, the 6th kind in the embodiment of the present application can It selects in implementation, the generation unit is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, it is corresponding to obtain RNN parameter The second parameter value, the RNN parameter of the 2nd RNN is set according to corresponding second parameter value of the RNN parameter, obtains third RNN;
It is generated according to first unit time set, second unit time set and the 3rd RNN described pre- Survey model.
Optionally, in conjunction with any optional realization side in first to the 6th kind of optional implementation of the embodiment of the present application Formula, in the 7th kind of optional implementation of the embodiment of the present application, described device 400 further include:
Determining module on the date on the same day where acquisition time for determining the acquisition target Region, is worked as described One time point of selection and the time point is determined as the corresponding splitting time of the target Region in the preceding date, described point Split the preset time point that the time is later than in the date on the same day.
Optionally, in conjunction with any optional realization side in first to the 7th kind of optional implementation of the embodiment of the present application Formula, in the 8th kind of optional implementation of the embodiment of the present application, described device 400 further include:
Memory module, for the data packet in the spatial cache of message system to be stored in Hbase's according to configuration file In Region;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, and theme is related Information includes at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least column belonging to field name and field Race.
Optionally, in conjunction with the 8th kind of optional implementation of the embodiment of the present application, the 9th kind in the embodiment of the present application can It selects in implementation, the memory module includes:
Third acquiring unit, for the corresponding subject identification of spatial cache according to the message system, from configuration file It is middle to obtain the subject correlation message including the subject identification, it further include the mark and object of Hbase table in the subject information The set identification of set;
4th acquiring unit, for obtaining the set identification from the data packet in the spatial cache of the message system The corresponding field contents of each field name in corresponding object set;
Storage unit, for each field contents of the acquisition to be formed data record, according to the object set In each field affiliated column family the data record is stored in the Hbase table the corresponding Hbase table of mark Region In.
In the embodiment of the present application, the first number that Region in the target time period is predicted by prediction module, according to First number it can be concluded that the Region for needing to divide in the target time period third number.So in the target time period when When detecting that filling rate is more than the target Region of default filling rate threshold value, if the target Region number obtained is less than or waits In third number, target Region can be divided in the splitting time of the more idle free time section of offer business by dividing module It splits, can reduce affected number of clients, while the target Region number also divided by the control of third number, It avoids dividing excessive Region, and the storage resource of a large amount of Region is caused to waste.
About the device in above-described embodiment, wherein modules execute the concrete mode of operation in related this method Embodiment in be described in detail, no detailed explanation will be given here.
Fig. 5 shows the structural block diagram of the terminal 500 of an illustrative embodiment of the invention offer, which can be with For total node in distributed memory system.The terminal 500 can be portable mobile termianl, such as: smart phone, plate electricity (Moving Picture Experts Group Audio Layer III, dynamic image expert compress mark for brain, MP3 player Quasi- audio level 3), MP4 (Moving Picture Experts Group Audio Layer IV, dynamic image expert compression Standard audio level 4) player, laptop or desktop computer.Terminal 500 is also possible to referred to as user equipment, portable Other titles such as terminal, laptop terminal, terminal console.
In general, terminal 500 includes: processor 501 and memory 502.
Processor 501 may include one or more processing cores, such as 4 core processors, 8 core processors etc..Place Reason device 501 can use DSP (Digital Signal Processing, Digital Signal Processing), FPGA (Field- Programmable Gate Array, field programmable gate array), PLA (Programmable Logic Array, may be programmed Logic array) at least one of example, in hardware realize.Processor 501 also may include primary processor and coprocessor, master Processor is the processor for being handled data in the awake state, also referred to as CPU (Central Processing Unit, central processing unit);Coprocessor is the low power processor for being handled data in the standby state.In In some embodiments, processor 501 can be integrated with GPU (Graphics Processing Unit, image processor), GPU is used to be responsible for the rendering and drafting of content to be shown needed for display screen.In some embodiments, processor 501 can also be wrapped AI (Artificial Intelligence, artificial intelligence) processor is included, the AI processor is for handling related machine learning Calculating operation.
Memory 502 may include one or more computer readable storage mediums, which can To be non-transient.Memory 502 may also include high-speed random access memory and nonvolatile memory, such as one Or multiple disk storage equipments, flash memory device.In some embodiments, the non-transient computer in memory 502 can Storage medium is read for storing at least one instruction, at least one instruction performed by processor 501 for realizing this Shen Please in embodiment of the method provide a kind of division Region method.
In some embodiments, terminal 500 is also optional includes: peripheral device interface 503 and at least one peripheral equipment. It can be connected by bus or signal wire between processor 501, memory 502 and peripheral device interface 503.Each peripheral equipment It can be connected by bus, signal wire or circuit board with peripheral device interface 503.Specifically, peripheral equipment includes: radio circuit 504, at least one of touch display screen 505, camera 506, voicefrequency circuit 507, positioning component 508 and power supply 509.
Peripheral device interface 503 can be used for I/O (Input/Output, input/output) is relevant outside at least one Peripheral equipment is connected to processor 501 and memory 502.In some embodiments, processor 501, memory 502 and peripheral equipment Interface 503 is integrated on same chip or circuit board;In some other embodiments, processor 501, memory 502 and outer Any one or two in peripheral equipment interface 503 can realize on individual chip or circuit board, the present embodiment to this not It is limited.
Radio circuit 504 is for receiving and emitting RF (Radio Frequency, radio frequency) signal, also referred to as electromagnetic signal.It penetrates Frequency circuit 504 is communicated by electromagnetic signal with communication network and other communication equipments.Radio circuit 504 turns electric signal It is changed to electromagnetic signal to be sent, alternatively, the electromagnetic signal received is converted to electric signal.Optionally, radio circuit 504 wraps It includes: antenna system, RF transceiver, one or more amplifiers, tuner, oscillator, digital signal processor, codec chip Group, user identity module card etc..Radio circuit 504 can be carried out by least one wireless communication protocol with other terminals Communication.The wireless communication protocol includes but is not limited to: WWW, Metropolitan Area Network (MAN), Intranet, each third generation mobile communication network (2G, 3G, 4G and 5G), WLAN and/or WiFi (Wireless Fidelity, Wireless Fidelity) network.In some embodiments, it penetrates Frequency circuit 504 can also include NFC (Near Field Communication, wireless near field communication) related circuit, this Application is not limited this.
Display screen 505 is for showing UI (User Interface, user interface).The UI may include figure, text, figure Mark, video and its their any combination.When display screen 505 is touch display screen, display screen 505 also there is acquisition to show The ability of the touch signal on the surface or surface of screen 505.The touch signal can be used as control signal and be input to processor 501 are handled.At this point, display screen 505 can be also used for providing virtual push button and/or dummy keyboard, also referred to as soft button and/or Soft keyboard.In some embodiments, display screen 505 can be one, and the front panel of terminal 500 is arranged;In other embodiments In, display screen 505 can be at least two, be separately positioned on the different surfaces of terminal 500 or in foldover design;In still other reality It applies in example, display screen 505 can be flexible display screen, be arranged on the curved surface of terminal 500 or on fold plane.Even, it shows Display screen 505 can also be arranged to non-rectangle irregular figure, namely abnormity screen.Display screen 505 can use LCD (Liquid Crystal Display, liquid crystal display), OLED (Organic Light-Emitting Diode, Organic Light Emitting Diode) Etc. materials preparation.
CCD camera assembly 506 is for acquiring image or video.Optionally, CCD camera assembly 506 include front camera and Rear camera.In general, the front panel of terminal is arranged in front camera, the back side of terminal is arranged in rear camera.One In a little embodiments, rear camera at least two is main camera, depth of field camera, wide-angle camera, focal length camera shooting respectively Any one in head, to realize that main camera and the fusion of depth of field camera realize background blurring function, main camera and wide-angle Camera fusion realizes that pan-shot and VR (Virtual Reality, virtual reality) shooting function or other fusions are clapped Camera shooting function.In some embodiments, CCD camera assembly 506 can also include flash lamp.Flash lamp can be monochromatic warm flash lamp, It is also possible to double-colored temperature flash lamp.Double-colored temperature flash lamp refers to the combination of warm light flash lamp and cold light flash lamp, can be used for not With the light compensation under colour temperature.
Voicefrequency circuit 507 may include microphone and loudspeaker.Microphone is used to acquire the sound wave of user and environment, and will Sound wave, which is converted to electric signal and is input to processor 501, to be handled, or is input to radio circuit 504 to realize voice communication. For stereo acquisition or the purpose of noise reduction, microphone can be separately positioned on the different parts of terminal 500 to be multiple.Mike Wind can also be array microphone or omnidirectional's acquisition type microphone.Loudspeaker is then used to that processor 501 or radio circuit will to be come from 504 electric signal is converted to sound wave.Loudspeaker can be traditional wafer speaker, be also possible to piezoelectric ceramic loudspeaker.When When loudspeaker is piezoelectric ceramic loudspeaker, the audible sound wave of the mankind can be not only converted electrical signals to, it can also be by telecommunications Number the sound wave that the mankind do not hear is converted to carry out the purposes such as ranging.In some embodiments, voicefrequency circuit 507 can also include Earphone jack.
Positioning component 508 is used for the current geographic position of positioning terminal 500, to realize navigation or LBS (Location Based Service, location based service).Positioning component 508 can be the GPS (Global based on the U.S. Positioning System, global positioning system), China dipper system or Russia Galileo system positioning group Part.
Power supply 509 is used to be powered for the various components in terminal 500.Power supply 509 can be alternating current, direct current, Disposable battery or rechargeable battery.When power supply 509 includes rechargeable battery, which can be wired charging electricity Pond or wireless charging battery.Wired charging battery is the battery to be charged by Wireline, and wireless charging battery is by wireless The battery of coil charges.The rechargeable battery can be also used for supporting fast charge technology.
In some embodiments, terminal 500 further includes having one or more sensors 510.The one or more sensors 510 include but is not limited to: acceleration transducer 511, gyro sensor 512, pressure sensor 513, fingerprint sensor 514, Optical sensor 515 and proximity sensor 516.
The acceleration that acceleration transducer 511 can detecte in three reference axis of the coordinate system established with terminal 500 is big It is small.For example, acceleration transducer 511 can be used for detecting component of the acceleration of gravity in three reference axis.Processor 501 can With the acceleration of gravity signal acquired according to acceleration transducer 511, touch display screen 505 is controlled with transverse views or longitudinal view Figure carries out the display of user interface.Acceleration transducer 511 can be also used for the acquisition of game or the exercise data of user.
Gyro sensor 512 can detecte body direction and the rotational angle of terminal 500, and gyro sensor 512 can To cooperate with acquisition user to act the 3D of terminal 500 with acceleration transducer 511.Processor 501 is according to gyro sensor 512 Following function may be implemented in the data of acquisition: when action induction (for example changing UI according to the tilt operation of user), shooting Image stabilization, game control and inertial navigation.
The lower layer of side frame and/or touch display screen 505 in terminal 500 can be set in pressure sensor 513.Work as pressure When the side frame of terminal 500 is arranged in sensor 513, user can detecte to the gripping signal of terminal 500, by processor 501 Right-hand man's identification or prompt operation are carried out according to the gripping signal that pressure sensor 513 acquires.When the setting of pressure sensor 513 exists When the lower layer of touch display screen 505, the pressure operation of touch display screen 505 is realized to UI circle according to user by processor 501 Operability control on face is controlled.Operability control includes button control, scroll bar control, icon control, menu At least one of control.
Fingerprint sensor 514 is used to acquire the fingerprint of user, collected according to fingerprint sensor 514 by processor 501 The identity of fingerprint recognition user, alternatively, by fingerprint sensor 514 according to the identity of collected fingerprint recognition user.It is identifying When the identity of user is trusted identity out, the user is authorized to execute relevant sensitive operation, the sensitive operation packet by processor 501 Include solution lock screen, check encryption information, downloading software, payment and change setting etc..Terminal can be set in fingerprint sensor 514 500 front, the back side or side.When being provided with physical button or manufacturer Logo in terminal 500, fingerprint sensor 514 can be with It is integrated with physical button or manufacturer Logo.
Optical sensor 515 is for acquiring ambient light intensity.In one embodiment, processor 501 can be according to optics The ambient light intensity that sensor 515 acquires controls the display brightness of touch display screen 505.Specifically, when ambient light intensity is higher When, the display brightness of touch display screen 505 is turned up;When ambient light intensity is lower, the display for turning down touch display screen 505 is bright Degree.In another embodiment, the ambient light intensity that processor 501 can also be acquired according to optical sensor 515, dynamic adjust The acquisition parameters of CCD camera assembly 506.
Proximity sensor 516, also referred to as range sensor are generally arranged at the front panel of terminal 500.Proximity sensor 516 For acquiring the distance between the front of user Yu terminal 500.In one embodiment, when proximity sensor 516 detects use When family and the distance between the front of terminal 500 gradually become smaller, touch display screen 505 is controlled from bright screen state by processor 501 It is switched to breath screen state;When proximity sensor 516 detects user and the distance between the front of terminal 500 becomes larger, Touch display screen 505 is controlled by processor 501 and is switched to bright screen state from breath screen state.
It will be understood by those skilled in the art that the restriction of the not structure paired terminal 500 of structure shown in Fig. 5, can wrap It includes than illustrating more or fewer components, perhaps combine certain components or is arranged using different components.
Those skilled in the art will readily occur to its of the application after considering specification and practicing application disclosed herein Its embodiment.This application is intended to cover any variations, uses, or adaptations of the application, these modifications, purposes or Person's adaptive change follows the general principle of the application and including the undocumented common knowledge in the art of the application Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the application are by following Claim is pointed out.
It should be understood that the application is not limited to the precise structure that has been described above and shown in the drawings, and And various modifications and changes may be made without departing from the scope thereof.Scope of the present application is only limited by the accompanying claims.

Claims (18)

1. a method of division subregion, which is characterized in that the described method includes:
Stored data record in Hbase table, prediction need in the target time section before being started according to target time section Occupy the first number of the subregion in the Hbase table;
When first number is greater than the second number, it is more than default storage that filling rate is obtained from the second number subregion For the subregion of rate threshold value as target partition, the second number subregion is pre-assigned to need in the target time section Occupy the subregion in the Hbase table;
When the number of the target partition of acquisition is less than or equal to third number, in the corresponding splitting time point of the target partition The target partition is split, the splitting time is later than the acquisition time for obtaining the target partition, and the third number is equal to institute It states the first number and subtracts second number.
2. the method as described in claim 1, which is characterized in that described to start to have deposited in preceding Hbase table according to target time section The data record of storage, prediction need to occupy in the target time section the first number of the subregion in the Hbase table, packet It includes:
According to used spatial content in the stored data record total amount of Hbase table and the Hbase table, described in calculating The average amount of every data record in Hbase table;
Pass through the number for the data to be stored record that prediction model prediction generates in the target time section;
According to number, the average amount and subregion capacity that the data to be stored records, calculate in the object time Need to occupy the first number of the subregion in the Hbase table in section.
3. method according to claim 2, which is characterized in that the prediction needs to occupy in the target time section described Before first number of the subregion in Hbase table, further includes:
The data record stored in the Hbase table before being started according to the target time section, generates the prediction model.
4. method as claimed in claim 3, which is characterized in that described to start the preceding Hbase according to the target time section The data record stored in table generates the prediction model, comprising:
Corresponding timestamp is recorded according to the pieces of data in the Hbase table, obtains the first unit time set and the second list Position time set, the first unit time set in each unit time at the first time between stamp and the second timestamp including generating The first data record number, second unit time set includes each list between the second timestamp and third timestamp The the first data record number generated in the time of position, it is corresponding to be that pieces of data in the Hbase table records for stamp at the first time From current earliest timestamp in timestamp, third timestamp is that the pieces of data in the Hbase table records the corresponding time From current nearest timestamp in stamp, the second timestamp is located between first time stamp and the third timestamp;
The first parameter value at least one Recognition with Recurrent Neural Network RNN parameter is obtained, according to the of at least one RNN parameter The RNN parameter of the first RNN is arranged in one parameter value, obtains the 2nd RNN;
The prediction mould is generated according to first unit time set, second unit time set and the 2nd RNN Type.
5. method as claimed in claim 4, which is characterized in that described according to first unit time set, described second Unit time set and the 2nd RNN generate the prediction model, comprising:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set At the first model;
Generated in the unit time between the second timestamp and third timestamp by first model prediction Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as to predict mould Type.
6. method as claimed in claim 5, which is characterized in that the method also includes:
When the second unit time set and third unit time set are unsatisfactory for preset condition, RNN parameter corresponding the is obtained The RNN parameter of the 2nd RNN is arranged according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN for two parameter values;
The prediction mould is generated according to first unit time set, second unit time set and the 3rd RNN Type.
7. the method as described in claim 1, which is characterized in that the filling rate that obtains from the second number subregion surpasses After the subregion for presetting filling rate threshold value is crossed as target partition, further includes:
It determines the date on the same day where the acquisition time for obtaining the target partition, a time is selected in the current date The time point is simultaneously determined as the corresponding splitting time of the target partition by point, and the splitting time is later than the date on the same day Interior preset time point.
8. method as described in any one of claim 1 to 7, which is characterized in that the method also includes:
The data packet in the spatial cache of message system is stored in the subregion of Hbase according to configuration file;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, subject correlation message Including at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least field name and the affiliated column family of field.
9. method according to claim 8, which is characterized in that it is described will be in the spatial cache of message system according to configuration file Data packet be stored in the subregion of Hbase, comprising:
According to the corresponding subject identification of the spatial cache of the message system, obtaining from configuration file includes the subject identification Subject correlation message, further include the mark of Hbase table and the set identification of object set in the subject information;
It is every in the corresponding object set of the set identification from being obtained in the data packet in the spatial cache of the message system The corresponding field contents of a field name;
Each field contents of the acquisition are formed into data record, according to the affiliated column family of each field in the object set The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table.
10. a kind of device for dividing subregion, which is characterized in that described device includes:
Prediction module is predicted for stored data record in Hbase table before being started according to target time section in the target Need to occupy the first number of the subregion in the Hbase table in period;
Module is obtained, for obtaining storage from the second number subregion when first number is greater than the second number Rate is more than the subregion of default filling rate threshold value as target partition, and the second number subregion is pre-assigned in the mesh Need to occupy the subregion in the Hbase table in the mark period;
Divide module, for when the number of the target partition of acquisition be less than or equal to third number when, in the target partition pair The splitting time answered divides the target partition, and the splitting time is later than the acquisition time for obtaining the target partition, described Third number is equal to first number and subtracts second number.
11. device as claimed in claim 10, which is characterized in that the prediction module includes:
First computing unit, for according to used sky in the stored data record total amount of Hbase table and the Hbase table Between capacity, calculate in the Hbase table every data record average amount;
Predicting unit, the number of the data to be stored record for being generated in the target time section by prediction model prediction Mesh;
Second computing unit, number, the average amount and subregion capacity for being recorded according to the data to be stored, meter Calculate the first number for needing to occupy the subregion in the Hbase table in the target time section.
12. device as claimed in claim 11, which is characterized in that described device further include:
Generation module, the data record for being stored in the Hbase table before being started according to the target time section, described in generation Prediction model.
13. device as claimed in claim 12, which is characterized in that the generation module includes:
First acquisition unit obtains the first unit for recording corresponding timestamp according to the pieces of data in the Hbase table Time set and the second unit time set, the first unit time set include every between stamp and the second timestamp at the first time The the first data record number generated in a unit time, when the second unit time set includes the second timestamp and third Between stamp between each unit time in generate the first data record number, at the first time stamp be each in the Hbase table Data is recorded from current earliest timestamp in corresponding timestamp, and third timestamp is each item number in the Hbase table According to recording from current nearest timestamp in corresponding timestamp, the second timestamp is located at first time stamp and the third Between timestamp;
Second acquisition unit, for obtaining the first parameter value at least one Recognition with Recurrent Neural Network RNN parameter, according to it is described extremely The RNN parameter of the first RNN is arranged in first parameter value of a few RNN parameter, obtains the 2nd RNN;
Generation unit, for raw according to first unit time set, second unit time set and the 2nd RNN At the prediction model.
14. device as claimed in claim 13, which is characterized in that the generation unit is used for:
It is raw by the 2nd RNN according to the first data record number of each unit time in the first unit time set At the first model;
Generated in the unit time between the second timestamp and third timestamp by first model prediction Two data record numbers obtain third unit time set;
When the second unit time set and third unit time set meet preset condition, the first model is determined as to predict mould Type.
15. device as claimed in claim 14, which is characterized in that the generation unit is also used to:
When the second unit time set and third unit time set are unsatisfactory for preset condition, RNN parameter corresponding the is obtained The RNN parameter of the 2nd RNN is arranged according to corresponding second parameter value of the RNN parameter, obtains the 3rd RNN for two parameter values;
The prediction mould is generated according to first unit time set, second unit time set and the 3rd RNN Type.
16. device as claimed in claim 10, which is characterized in that described device further include:
Determining module, the date on the same day where acquisition time for determining the acquisition target partition, in the current date The time point is simultaneously determined as the corresponding splitting time of the target partition, the splitting time evening by one time point of interior selection In the preset time point in the date on the same day.
17. such as the described in any item devices of claim 10 to 16, which is characterized in that described device further include:
Memory module, for the data packet in the spatial cache of message system to be stored in the subregion of Hbase according to configuration file In;
Wherein, the configuration file includes at least one subject correlation message and at least one object set, subject correlation message Including at least the set identification of subject identification, the mark of Hbase table and object set;
Object set includes at least one field domain information, and field domain information includes at least field name and the affiliated column family of field.
18. device as claimed in claim 17, which is characterized in that the memory module includes:
Third acquiring unit is obtained from configuration file for the corresponding subject identification of spatial cache according to the message system The subject correlation message including the subject identification is taken, further includes the mark and object set of Hbase table in the subject information Set identification;
4th acquiring unit, it is corresponding for obtaining the set identification from the data packet in the spatial cache of the message system Object set in the corresponding field contents of each field name;
Storage unit, for each field contents of the acquisition to be formed data record, according in the object set The data record is stored in the subregion of the corresponding Hbase table of mark of the Hbase table by each affiliated column family of field.
CN201810494401.5A 2018-05-22 2018-05-22 Method and device for splitting partitions Active CN110519319B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810494401.5A CN110519319B (en) 2018-05-22 2018-05-22 Method and device for splitting partitions

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810494401.5A CN110519319B (en) 2018-05-22 2018-05-22 Method and device for splitting partitions

Publications (2)

Publication Number Publication Date
CN110519319A true CN110519319A (en) 2019-11-29
CN110519319B CN110519319B (en) 2022-02-11

Family

ID=68621791

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810494401.5A Active CN110519319B (en) 2018-05-22 2018-05-22 Method and device for splitting partitions

Country Status (1)

Country Link
CN (1) CN110519319B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110888846A (en) * 2019-12-10 2020-03-17 北京北龙云海网络数据科技有限责任公司 Data memory management method and system

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6023706A (en) * 1997-07-11 2000-02-08 International Business Machines Corporation Parallel file system and method for multiple node file access
US20160241890A1 (en) * 2013-08-20 2016-08-18 Lg Electronics Inc. Apparatus for transmitting media data via streaming service, apparatus for receiving media data via streaming service, method for transmitting media data via streaming service and method for receiving media data via streaming service
CN105988995A (en) * 2015-01-27 2016-10-05 杭州海康威视数字技术股份有限公司 HFile based data batch loading method
CN107169009A (en) * 2017-03-31 2017-09-15 北京奇艺世纪科技有限公司 A kind of data splitting method and device of distributed memory system
CN107480205A (en) * 2017-07-24 2017-12-15 北京京东尚科信息技术有限公司 A kind of method and apparatus for carrying out data partition
CN107943412A (en) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 A kind of subregion division, the method, apparatus and system for deleting data file in subregion

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6023706A (en) * 1997-07-11 2000-02-08 International Business Machines Corporation Parallel file system and method for multiple node file access
US20160241890A1 (en) * 2013-08-20 2016-08-18 Lg Electronics Inc. Apparatus for transmitting media data via streaming service, apparatus for receiving media data via streaming service, method for transmitting media data via streaming service and method for receiving media data via streaming service
CN105988995A (en) * 2015-01-27 2016-10-05 杭州海康威视数字技术股份有限公司 HFile based data batch loading method
CN107943412A (en) * 2016-10-12 2018-04-20 阿里巴巴集团控股有限公司 A kind of subregion division, the method, apparatus and system for deleting data file in subregion
CN107169009A (en) * 2017-03-31 2017-09-15 北京奇艺世纪科技有限公司 A kind of data splitting method and device of distributed memory system
CN107480205A (en) * 2017-07-24 2017-12-15 北京京东尚科信息技术有限公司 A kind of method and apparatus for carrying out data partition

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
XIAOFENG LEI: ""BESIII Physics Data Storing and Processing on HBase and MapReduce"", 《21ST INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS》 *
王卓: ""基于增量式分区策略的MapReduce数据均衡方法"", 《计算机学报》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110888846A (en) * 2019-12-10 2020-03-17 北京北龙云海网络数据科技有限责任公司 Data memory management method and system

Also Published As

Publication number Publication date
CN110519319B (en) 2022-02-11

Similar Documents

Publication Publication Date Title
CN110110263B (en) Webpage display method, device, terminal and storage medium
CN110290421A (en) Frame per second method of adjustment, device, computer equipment and storage medium
CN109600678A (en) Information displaying method, apparatus and system, server, terminal, storage medium
CN110213616A (en) Video providing method, acquisition methods, device and equipment
CN109656930A (en) Data query method, apparatus and system
CN109816042B (en) Data classification model training method and device, electronic equipment and storage medium
CN109618212A (en) Information display method, device, terminal and storage medium
CN110019929A (en) Processing method, device and the computer readable storage medium of web page contents
CN110166786A (en) Virtual objects transfer method and device
CN109302385A (en) Multimedia resource sharing method, device and storage medium
CN108449651A (en) Subtitle adding method and device
CN110278464A (en) The method and apparatus for showing list
CN110147503B (en) Information issuing method and device, computer equipment and storage medium
CN109922356A (en) Video recommendation method, device and computer readable storage medium
CN110380904A (en) Bandwidth allocation methods, device, electronic equipment and storage medium
CN108900925A (en) The method and apparatus of live streaming template are set
CN110418152A (en) It is broadcast live the method and device of prompt
CN111031391A (en) Video dubbing method, device, server, terminal and storage medium
CN109218751A (en) The method, apparatus and system of recommendation of audio
CN110535890A (en) The method and apparatus that file uploads
CN113032587A (en) Multimedia information recommendation method, system, device, terminal and server
CN110045958A (en) Data texturing generation method, device, storage medium and equipment
CN109889858A (en) Information processing method, device and the computer readable storage medium of virtual objects
CN109600301A (en) Message treatment method and device
CN110471614A (en) A kind of method of storing data, the method and device for detecting terminal

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant