CN104484285B

CN104484285B - A kind of memory management method and device

Info

Publication number: CN104484285B
Application number: CN201410750737.5A
Authority: CN
Inventors: 洪斯宝; 夏命榛
Original assignee: Hangzhou Huawei Digital Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2014-12-09
Filing date: 2014-12-09
Publication date: 2017-11-17
Anticipated expiration: 2034-12-09
Also published as: CN104484285A

Abstract

The present embodiments relate to stream calculation technical field.The embodiment of the present invention provides memory management method, including：Receive a plurality of data record from the stream calculation its exterior；At least two packets are created in the spatial cache of the stream calculation system, are that a continuous memory space of physical address is allocated in each packet at least two packet in advance from the memory space of the stream calculation system；To being recorded in a plurality of data record of reception per data, stored according to the processing method below for data record r：According to the metadata of the data record r, the packet for storing the data record r is selected from described at least two packets, and the data record r is stored in the memory space for being used to store the packet of the data record r.It can accelerate the speed of service of streaming application using technical scheme provided in an embodiment of the present invention, the treatment effeciency of streaming application is provided.The embodiment of the present invention also provides corresponding device.

Description

A kind of memory management method and device

Technical field

The present invention relates to stream calculation technical field, more particularly to a kind of memory management method and device.

Background technology

With the arrival in big data epoch, mass data progress is handled in real time, analyzed, the market demand of decision-making constantly expands Greatly, stream calculation system is arisen at the historic moment.The application program disposed in stream calculation system is referred to as streaming application, and streaming application is to flow Data record in computing system is process object, and data record is in units of bar.

In the prior art, a plurality of data record of the spatial cache of the stream calculation system is stored in, is according to reception Time sequencing storage.Streaming application combines operationally, it is necessary to according to the application logic of itself and is located at stream calculation system The metadata recorded in a plurality of data record per data, process object is selected from a plurality of data record, wherein, data The metadata of record describes the attribute information of the data record.The defects of this scheme, is：It is determined that a certain streaming application Process object when, it is necessary to be selected one by one according to condition from positioned at a plurality of data record of stream calculation system, so as to lead Cause to determine streaming application the speed of service is slow, treatment effeciency is than relatively low.

The content of the invention

The present invention is used for the upgrade method and intelligent grid for providing a kind of intelligent grid, for simplified pair to a certain extent Intelligent grid carries out process during the whole network upgrading.

In a first aspect, the embodiment of the present invention provides a kind of memory management method, applied to stream calculation system, this method bag Include：

Receive a plurality of data record from the stream calculation its exterior；

At least two packets are created in the spatial cache of the stream calculation system, from the memory space of the stream calculation system In be that a continuous memory space of physical address is allocated in each packet at least two packet in advance, described at least two Each packet in individual packet includes the data record of one or more, and at least one at least two packet is grouped Including more than two data records, all data records that each packet at least two packet includes are stored in often The one respective memory space of packet；

Each packet comprises at least following attribute informations at least two packet：Metadata and initial address；Institute It is identical with the metadata of each each self-contained data record of packet to state the metadata of each packet at least two packets；It is described The initial address of each packet and the first data of the respective memory space storage of each packet record at least two packets Initial address it is identical；

To being recorded in a plurality of data record of reception per data, according to the processing side below for data record r Method is stored：

According to the metadata of the data record r, selected from described at least two packets for storing the data note R packet is recorded, the metadata of packet for being used to store the data record r and the metadata of the data record r are phases With；

And the data record r is stored in the memory space for being used to store the packet of the data record r.

With reference in a first aspect, under the first embodiment of first aspect,

It has been filled with the memory space of the packet for storing the data record r, it is impossible to be restored again into the data In the case of recording r,

It is described that the data record r is stored in the memory space for being used to store the packet of the data record r, specifically Including：

The timestamp that every a data according to being included in the packet for storing the data record r records, choosing Reception time earliest data record is selected out, and deletes the reception time earliest data record, it is described to remember per a data The timestamp of record records the respective reception time for recording every a data；The data record r is stored in for depositing Store up the memory space of the reception time earliest data record.

With reference in a first aspect, under second of embodiment of first aspect,

Each packet also includes following attribute informations at least two packet：Group character, described at least two points Different packets has different group characters in group；

Group character is added in the data record r, the group character added in the data record r is used for described It is identical to store the group character of the packet of the data record r, and data record r deposits is described for storing the number According to the memory space of record r packet.

With reference to the first embodiment of first aspect, first aspect or second of embodiment of first aspect, Under the third embodiment of first aspect,

At least two packet is source packet, and the data record positioned at the source packet is to come from the stream calculation Its exterior.

With reference to the third embodiment of first aspect, under the 4th kind of embodiment of first aspect, described pair of reception The a plurality of data record in per data record stored after, in addition to：

Receive at least one data record from the stream calculation its exterior；

Recorded for every a data at least one data record, according to following behaviour for data record p Stored as method：

, will when the source packet for storing the data record p in all source packets of the spatial cache be present The data record p deposits source packet for being used to store the data record p, wherein, it is described to be used to store the data The metadata for recording p source packet is identical with the metadata of the data record p.

With reference to the third embodiment of first aspect, under the 5th kind of embodiment of first aspect,

The metadata of each source packet is with the data record p's in all source packets positioned at the spatial cache In the case that metadata differs, methods described also includes：

A source packet is created in the spatial cache, and it is continuous for source packet one section of physical address of distribution of the establishment Memory space；

The data record p is stored in the memory space of the source packet of the establishment, first number of the source packet of the establishment According to identical with the metadata of the data record p.

Second aspect, the embodiment of the present invention provides a kind of operation method of application program, applied to stream calculation system, the party Method includes：

Start the first streaming application positioned at the stream calculation system；

When starting the first streaming application, a result packet is created in the spatial cache of the stream calculation system, And it is result packet one section of continuous memory space of physical address of distribution of the establishment；

When starting the first streaming application, when being applied from operation first streaming, according to empty positioned at the caching Between all packets in each packet metadata, at least one packet is selected from all packets, the selection is extremely At least one packet includes the data record of more than two in a few packet, every at least one packet of the selection One packet is used as targeted packets, and the metadata of the data record in the targeted packets meets that first streaming should Treatment conditions；First number of the metadata of each packet and each each self-contained data record of packet in all packets According to identical；

All packets are preassigned one section of continuous memory space of physical address, every in all packets One packet includes the data record of one or more, the data of one or more note that each packet in all packets includes Record is stored in each respective memory space of packet, and each packet in all packets comprises at least following attributes and believed Breath：Metadata and initial address, the initial address of each packet and each respective memory space of packet in all packets The initial address of the first data record of storage is identical；

The first streaming application is run, it is every in being recorded in order to the total data included according to all targeted packets The reception time of one data record, according to the order of time from morning to night is received, all targeted packets are included successively Total data record is handled, and obtains the first result, first result includes one or more of number of results According to record, in the case where first result includes a plurality of result data record, a plurality of result data record Metadata is identical；

A result packet is created in the spatial cache of the stream calculation system, and is the result packet distribution of the establishment One section of continuous memory space of physical address；

The one or more of result data records that first result is included are stored in the knot of the establishment Fruit packet memory space, positioned at the result packet data record be from the stream calculation internal system, it is described The one or more of result data records that the metadata and first result of the result packet of establishment include Metadata is identical.

With reference to second aspect, under the first embodiment of second aspect,

Each packet also includes following attribute informations in all packets：Group character, it is different in all packets Packet there is different group characters；It is each in all packets to be grouped the group character of data record that includes and each It is identical to be grouped respective group character；

The operation the first streaming application, with since the initial address of at least one first object packet pair All data records that at least one first object packet includes are handled, and are obtained the first result, are specifically included：

The first streaming application is run, with since the initial address of at least one first object packet, to institute Have and handled with the data record with first object packet identical group character, obtain the first processing knot Fruit.

With reference to the first of second aspect or second aspect embodiment, under second of embodiment of second aspect,

All packets are source packet, and the source packet refers to that the data record of one or more in packet is equal It is from the stream calculation its exterior；Or,

All packets are result packet；Or,

All packets include source packet and result packet.

The third aspect, the embodiment of the present invention provides a kind of memory management unit, applied to stream calculation system, including：

Receiving unit, for receiving a plurality of data record from the stream calculation its exterior；

Creating unit, for creating at least two packets in the spatial cache of the stream calculation system, from the stream calculation It is that each packet at least two packet is allocated a physical address in advance and continuously stored in the memory space of system Space, each packet at least two packet include the data record of one or more, and at least two packet In at least one packet include the data record of more than two, described at least two be grouped in owning of including of each packet Data record is stored in each respective memory space of packet；

Memory cell, for being recorded in a plurality of data record to reception per data, according to below for data Record r processing method is stored：

With reference to the third aspect, under the first embodiment of the third aspect,

The memory cell is specifically used for according to each included in the packet for storing the data record r The timestamp of data record, select and receive time earliest data record, and delete the reception time earliest data note Record, the timestamp of the record per a data record the respective reception time for recording every a data；By described in Data record r is stored in the memory space for storing the reception time earliest data record.

With reference to the first of the third aspect or the third aspect embodiment, under second of embodiment of the third aspect,

The memory cell is additionally operable to add group character in the data record r, is added in the data record r Group character is identical with the group character of the packet for storing the data record r, and the data record r is stored in The memory space for being used to store the packet of the data record r.

With reference to the first embodiment of the third aspect, the third aspect or second of embodiment of the third aspect, Under the third embodiment of three aspects, in addition to：

With reference to the third embodiment of the third aspect, under the 4th kind of embodiment of the third aspect,

After per data, record stores during the memory cell is to a plurality of data record of reception, also wrap Include：

The receiving unit is additionally operable to receive at least one data record from the stream calculation its exterior；

The memory cell is additionally operable to at least one data record, according to following behaviour for data record p Stored as method：

With reference to the third embodiment of the third aspect, under the 5th kind of embodiment of the third aspect,

The metadata of each source packet is with the data record p's in all source packets positioned at the spatial cache In the case that metadata differs, in addition to：

The creating unit is additionally operable to create a source packet in the spatial cache, and is the source packet point of the establishment With one section of continuous memory space of physical address；

The memory cell is additionally operable to the data record p being stored in the memory space of the source packet of the establishment, described The metadata of the source packet of establishment is identical with the metadata of the data record p.

Fourth aspect, the embodiment of the present invention also provide a kind of running gear of application program, should applied to stream calculation system Device includes：

Starting module, for starting the first streaming application positioned at the stream calculation system；

When the starting module starts the first streaming application,

Creation module, for creating a result packet in the spatial cache of the stream calculation system, and it is the establishment Result packet distribution one section of continuous memory space of physical address；

When the starting module starts the first streaming application,

Selecting module, for from when running first streaming application, according to all packets positioned at the spatial cache In each packet metadata, at least one packet is selected from all packets, at least one packet of the selection At least one packet includes the data record of more than two, and each packet at least one packet of the selection is made For targeted packets, the metadata of the data record in the targeted packets meets the processing bar of the first streaming application Part；The metadata of each packet is identical with the metadata of each each self-contained data record of packet in all packets；

Module is run, it is complete in order to be included according to all targeted packets for running the first streaming application The reception time of each data record in portion's data record, according to the order of time from morning to night is received, successively to all described The total data record that targeted packets include is handled, and obtains the first result, and first result includes one Or a plurality of result data record, it is described a plurality of in the case where first result includes a plurality of result data record The metadata of result data record is identical；

Memory module, for the one or more of result data record deposits for including first result In the memory space of the result packet of the establishment, the data record positioned at result packet is to come from the stream calculation system System is internal, the one or more of knots that the metadata and first result that the result of the establishment is grouped include The metadata of fruit data record is identical.

With reference to fourth aspect, under the first embodiment of fourth aspect,

The operation module is specifically used for running the first streaming application, to be grouped from least one first object Initial address start, to it is all have with the first object be grouped identical group character data record handle, Obtain first result.

With reference to the first of fourth aspect or fourth aspect embodiment, under second of embodiment of fourth aspect,

All packets are result packet；Or,

All packets include source packet and result packet.

The present invention provides a kind of memory management method and device, additionally provides the operation method and dress of a kind of application program Put.It is not as prior art for a plurality of data record received in memory management method provided in an embodiment of the present invention The technical scheme of offer is such, is stored according to the time sequencing of reception in the memory space of stream calculation system, but according to connecing The metadata of the data record received, the data record is stored in the corresponding packet being pre-created, wherein, the packet is extremely Include following attribute informations less：Metadata and initial address, then after data record is received, according to the data record received Metadata and packet metadata be assured that out the data record received should be stored in which packet storage In space.Using memory management method provided in an embodiment of the present invention, the data record with same metadata is stored in same The memory space of packet, then streaming is using operationally, it is only necessary to is assured that out according to the metadata of packet positioned at flowmeter Packet where in calculation system, all qualified data records, it is not necessary to further according to streaming application treatment conditions and The metadata of data record selects qualified data record one by one, has saved prior art and has selected in stream calculation system Qualified data record time；Furthermore packet also includes attribute information：Initial address, then streaming application can be straight Connect and call data record to be handled at the initial address, avoid that streaming application is also predefined out in the packet first Data record initial address and spend the time.Understood based on above-mentioned analysis, using storage tube provided in an embodiment of the present invention Reason method, the speed of service of the streaming application in stream calculation system can be improved, improve the treatment effeciency of streaming application.

Brief description of the drawings

In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be in embodiment or description of the prior art The required accompanying drawing used is briefly described, it should be apparent that, drawings in the following description are only some realities of the present invention Example is applied, for those of ordinary skill in the art, on the premise of not paying creative work, can also be according to these accompanying drawings Obtain other accompanying drawings.

Fig. 1 is a kind of structural representation of stream calculation system provided in an embodiment of the present invention；

Fig. 2 a are a kind of schematic flow sheet of memory management method provided in an embodiment of the present invention；

Fig. 2 b are the schematic flow sheet of another memory management method provided in an embodiment of the present invention；

Fig. 2 c are the structural representation of the spatial cache of stream calculation system provided in an embodiment of the present invention；

Fig. 3 is a kind of schematic flow sheet of the operation method of application program provided in an embodiment of the present invention；

Fig. 4 is a kind of structural representation of memory management unit provided in an embodiment of the present invention；

Fig. 5 is a kind of structural representation of the running gear of application program provided in an embodiment of the present invention.

Embodiment

Below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme in the embodiment of the present invention is carried out clear, complete Site preparation describes, it is clear that described embodiment is part of the embodiment of the present invention, rather than whole embodiments.Based on this hair Embodiment in bright, the every other implementation that those of ordinary skill in the art are obtained under the premise of creative work is not made Example, belongs to the scope of protection of the invention.

Technical scheme provided by the invention is applied to stream calculation system, and the stream calculation system includes two or more circulation road, At least exist in technical scheme provided by the invention, in the two above circulation roads two such circulation roads (assuming that this two Individual circulation road be circulation road 1. with circulation road 2.), be 1. input to through circulation road stream calculation system data record metadata and The metadata that the data record of stream calculation system is 2. input to through circulation road is different.It is it will be understood by those skilled in the art that public What is known is：The a plurality of data record for being input to stream calculation system through same circulation road has identical metadata, passes through not cocurrent flow The metadata that passage is input to the data record of stream calculation system may be identical, it is also possible to different.In technology provided by the invention In scheme, the data record that stream calculation system is input to through same circulation road belongs to same packet, by different circulation roads The data record for being input to stream calculation system is likely to belong to same packet, it is also possible to belongs to different packets.

It should be noted that the metadata of data record refers to the attribute information of data record, the attribute letter of data record Breath can be one or more dimension., can be from only " sex " as when the attribute information of people is described Dimension go to limit, can also simultaneously go to limit from multiple dimensions such as " ages ", " sex " and " nationality ".Remember for a data For record, the metadata of the data record can be the keyword that the data record includes, or, the metadata of the data record For the length of the data record, or, the metadata of the data record is reception time of the data record etc..

As shown in Figure 1, it is first-class computing system, the stream calculation system is the applied field of embodiment provided by the invention Scape.Specifically, in the stream calculation system shown in Fig. 1, including a spatial cache, in addition to be deployed in the stream calculation system Multiple streaming applications, represent to be deployed in the stream calculation system with streaming application A, streaming application B and streaming application C in Fig. 1 Multiple streaming applications.

Further, as shown in Figure 1, data record is input to stream calculation system by external data source by circulation road Spatial cache.It should be appreciated that although illustrate only data in Fig. 1 is recorded in external data source and stream calculation system The transmission path of spatial cache, circulation road is not marked, but a stream calculation system generally includes multiple circulation roads, it is right Belong to common knowledge for those skilled in the art.

For any one streaming application in stream calculation system, illustrated by taking streaming application A as an example, streaming application Operationally, streaming application A can be according to the data record being located at using logical sum in the spatial cache of stream calculation system of itself by A Metadata, from a plurality of data record of the spatial cache memory storage of stream calculation system subscribe to process object, then streaming should The process object is handled with A, obtains result, and the result is published in the spatial cache again, then is existed When follow-up streaming application is run, process object of the result possible as the streaming application of follow-up operation.Due to this In the technical scheme that inventive embodiments provide, the result of streaming application output is all issued in spatial cache, so, to obtain The result that some streaming is applied is taken, it is necessary to be subscribed to from the spatial cache.

Embodiment one

As depicted in figure 2, the embodiment of the present invention provides a kind of memory management method, applied to the stream calculation system shown in Fig. 1 In system, specifically, the memory management method comprises the steps：

S201, receive a plurality of data record from the stream calculation its exterior；

It should be noted that the data record from the stream calculation its exterior can be structural data, Can be unstructured data, but the spatial cache of the stream calculation system only receives structural data, so described next From the data record of the stream calculation its exterior be unstructured data in the case of, it is necessary to first come from the flowmeter to described The data record for calculating its exterior carries out data conversion, data cleansing or the data operation such as regular, and non-structural data are converted For structural data, the data record for then translating into structural data again is transferred to the caching sky of the stream calculation system Between.

S203, the packet of spatial cache establishment at least two in the stream calculation system, from depositing for the stream calculation system Storage space in be it is described at least two packet in each packet allocate a continuous memory space of physical address in advance, it is described Each packet at least two packets includes the data record of one or more, and at least one at least two packet Individual packet includes the data record of more than two, and all data records that each packet at least two packet includes are deposited Storage is in each respective memory space of packet；

It is worth noting that, described is every at least two packet from the memory space of the stream calculation system A continuous memory space of physical address is allocated in one packet in advance, it is preferred that each packet at least two packet The size of pre-assigned memory space is depositing of being preassigned of each packet in identical, namely the stream calculation system The size for storing up space is identical.It is further preferred that the size for the memory space that each packet is preassigned is by grasping Make the digit decision of system, in general, the operating system of 32 can select 32MB memory space, the operating system of 64 64MB memory space can be selected.By taking the operating system of 64 as an example, that is to say, that each packet memory space size be 64MB*N, N are that the number of the data record included by packets inner determines and N is more than 0.Assuming that at the beginning of most, some It is grouped no any record, then its initial size is 64MB, as record number increases, progressively fills up this 64MB space, When find 64MB spaces according to it is inadequate when, at this moment can apply for a 64MB again, go to deposit new record, by that analogy.

Packet is not present in technical scheme provided in an embodiment of the present invention, the spatial cache for being directed to the stream calculation system Situation.In the case where packet is not present in the spatial cache of the stream calculation system, the stream calculation system is come from receiving During outside data record of uniting, it is necessary in creating packets for the data record of reception being placed on and being suitably grouped, tool herein It is as follows that body performs step：

At least two packets are created in the spatial cache of the stream calculation system, are allocated in advance for described at least two packets One continuous memory space of physical address；

For each packet of described at least two packets, operated according to the processing method below for packet K:

A data record j is selected from a plurality of data record from the stream calculation its exterior received, its In, data record j is typically that time earliest data record is received in a plurality of data record, or, it is a plurality of number Recorded according to a data randomly choosed out in record；

Data record j is stored in the original position of packet K memory space, the memory space memory storage as packet K The first data record, then be grouped K metadata it is identical with data record j metadata, and be grouped K initial address for count According to record j address is played in the storage of packet K memory space.

After the memory space of each packet of described at least two packets is stored with least one data record, then Data record is stored in when described at least two one be grouped are grouped the metadata, it is necessary to according to data record to be stored With the metadata of each packet of described at least two packets, if targeted packets be present at least two packet, the target The metadata of packet is identical with the metadata of data record to be stored, then data record to be stored is stored in into the target point In group；If targeted packets are not present at least two packet, need it is new create one and be grouped, and by data to be stored Record storage in the packet of the new establishment, the metadata of the packet of the new establishment and the data record to be stored Metadata is identical.

Following attribute informations are comprised at least for each packet in described at least two packets：Metadata and starting point Location.It should be noted that the metadata of each packet is used for the metadata for representing each each self-contained data record of packet, often The initial address of one packet is used for the initial memory address for representing each self-contained first data record of each packet.

S205, every data in a plurality of data record of reception is recorded, according to the place below for data record r Reason method is stored：

It is worth noting that, when every data record in a plurality of data record to reception stores, lead to It it is often the reception time of every data record in a plurality of data record, according to the time of reception time from morning to night Order is stored successively.

Further, in step S203 it is described at least two points from the memory space of the stream calculation system A continuous memory space of physical address is allocated in each packet in group in advance, it is known that, the memory space of each packet it is big It is small be allocate in advance it is good, be changeless, it is therefore desirable to control management method or space effective according to available time Property control management method, the memory space is managed.Specifically, available time control management method includes：Message has Effect survival period, until within time to live, will be labeled with out-of-date label afterwards from starting to produce the moment, not can use, Then removal, cleaning are timed.Dubious zone control management method includes：The queue size of message groups limits, when more than big After the small upper limit, the new message that adds squeezes away most ancient data, and ancient data will be unavailable, is then timed removal cleaning.At this In technical scheme described in inventive embodiments, deposited in the memory space of the packet for storing the data record r It is full, it is impossible in the case of being restored again into new data record, with reference to the conviction of stream calculation system：The stream of the value of data over time Die and reduce, can be operated by the following method：

It should be noted that each packet also includes following attribute informations at least two packet：Group character (or Person be referred to as " Group ID "), the group character of each packet is used to uniquely identify the packet, at least two packet not Same packet has different group characters.In this case, it is described that data record r deposits is described for storing institute The memory space of data record r packet is stated, is specifically included：Group character, the data note are added in the data record r The group character added is identical with the group character of the packet for storing the data record r in record r, and by the number According to the record r deposits memory space for being used to store the packet of the data record r.

It is worth noting that, the packet positioned at the spatial cache of the stream calculation system is also divided into source packet and result point Group, that is, can be the packet addition attribute information of the spatial cache positioned at the stream calculation system：Source or results, wherein, source It is the packet from stream calculation its exterior to be grouped the data record for referring to include, and as a result packet refers to the data record included It is the packet from stream calculation internal system.Be readily apparent that, created in step S203 in the spatial cache to Few two packets are source packet.

From the foregoing, it will be observed that in memory management method provided in an embodiment of the present invention, for a plurality of data record received, root The metadata recorded according to every data, the data record with same metadata is stored in same packet, wherein be grouped Metadata is identical with the metadata for the data record that the packet includes, that is, learning this point according to the metadata can of packet The metadata of the data record included in group.Using memory management method provided in an embodiment of the present invention, streaming, which is applied, to be run When, it is only necessary to it is assured that out that in stream calculation system, all qualified data are remembered according to the metadata of packet Packet where recording, it is not necessary to which the metadata for the treatment of conditions and data record further according to streaming application is selected eligible one by one Data note

Record, has saved the time that prior art selects the qualified data record in stream calculation system, so as to The speed of service of the streaming application in stream calculation system is improved, improves the treatment effeciency of streaming application.

As another embodiment of the present invention, referring to accompanying drawing 2b, in a plurality of number to reception described in step S205 After per data, record is stored in record, also comprise the steps：

S216, receive at least one data record from the stream calculation its exterior；

Easily it is worth, because step S216 is carried out after step S205, so the embodiment of the present invention provides Technical scheme, be directed to the stream calculation system spatial cache there is a situation where packet.

S217, recorded for every a data at least one data record, data record p is directed to according to following Operating method stored：

Generally, it is according to institute when each stored record in at least one data record is stored The reception time sequencing of each stored record at least one data record is stated, according to receiving time from morning to night suitable Sequence, stored one by one according to the operating method for data record p.

It is worth noting that, in all source packets positioned at the spatial cache each source packet metadata and institute State data record p metadata differ in the case of, methods described also includes：A source point is created in the spatial cache Group, and distribute one section of continuous memory space of physical address for the source packet of the establishment；By described in data record p deposits The memory space of the source packet of establishment, the metadata of the source packet of the establishment are identical with the metadata of the data record p.

It is readily apparent that, the source packet that the data record p is stored in the establishment, specifically includes：By described in Data record p is stored in the original position of the memory space of the source packet of the establishment, and the metadata of the source packet of the establishment It is identical with the metadata of the data record p.

Referring to accompanying drawing 2c, from the T1 moment to the T6 moment, 14 datas record is have received altogether, this 14 data record adheres to separately In two packets：AluHoCase and AluMocCase, wherein, AluHoCase and AluMocCase are respectively positioned on stream calculation system Spatial cache, AluHoCase include 7 datas record, AluMocCase include 4 datas record, AluHoCase include 7 Data record has identical metadata, and the 4 datas record that Al uMocCase are included has identical metadata, and The metadata for the data record that AluHoCase is included is different from the metadata for the data record that AluMocCase is included.

It should be appreciated that the source packet of the establishment also includes attribute information：Group character, the source packet of the establishment Group character and other each packet positioned at the spatial cache group character it is different.

Further, using technical scheme provided in an embodiment of the present invention, all data records received are all according to quilt Orderly is stored in corresponding packet, and each all data records included that are grouped have identical metadata, based on this hair The memory management method that bright embodiment provides, streaming application is operationally, it is only necessary to is assured that according to the metadata of packet The packet gone out where in stream calculation system, all qualified data record, it is not necessary to further according to streaming application The metadata for the treatment of conditions and data record selects qualified data record one by one, has saved the time, has been located at so as to improve The speed of service of streaming application in stream calculation system, improve the treatment effeciency of streaming application.

Furthermore the attribute information of each packet also includes each respective initial address of packet, and streaming, which is applied, to be determined After targeted packets, directly since the initial address of targeted packets, processing successively is located at every a data note in targeted packets Record.So as to which without picture, in the prior art, streaming is applied after a data record has been handled, turning in another number Handled according to the storage location of record.So using technical scheme provided in an embodiment of the present invention, streaming application is improved Treatment effeciency.

Embodiment two

As shown in Figure 3, the embodiment of the present invention provides a kind of operation method of application program, applied to the stream shown in Fig. 1 In computing system, it is notable that, can be according to application program provided in an embodiment of the present invention after the completion of embodiment one Operation method operation be located at stream calculation system in streaming application, specifically, the operation method of the application program include it is following Step：

S301, start the first streaming application for being located at the stream calculation system；

Generally, the spatial cache of the stream calculation system includes a plurality of streaming application, then a plurality of streaming application can be with one The startup optimization of individual one, can also more than two streamings application startup optimization simultaneously, start fortune regardless of specific Line mode, it can be operated according to the operation method of application program provided in an embodiment of the present invention.

S302, when starting first streaming application, create a structure packet, and be grouped for the result of the establishment Distribute one section of continuous memory space of physical address；

S303, when starting first streaming application, from when running the first streaming application, according to positioned at described slow The metadata of each packet in all packets in space is deposited, at least one packet, the selection are selected from all packets At least one packet at least one packet include the data record of more than two, at least one packet of the selection Each packet be used as targeted packets, the metadata of the data record in the targeted packets meets described first-class The treatment conditions of formula application；The metadata of each packet and each each self-contained data record of packet in all packets Metadata is identical；

It should be noted that it is the first streaming application selection target point to start the first streaming using and from spatial cache , it is necessary to according to the metadata of each packet in all packets of the spatial cache during group, selected from all packets At least one packet is selected, each packet at least one packet of the selection is used as targeted packets.In the caching In the case that the metadata of all packet neither ones packet in space meets the treatment conditions of first streaming application, then institute The application of the first streaming is stated to stop starting.

S305, operation the first streaming application, in order to the total data note included according to all targeted packets The reception time of each data record in record, according to the order of time from morning to night is received, successively to all targeted packets Including total data record handled, obtain the first result, first result includes one or more of Result data records, in the case where first result includes a plurality of result data record, a plurality of result data The metadata of record is identical；

It is worth noting that, in technical scheme provided in an embodiment of the present invention, positioned at all of the stream calculation system Data record includes following attribute informations：Timestamp, the reception time for record data record.Then run described first-class Formula application, the reception time of each data record in being recorded in order to the total data included according to all targeted packets, According to the order of time from morning to night is received, the total data record included successively to all targeted packets is handled, The first result is obtained, is specifically included：The first streaming application is run, in order to include according to all targeted packets Total data record in each data record timestamp, according to time order from morning to night is received, successively to all institutes The total data record that stating targeted packets includes is handled, and obtains the first result.

Recorded it should be noted that first result may also not include any result data, then the establishment Result packet include result data record number be 0.

S307, the one or more of result data records for including first result are stored in the establishment Result packet memory space, positioned at the result packet data record be from the stream calculation internal system, The one or more of result data notes that the metadata and first result of the result packet of the establishment include The metadata of record is identical.

It should be noted that the result of a streaming application is stored separately on result packet corresponding to streaming application Memory space in, be not in two different streamings applications result be stored in same result packet storage it is empty Between situation.

From the foregoing, it will be observed that the operation method of application program provided in an embodiment of the present invention, when starting streaming application, from positioned at At least one packet is selected in all packets in stream calculation system cache space, due at least one at least one packet Packet includes the data record of more than two, and each packet at least one packet of the selection is targeted packets, institute There are the treatment conditions that the data record in targeted packets is satisfied by streaming application.For prior art, the stream Formula applies the metadata not having to according to every data record in all data records of spatial cache, remembers from all data In record one by one select the data record for meeting the streaming application treatment conditions, so as to improve the streaming in stream calculation system The speed of service of application, improve the treatment effeciency of streaming application.

Embodiment three

As shown in Figure 4, it is a kind of memory management unit 40 provided in an embodiment of the present invention, applied to the stream shown in Fig. 1 Computing system, the memory management unit 40 described in the embodiment of the present invention can perform the memory management method described in embodiment one. Specifically, as shown in figure 4, memory management unit 40 includes following components：

Receiving unit 41, for receiving a plurality of data record from the stream calculation its exterior；

Creating unit 43, for creating at least two packets in the spatial cache of the stream calculation system, from the flowmeter It is that each packet at least two packet is allocated a physical address in advance and continuously deposited in the memory space of calculation system Space is stored up, each packet at least two packet includes the data record of one or more, and described at least two points At least one packet includes the data record of more than two, the institute that each packet at least two packet includes in group There is data record to be stored in each respective memory space of packet；

Packet is not present in technical scheme provided in an embodiment of the present invention, the spatial cache for being directed to the stream calculation system Situation.In the case where packet is not present in the spatial cache of the stream calculation system, the stream calculation system is come from receiving During outside data record of uniting, it is necessary in creating packets for the data record of reception being placed on and being suitably grouped.

It should be noted that at least two packet is source packet, the data record positioned at the source packet is From the stream calculation its exterior.

Memory cell 45, for being recorded in a plurality of data record to reception per data, according to below for number Stored according to record r processing method：

It is worth noting that, every data record of the memory cell 45 in a plurality of data record to reception is carried out During storage, the reception time of every data record typically in a plurality of data record, according to reception time from morning Time sequencing to evening is stored successively.

It should be noted that the memory space in the packet for storing the data record r has been filled with, it is impossible to again In the case of being stored in the data record r,

Memory cell 45 is specifically used for according to each number included in the packet for storing the data record r According to the timestamp of record, select and receive time earliest data record, and delete the reception time earliest data record, The timestamp of the record per a data records the respective reception time for recording every a data；By the data Record r is stored in the memory space for storing the reception time earliest data record.

It is worth noting that, each packet also includes following attribute informations at least two packet：Group character, institute Stating packet different at least two packets has different group characters；

Memory cell 45 is additionally operable to add group character in the data record r, point added in the data record r Group mark is identical with the group character of the packet for storing the data record r, and the data record r is stored in into institute State the memory space of the packet for storing the data record r.

As an alternative embodiment of the invention, every number in memory cell 45 is to a plurality of data record of reception After being stored according to record, in addition to：

Receiving unit 41 is additionally operable to receive at least one data record from the stream calculation its exterior；

It should be appreciated that technical scheme provided in an embodiment of the present invention, the caching of the stream calculation system is directed to Packet be present in space.

Memory cell 45 is additionally operable to at least one data record, according to following operations for data record p Method is stored：

Generally, each stored record of the memory cell 45 in at least one data record is stored When, be each stored record at least one data record reception time sequencing, according to receive the time from The early order to evening, stored one by one according to the operating method for data record p.

It is worth noting that, in all source packets positioned at the spatial cache each source packet metadata and institute State data record p metadata differ in the case of, in addition to：

Creating unit 43 is additionally operable to create a source packet in the spatial cache, and is distributed for the source packet of the establishment One section of continuous memory space of physical address；

Memory cell 45 is additionally operable to the data record p being stored in the memory space of the source packet of the establishment, the wound The metadata for the source packet built is identical with the metadata of the data record p.

It is readily apparent that, memory cell 45 is specifically used for the source packet of the data record p deposits establishment The original position of memory space, and the metadata of the source packet of the establishment is identical with the metadata of the data record p.

From the foregoing, it will be observed that in memory management unit provided in an embodiment of the present invention, for a plurality of data record received, root The metadata recorded according to every data, the data record with same metadata is stored in same packet, wherein be grouped Metadata is identical with the metadata for the data record that the packet includes, that is, learning this point according to the metadata can of packet The metadata of the data record included in group.Using memory management unit provided in an embodiment of the present invention, streaming, which is applied, to be run When, it is only necessary to it is assured that out that in stream calculation system, all qualified data are remembered according to the metadata of packet Packet where recording, it is not necessary to which the metadata for the treatment of conditions and data record further according to streaming application is selected eligible one by one Data record, saved the time that prior art selects the qualified data record in stream calculation system, so as to The speed of service of the streaming application in stream calculation system is improved, improves the treatment effeciency of streaming application.

Example IV

As shown in Figure 5, it is a kind of running gear 50 of application program provided in an embodiment of the present invention, applied to Fig. 1 institutes The stream calculation system shown, the running gear 50 of the application program in the embodiment can perform the application program described in embodiment two Operation method.Specifically, referring to accompanying drawing 5, the running gear 50 of application program includes：

Starting module 51, for starting the first streaming application positioned at the stream calculation system；

When starting module 51 starts the first streaming application,

Creation module 52, for creating a result packet in the spatial cache of the stream calculation system, and it is the wound The result packet one section of continuous memory space of physical address of distribution built；

When starting module 51 starts the first streaming application,

Selecting module 53, for from when running first streaming application, according to positioned at all points of the spatial cache The metadata of each packet in group, at least one packet, at least one packet of the selection are selected from all packets In at least one packet include the data record of more than two, each at least one packet of the selection is grouped As targeted packets, the metadata of the data record in the targeted packets meets the processing bar of the first streaming application Part；The metadata of each packet is identical with the metadata of each each self-contained data record of packet in all packets；

Module 55 is run, for running the first streaming application, in order to what is included according to all targeted packets The reception time of each data record in total data record, according to the order of time from morning to night is received, successively to all institutes The total data record that stating targeted packets includes is handled, and obtains the first result, and first result includes one Bar or a plurality of result data record are described more in the case where first result includes a plurality of result data record The metadata of bar result data record is identical；

It should be noted that in technical scheme provided in an embodiment of the present invention, positioned at all of the stream calculation system Data record includes following attribute informations：Timestamp, the reception time for record data record.It is specific then to run module 55 For running the first streaming application, each number in being recorded in order to the total data included according to all targeted packets According to the timestamp of record, according to the order of reception time from morning to night, the whole numbers included successively to all targeted packets Handled according to record, obtain the first result.It is worth noting that, each packet is also including following in all packets Attribute information：Group character, in all packets different packets there is different group characters；It is every in all packets The group character for the data record that one packet includes is identical with each respective group character of packet；

Memory module 56, the one or more of result data records for first result to be included are deposited In the memory space of result packet for entering the establishment, the data record positioned at result packet is to come from the stream calculation Internal system, the metadata and first result of the packet of the result of the establishment include described one or more of The metadata of result data record is identical.

It should be noted that it is described it is all packet be source packet, the source packet refer to positioned at packet in one with On data record be from the stream calculation its exterior；Or,

All packets are result packet；Or, all packets include source packet and result packet.

From the foregoing, it will be observed that the running gear of application program provided in an embodiment of the present invention, starting module should for starting streaming With at least one packet being selected from all packets positioned at stream calculation system cache space, due at least one packet At least one packet includes the data record of more than two, and each packet at least one packet of the selection is mesh Mark packet, all data records in targeted packets are satisfied by the treatment conditions of streaming application.Relative to prior art For, starting module is when starting streaming application, without according to every data in all data records of spatial cache The metadata of record, from all data records one by one select the data record for meeting the streaming application treatment conditions, from And the speed of service of the streaming application in stream calculation system is improved, improve the treatment effeciency of streaming application.

Can be with cross-reference wherein between above-described embodiment.

The unit illustrated as separating component can be or may not be physically separate, show as unit The part shown can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple On NE.Some or all of unit therein can be selected to realize the mesh of this embodiment scheme according to the actual needs 's.

In addition, each functional unit in the network equipment that each embodiment of the present invention provides can be integrated in a processing In unit or unit is individually physically present, can also two or more units it is integrated in a unit. Above-mentioned integrated unit can both be realized in the form of hardware, can also be realized in the form of SFU software functional unit.

If the integrated unit is realized in the form of SFU software functional unit and is used as independent production marketing or use When, it can be stored in a computer read/write memory medium.Based on such understanding, technical scheme is substantially The part to be contributed in other words to prior art or all or part of the technical scheme can be in the form of software products Embody, the computer software product is stored in a storage medium, including some instructions are causing a computer Equipment (can be personal computer, server, or network equipment etc.) performs the complete of each embodiment methods described of the present invention Portion or part steps.And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disc or CD etc. are various can store journey The medium of sequence code.

Those of ordinary skill in the art are it is to be appreciated that the list of each example described with reference to the embodiments described herein Member and algorithm steps, it can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually Performed with hardware or software mode, application-specific and design constraint depending on technical scheme.Professional and technical personnel Described function can be realized using distinct methods to each specific application, but this realization is it is not considered that exceed The scope of the present invention.

Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment Divide mutually referring to what each embodiment stressed is the difference with other embodiment.It is real especially for system For applying example, because it is substantially similar to embodiment of the method, so description is fairly simple, related part is referring to embodiment of the method Part explanation.

The embodiments of the present invention described above are not intended to limit the scope of the present invention.It is any in the present invention Spirit and principle within the modifications, equivalent substitutions and improvements made etc., should be included in the scope of the protection.

Claims

1. a kind of memory management method, applied to stream calculation system, the stream calculation system includes at least one streaming application, its It is characterised by, methods described includes：

Receive a plurality of data record from the stream calculation its exterior；

At least two packets are created in the spatial cache of the stream calculation system, are from the memory space of the stream calculation system A continuous memory space of physical address is allocated in each packet at least two packet in advance, described at least two points Each packet in group includes the data record of one or more, and at least one packet includes at least two packet The data record of more than two, all data records that each packet at least two packet includes are stored in each point The respective memory space of group；

Each packet comprises at least following attribute informations at least two packet：Metadata and initial address；It is described extremely The metadata of each packet is identical with the metadata of each each self-contained data record of packet in few two packets；It is described at least What the initial address of each packet and the first data of the respective memory space storage of each packet recorded in two packets rises Beginning, address was identical；

To being recorded in a plurality of data record of reception per data, enter according to the processing method below for data record r Row storage：

According to the metadata of the data record r, selected from described at least two packets for storing the data record r's Packet, the metadata of packet for being used to store the data record r and the metadata of the data record r are identicals；

And the data record r is stored in the memory space for being used to store the packet of the data record r；

Methods described also includes：In each streaming application in starting at least one streaming application, corresponding to establishment One result packet, and be result packet one section of continuous memory space of physical address of distribution of the establishment.

2. according to the method for claim 1, it is characterised in that

It has been filled with the memory space of the packet for storing the data record r, it is impossible to be restored again into the data record r In the case of,

It is described, for the memory space for the packet for storing the data record r, specifically to be wrapped described in data record r deposits Include：

The timestamp that every a data according to being included in the packet for storing the data record r records, is selected Time earliest data record is received, and deletes the reception time earliest data record, the record per a data Timestamp records the respective reception time for recording every a data；The data record r is stored in for storing institute State the memory space for receiving time earliest data record.

3. according to the method for claim 1, it is characterised in that

Each packet also includes following attribute informations at least two packet：Group character, at least two packet Different packets have different group characters；

Group character is added in the data record r, the group character added in the data record r is used to store with described The group character of the packet of the data record r is identical, and data record r deposits is described for storing the data note Record the memory space of r packet.

4. according to the method described in any one of claims 1 to 3, it is characterised in that methods described also includes：

At least two packet is source packet, and the data record positioned at the source packet is to come from the stream calculation system Outside.

5. according to the method for claim 4, it is characterised in that every number in a plurality of data record of described pair of reception After being stored according to record, methods described also includes：

Receive at least one data record from the stream calculation its exterior；

Recorded for every a data at least one data record, according to following operation sides for data record p Method is stored：

When the source packet for storing the data record p in all source packets of the spatial cache be present, by described in The data record p deposits source packet for being used to store the data record p, wherein, it is described to be used to store the data record p Source packet metadata it is identical with the metadata of the data record p.

6. according to the method for claim 4, it is characterised in that

First number of the metadata of each source packet with the data record p in all source packets positioned at the spatial cache In the case of differing, methods described also includes：

A source packet is created in the spatial cache, and is continuously deposited for source packet one section of physical address of distribution of the establishment Store up space；

The data record p is stored in the memory space of the source packet of the establishment, the metadata of the source packet of the establishment and The metadata of the data record p is identical.

A kind of 7. operation method of application program, applied to stream calculation system, it is characterised in that methods described includes：

When starting the first streaming application, a result packet is created in the spatial cache of the stream calculation system, and be Result packet one section of continuous memory space of physical address of distribution of the establishment；

When starting the first streaming application, when being applied from operation first streaming, according to positioned at the spatial cache The metadata of each packet in all packets, selects at least one packet from all packets, and at least the one of the selection At least one packet includes the data record of more than two in individual packet, each at least one packet of the selection Packet is used as targeted packets, and the metadata of the data record in the targeted packets meets the first streaming application Treatment conditions；The metadata phase of the metadata of each packet and each each self-contained data record of packet in all packets Together；

All be grouped is preassigned one section of continuous memory space of physical address, each point in all packets Group includes the data record of one or more, the data record quilt of one or more that each packet in all packets includes Each respective memory space of packet is stored in, each packet in all packets comprises at least following attribute informations： Metadata and initial address, the initial address of each packet stores with each respective memory space of packet in all packets The first data record initial address it is identical；

The first streaming application is run, each number in being recorded in order to the total data included according to all targeted packets According to the reception time of record, according to the order of reception time from morning to night, the whole included successively to all targeted packets Data record is handled, and obtains the first result, and first result is remembered comprising one or more of result datas Record, in the case where first result includes a plurality of result data record, first number of a plurality of result data record According to being identical；

The one or more of result data records that first result is included are stored in the result point of the establishment The memory space of group, the data record positioned at result packet is from the stream calculation internal system, the establishment Result packet metadata and first number of the one or more of result datas record for including of first result According to being identical.

8. according to the method for claim 7, it is characterised in that

Each packet also includes following attribute informations in all packets：Group character, different points in all packets Group has different group characters；Each group character for being grouped the data record included and each packet in all packets Respective group character is identical；

The operation the first streaming application, with the initial address that is grouped since at least one first object at least one All data records that the first object packet includes are handled, and are obtained the first result, are specifically included：

The first streaming application is run, with since the initial address of at least one first object packet, to all tools There is the data record with first object packet identical group character to be handled, obtain first result.

9. the method according to claim 7 or 8, it is characterised in that also include：

All packets are source packet, and the source packet refers to that the data record of one or more in packet is come From the stream calculation its exterior；Or,

All packets are result packet；Or,

All packets include source packet and result packet.

10. a kind of memory management unit, applied to stream calculation system, the stream calculation system includes at least one streaming application, It is characterised in that it includes：

Creating unit, for creating at least two packets in the spatial cache of the stream calculation system, from the stream calculation system Memory space in be it is described at least two packet in each packet allocate a continuous memory space of physical address in advance, Each packet at least two packet includes the data record of one or more, and at least two packet at least There are the data record that a packet includes more than two, all data note that each packet at least two packet includes Address book stored is in each respective memory space of packet；

Memory cell, for being recorded in a plurality of data record to reception per data, according to below for data record r Processing method stored：

The creating unit is additionally operable to：In each streaming application in starting at least one streaming application, establishment pair The result packet answered, and be result packet one section of continuous memory space of physical address of distribution of the establishment.

11. device according to claim 10, it is characterised in that

The memory cell is specifically used for according to the every a data included in the packet for storing the data record r The timestamp of record, select and receive time earliest data record, and delete the reception time earliest data record, institute The timestamp for stating every a data record records the respective reception time for recording every a data；The data are remembered Record r is stored in the memory space for storing the reception time earliest data record.

12. the device according to claim 10 or 11, it is characterised in that

The memory cell is additionally operable to add group character in the data record r, the packet added in the data record r Mark is identical with the group character of the packet for storing the data record r, and by described in data record r deposits For the memory space for the packet for storing the data record r.

13. the device according to claim 10 or 11, it is characterised in that also include：

14. device according to claim 13, it is characterised in that

After per data, record stores during the memory cell is to a plurality of data record of reception, in addition to：

The memory cell is additionally operable to at least one data record, according to following operation sides for data record p Method is stored：

15. device according to claim 13, it is characterised in that

First number of the metadata of each source packet with the data record p in all source packets positioned at the spatial cache In the case of differing, in addition to：

The creating unit is additionally operable to create a source packet in the spatial cache, and is the source packet distribution one of the establishment The section continuous memory space of physical address；

The memory cell is additionally operable to the data record p being stored in the memory space of the source packet of the establishment, the establishment Source packet metadata it is identical with the metadata of the data record p.

A kind of 16. running gear of application program, applied to stream calculation system, it is characterised in that described device includes：

When the starting module starts the first streaming application,

Creation module, for creating a result packet in the spatial cache of the stream calculation system, and it is the knot of the establishment Fruit packet one section of continuous memory space of physical address of distribution；

When the starting module starts the first streaming application,

Selecting module, for from when running first streaming application, according to every in all packets of the spatial cache The metadata of one packet, selects at least one packet, at least one packet of the selection at least from all packets There is one to be grouped the data record for including more than two, each packet at least one packet of the selection is used as mesh Mark packet, the metadata of the data record in the targeted packets meet the treatment conditions of the first streaming application；Institute It is identical with the metadata of each each self-contained data record of packet to state the metadata of each packet in all packets；

Module is run, for running the first streaming application, in order to the whole numbers included according to all targeted packets According to the reception time of each data record in record, according to the order of time from morning to night is received, successively to all targets The total data record that being grouped includes is handled, and obtains the first result, first result include one or A plurality of result data record, in the case where first result includes a plurality of result data record, a plurality of result The metadata of data record is identical；

Memory module, the one or more of result datas for first result to be included are recorded described in deposit In the memory space of the result packet of establishment, the data record positioned at result packet is in the stream calculation system Portion, the one or more of number of results that the metadata and first result of the packet of the result of the establishment include Metadata according to record is identical.

17. device according to claim 16, it is characterised in that

The operation module is specifically used for running the first streaming application, with from the starting point that at least one first object is grouped Location starts, and to all there is the data record for being grouped identical group character with the first object to handle, obtains described First result.

18. the device according to claim 16 or 17, it is characterised in that also include：

All packets are result packet；Or,

All packets include source packet and result packet.