CN107423316B - A kind of big data processing method, apparatus and system - Google Patents

A kind of big data processing method, apparatus and system Download PDF

Info

Publication number
CN107423316B
CN107423316B CN201710178126.1A CN201710178126A CN107423316B CN 107423316 B CN107423316 B CN 107423316B CN 201710178126 A CN201710178126 A CN 201710178126A CN 107423316 B CN107423316 B CN 107423316B
Authority
CN
China
Prior art keywords
data
user equipment
parsing
end
corresponding
Prior art date
Application number
CN201710178126.1A
Other languages
Chinese (zh)
Other versions
CN107423316A (en
Inventor
林勤鑫
Original Assignee
珠海格力电器股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 珠海格力电器股份有限公司 filed Critical 珠海格力电器股份有限公司
Priority to CN201710178126.1A priority Critical patent/CN107423316B/en
Publication of CN107423316A publication Critical patent/CN107423316A/en
Application granted granted Critical
Publication of CN107423316B publication Critical patent/CN107423316B/en

Links

Abstract

The present invention relates to data analysis technique field, in particular to a kind of big data processing method, apparatus and system.This method are as follows: according to each corresponding number of user equipment in parsing end, determine the corresponding least parsing end of number of user equipment, and initial data is sent to the parsing end, and trigger the parsing end and execute following operation: data conversion treatment is carried out to the initial data received, obtain corresponding set state data, and by set state data publication into Kafka cluster, so that user obtains the set state data subscribed to from Kafka cluster.Using the above method, realize the dynamic load leveling between each parsing end, further, user can get the corresponding set state data of each user equipment in real time, and according to the corresponding set state data of each user equipment, detect that each user equipment with the presence or absence of the user equipment being operating abnormally, to reduce the failure rate of user equipment, improves user experience in advance.

Description

A kind of big data processing method, apparatus and system

Technical field

The present invention relates to data analysis technique field, in particular to a kind of big data processing method, apparatus and system.

Background technique

It is more prevalent with the Network Information epoch, with the arriving of big data era, the application field of big data It is more and more common, e.g., in the application of field of air conditioning, specifically, handling the big data of concurrent air-conditioning equipment in real time.

Currently, data parse end to each air-conditioning equipment received when handling the big data of air-conditioning equipment of high concurrent The initial data of feedback carries out dissection process, and (e.g., the data after dissection process are write direct corresponding first database SQL Server) in, and according to preset rules, be written in first database within period of the preset time point by setting Data conversion storage into the second database.

However, in the prior art, due to being service competition relationship between each data parsing end, in this way it is possible to lead to certain The number of the corresponding air-conditioning equipment in a little data parsings end is more, and the number of the corresponding air-conditioning equipment in certain data parsing end is few, from And lead to data parsing end parsing load imbalance.

Further, it in the case where constantly mass data is written in first database, is if desired read from first database Access evidence, then desired data can just be read by needing to take a long time, even due to being frequently written, reading data Operation, cause first database performance to decline, and then first database caused to be paralysed.Therefore, in the prior art, counted It is investigated that data can only be read out from the second database when asking, and do not stored in the second database air-conditioned real-time Data, in this way, will lead to the poor in timeliness of data inquired, thus cannot before Air conditioning equipment failure generation, according to The real time data of air-conditioning equipment detects the abnormal conditions of air-conditioning equipment in advance, to carry out respective handling in time.

In conclusion needing to design the new big data processing method of one kind, apparatus and system, deposited in the prior art with making up Defect and shortcoming.

Summary of the invention

The embodiment of the present invention provides a kind of big data processing method, apparatus and system, exists in the prior art to solve Each data parse end load imbalance, and the poor in timeliness of the data due to inquiring, so as to cause that cannot set in air-conditioning Before standby failure occurs, according to the real time data of air-conditioning equipment, detect the abnormal conditions of air-conditioning equipment in advance, so as in time into The problem of row respective handling.

Specific technical solution provided in an embodiment of the present invention is as follows:

A kind of big data processing method, comprising:

Corresponding data queue is arranged when having monitored user equipment access, for the user equipment in server-side, and The initial data that the user equipment uploads is stored in the data queue;

Server-side determines each corresponding number of user equipment in parsing end respectively, and determines corresponding user device quantity The least parsing end of mesh;

The initial data is sent to the least parsing end of the corresponding number of user equipment by server-side, and triggers institute It states the corresponding least parsing end of number of user equipment and executes following operation: the initial data received is carried out at data conversion Reason, obtains corresponding engineering equipment data and set state data, and by the set state data publication to Kafka cluster In, so that user obtains the set state data that the user has subscribed to from the Kafka cluster.

Optionally, server-side further comprises before having monitored user equipment access:

Server-side is monitoring that parsing termination is fashionable, creates one for protecting according to the physical address information at the parsing end The data list of the parsing corresponding number of user equipment in end and each customer equipment identification information is deposited, and is the parsing end A counter is distributed, and based on the corresponding number of user equipment in parsing end described in the counter records.

Optionally, the data queue is a kind of linear list of the data structure of fifo fifo.

A kind of big data processing method, comprising:

It parses end and receives the initial data that the user equipment that server-side is sent uploads, wherein the parsing end is the clothes Business end parses what the corresponding number of user equipment in end was determined according to each, the corresponding least parsing of number of users End;

Parsing end carries out data conversion treatment to the initial data received, obtains corresponding engineering equipment data and unit Status data;

End is parsed by the set state data publication into Kafka cluster, so that user is real from the Kafka cluster When obtain the set state data subscribed to.

Optionally, parsing end further comprises after obtaining corresponding engineering equipment data and set state data:

Parsing end judges in memory database with the presence or absence of the engineering equipment data;

If parsing end determines that there are the engineering equipment data in memory database, it is determined that created in memory database The engineering of the corresponding user equipment of the engineering equipment data.

Optionally, further comprise:

If parsing end determine memory database in be not present the engineering equipment data, judge be in relevant database It is no that there are the engineering equipment data;

If parsing end determines that there are the engineering equipment data in the relevant database, it is determined that the relationship type number According to the engineering of the corresponding user equipment of the engineering equipment data existing in library, and by the corresponding use of the engineering equipment information The engineering of family equipment is added to memory database;

If parsing end to determine that the engineering equipment data are not present in the relevant database, creates the engineering and set The engineering of the standby corresponding user equipment of data, and the engineering of the corresponding user equipment of the engineering equipment data is added to memory In database.

Optionally, parsing end by the set state data publication into Kafka cluster, so that user is from the Kafka When obtaining the set state data subscribed in cluster in real time, specifically include:

It parses end and the set state data is subjected to classification processing according to different themes;

End is parsed using the set state data of the different themes as news release to Kafka cluster, to have subscribed to The user of the set state data of the different themes obtains the unit shape of the different themes in real time from the Kafka cluster State data.

A kind of big data processing unit, comprising:

Setting unit, for corresponding data to be arranged for the user equipment when having monitored user equipment access Queue, and the initial data that the user equipment uploads is stored in the data queue;

Determination unit for determining each corresponding number of user equipment in parsing end respectively, and determines corresponding use The least parsing end of family number of devices;

Transmission unit, for the initial data to be sent to the least parsing end of the corresponding number of user equipment, And trigger the least parsing end of the corresponding number of user equipment and execute following operation: the initial data received is counted According to conversion process, corresponding engineering equipment data and set state data are obtained, and extremely by the set state data publication In Kafka cluster, so that user obtains the set state data that the user has subscribed to from the Kafka cluster.

Optionally, before having monitored user equipment access, the setting unit is further used for:

Monitoring that parsing termination is fashionable, it is described for saving according to physical address information creation one of the parsing end The data list of the corresponding number of user equipment in end and each customer equipment identification information is parsed, and is parsing end distribution one A counter, and based on the corresponding number of user equipment in parsing end described in the counter records.

Optionally, the data queue is a kind of linear list of the data structure of fifo fifo.

A kind of big data processing unit, comprising:

Receiving unit, the initial data that the user equipment for receiving server-side transmission uploads, wherein at the big data The reason device server-side is determined according to each corresponding number of user equipment of big data processing unit, corresponding The least big data processing unit of number of users;

Processing unit obtains corresponding engineering equipment number for carrying out data conversion treatment to the initial data received According to set state data;

Release unit, for by the set state data publication into Kafka cluster, so that user is from the Kafka The set state data subscribed to are obtained in cluster in real time.

Optionally, after obtaining corresponding engineering equipment data and set state data, the release unit is further For:

Judge in memory database with the presence or absence of the engineering equipment data;

If it is determined that there are the engineering equipment data in memory database, it is determined that created the work in memory database The engineering of the corresponding user equipment of journey device data.

Optionally, the release unit is further used for:

If it is determined that the engineering equipment data are not present in memory database, then judge to whether there is in relevant database The engineering equipment data;

If it is determined that there are the engineering equipment data in the relevant database, it is determined that in the relevant database The engineering of the corresponding user equipment of existing engineering equipment data, and by the corresponding user equipment of the engineering equipment information Engineering be added to memory database;

If it is determined that the engineering equipment data are not present in the relevant database, then the engineering equipment data are created The engineering of corresponding user equipment, and the engineering of the corresponding user equipment of the engineering equipment data is added to memory database In.

Optionally, by the set state data publication into Kafka cluster, so that user is from the Kafka cluster When obtaining the set state data subscribed in real time, the release unit is used for:

The set state data are subjected to classification processing according to different themes;

Using the set state data of the different themes as news release to Kafka cluster, so as to subscribed to it is described not Obtain the set state number of the different themes in real time from the Kafka cluster with the user of the set state data of theme According to.

A kind of big data processing system includes at least server-side, several parsing ends and several user equipmenies, wherein

Server-side, for corresponding data team to be arranged for the user equipment when having monitored user equipment access Column, and the initial data that the user equipment uploads is stored in the data queue, each parsing end pair is determined respectively The number of user equipment answered determines the corresponding least parsing end of number of user equipment, and the initial data is sent to The least parsing end of corresponding number of user equipment is triggered the least parsing end of corresponding number of user equipment and is executed It operates below: data conversion treatment being carried out to the initial data received, obtains corresponding engineering equipment data and set state Data, and by the set state data publication into Kafka cluster, obtained from the Kafka cluster so as to user described in The set state data that user has subscribed to;

End is parsed, the initial data that the user equipment for receiving server-side transmission uploads, wherein the parsing end is institute State what server-side was determined according to each corresponding number of user equipment in parsing end, the corresponding least solution of number of users End is analysed, data conversion treatment is carried out to the initial data received, obtains corresponding engineering equipment data and set state data, By the set state data publication into Kafka cluster, subscribed to so that user obtains from the Kafka cluster in real time Set state data.

The present invention has the beneficial effect that:

In conclusion during carrying out big data processing, server-side is monitoring user in the embodiment of the present invention When equipment accesses, corresponding data queue, and the initial data that above-mentioned user equipment is uploaded are set for above-mentioned user equipment It is stored in above-mentioned data queue, determines each corresponding number of user equipment in parsing end respectively, and determine corresponding use Above-mentioned initial data is sent to the above-mentioned least parsing of corresponding number of user equipment by the least parsing end of family number of devices End, and trigger the least parsing end of above-mentioned corresponding number of user equipment and execute following operate: to the initial data received into Row data conversion treatment, obtains corresponding engineering equipment data and set state data, and by above-mentioned set state data publication Into Kafka cluster, so that user obtains the set state data that above-mentioned user has subscribed to from above-mentioned Kafka cluster.

Using above-mentioned big data processing method, the dynamic load leveling between each parsing end, further, user are realized The corresponding set state data of each user equipment can also be got in real time, ensure that the timeliness of set state data, And according to the corresponding set state data of each user equipment, detected in each user equipment in advance with the presence or absence of operation exception User equipment improve user equipment after-sale service quality, and then improve user's body to reduce the failure rate of user equipment It tests.

Detailed description of the invention

Fig. 1 is a kind of system architecture figure of big data processing system in the embodiment of the present invention;

Fig. 2 is the detail flowchart of the first big data processing method in the embodiment of the present invention;

Fig. 3 is the detail flowchart of second of big data processing method in the embodiment of the present invention;

Fig. 4 is the structural schematic diagram of the first big data processing unit in the embodiment of the present invention;

Fig. 5 is the structural schematic diagram of second of big data processing unit in the embodiment of the present invention.

Specific embodiment

In order to solve each data parsing end load imbalance existing in the prior art, and the data due to inquiring Poor in timeliness, so as to cause that, according to the real time data of air-conditioning equipment, cannot be detected in advance before Air conditioning equipment failure generation The abnormal conditions of air-conditioning equipment the problem of to carry out respective handling in time, provide a kind of new big in the embodiment of the present invention Data processing method, apparatus and system, this method are as follows: server-side is when having monitored user equipment access, for above-mentioned user Corresponding data queue is arranged in equipment, and the initial data that above-mentioned user equipment uploads is stored in above-mentioned data queue, point Each corresponding number of user equipment in parsing end is not determined, and determines the corresponding least parsing end of number of user equipment, Above-mentioned initial data is sent to the least parsing end of above-mentioned corresponding number of user equipment, and triggers above-mentioned corresponding user and sets The standby least parsing end of number executes following operation: carrying out data conversion treatment to the initial data received, obtains corresponding Engineering equipment data and set state data, and by above-mentioned set state data publication into Kafka cluster, so that user is from upper It states and obtains the set state data that above-mentioned user has subscribed in Kafka cluster.

Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, is not whole embodiments.It is based on Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other Embodiment shall fall within the protection scope of the present invention.

The solution of the present invention will be described in detail by specific embodiment below, certainly, the present invention is not limited to Lower embodiment.

As shown in fig.1, devising a kind of big data processing system in the embodiment of the present invention, server-side is included at least, if Dry parsing end and several user equipmenies, wherein

Server-side, for corresponding data team to be arranged for above-mentioned user equipment when having monitored user equipment access Column, and the initial data that above-mentioned user equipment uploads is stored in above-mentioned data queue, each parsing end pair is determined respectively The number of user equipment answered determines the corresponding least parsing end of number of user equipment, and above-mentioned initial data is sent to The above-mentioned least parsing end of corresponding number of user equipment is triggered the above-mentioned least parsing end of corresponding number of user equipment and is executed It operates below: data conversion treatment being carried out to the initial data received, obtains corresponding engineering equipment data and set state Data, and by above-mentioned set state data publication into Kafka cluster, so as to user obtained from above-mentioned Kafka cluster it is above-mentioned The set state data that user has subscribed to.

End is parsed, the initial data that the user equipment for receiving server-side transmission uploads, wherein above-mentioned parsing end is upper State what server-side was determined according to each corresponding number of user equipment in parsing end, the corresponding least solution of number of users End is analysed, data conversion treatment is carried out to the initial data received, obtains corresponding engineering equipment data and set state data, By above-mentioned set state data publication into Kafka cluster, subscribed to so that user obtains from above-mentioned Kafka cluster in real time Set state data.

Specifically, as shown in fig.2, a kind of detailed process of big data processing method is as follows in the embodiment of the present invention:

Step 200: server-side is monitoring that parsing termination is fashionable, according to the physical address information at above-mentioned parsing end creation one A data list for being used to save the corresponding number of user equipment in above-mentioned parsing end and each customer equipment identification information, and be upper It states parsing end and distributes a counter, and be based on the corresponding number of user equipment in the above-mentioned parsing end of above-mentioned counter records.

Specifically, the company between server-side itself and each parsing end can be monitored in real time in server-side when executing step 200 State is connect, has new parsing termination fashionable if server-side determines, obtains the MAC code at the new parsing end, and according to getting The MAC code at new parsing end create a data list, wherein the data list is used to save the new parsing end and currently bear Blame the number of the user equipment of parsing and the unique identification information of the current each user equipment for being responsible for parsing.

For example, it is assumed that the parsing end being connected with server-side is parsing end 1 and parsing end 2, when server-side is detecting solution When analysis end 3 accesses local, the physical address (e.g., 64-00-6A-14-9E) at parsing end 3 is obtained, and according to the physics at parsing end 3 Address creation one is used to save parsing end 3 and is currently responsible for the number of the user equipment parsed and is currently responsible for each of parsing Data list (e.g., the number of user equipment of responsible parsing: 0 of the unique identification information of user equipment;……).

Preferably, server-side can be respectively set one for each the parsing end having connected in the embodiment of the present invention Counter, i.e., the corresponding counter in one parsing end, a counter is for recording solution corresponding with said one counter It is responsible for the number of user equipment of parsing in analysis end.When server-side is that a user equipment for being responsible for parsing is distributed at a parsing end, Then this corresponding counter in parsing end just adds 1;And when a user equipment and server-side for parsing is responsible at a parsing end When disconnecting, then this corresponding counter in parsing end just subtracts 1.

For example, it is assumed that server-side be parse end 1 distribute counter be counter 1, and parse end 1 currently be responsible for parsing The number of user equipment is 3, i.e. counter 1 is shown as 3, then, when there is new user equipment access service end, and service New user equipment is distributed to parsing end 1 by end, at this point, 1 can be added the number 3 that counter 1 is shown, counter 1 is worked as Before be shown as 4, (3+1=4).

In another example, it is assumed that server-side is that the counter that parsing end 2 is distributed is counter 2, and parses the currently responsible parsing of end 2 User equipment number be 5, i.e. counter 2 is shown as 5, then, when parsing end 2 currently be responsible for parsing a user set It is standby when being disconnected with server-side, i.e., when no longer needing to parse end 2 and carrying out data parsing, so that it may the number for showing counter 2 Word 5 subtracts 1, and counter 2 is currently shown in 4, (5-1=4).

Step 210: corresponding data are arranged when having monitored user equipment access, for above-mentioned user equipment in server-side Queue, and the initial data that above-mentioned user equipment uploads is stored in above-mentioned data queue.

Specifically, server-side can be monitored in real time between server-side itself and each user equipment when executing step 210 Data queue corresponding with the new user equipment is arranged if server-side determines when having new user equipment access in connection status, The data list can be used for storing the initial data that the new user equipment uploads.Preferably, the data queue is a kind of advanced First go out the linear list of the data structure of (First Input First Output, FIFO).

Step 220: server-side determines each corresponding number of user equipment in parsing end respectively, and determines corresponding use The least parsing end of family number of devices.

Specifically, server-side is each with not obtaining when determining has new user equipment access when executing step 220 The count value of a corresponding counter in parsing end having connected, and according to the corresponding counter in each parsing end got Count value selects at least one corresponding parsing end of the least counter of at least one count value.

For example, it is assumed that the parsing end that server-side has connected is parsing end 1 and parsing end 2, the corresponding counter 1 in end 1 is parsed Count value be 5, parsing corresponding 2 count value of counter in end 2 be 4.If server-side determines that user equipment 1 accesses, service End then obtains the count value 5 of the parsing corresponding counter 1 in end 1 and the count value 4 of the parsing corresponding counter 2 in end 2, shows So, 4 < 5, then selecting parsing end 2 is the least parsing end of corresponding number of user equipment.

It wherein, can be according to parsing end if selecting at least two corresponding least parsing ends of number of user equipment Number order or randomly selected mode selected from the above-mentioned at least two corresponding least parsing ends of number of user equipment One parsing end is the parsing end for being responsible for parsing the initial data that the new user equipment uploads, certainly, specific selection mode It is specific fixed that limit is not done herein.

Step 230: above-mentioned initial data is sent to the least parsing end of above-mentioned corresponding number of user equipment by server-side, And trigger the least parsing end of above-mentioned corresponding number of user equipment and execute following operation: the initial data received is counted According to conversion process, corresponding engineering equipment data and set state data are obtained, and extremely by above-mentioned set state data publication In Kafka cluster, so that user obtains the set state data that above-mentioned user has subscribed to from above-mentioned Kafka cluster.

Specifically, when executing step 230, server-side determine the corresponding least parsing end of number of user equipment it Afterwards, that the initial data which uploads is sent to the above-mentioned corresponding number of user equipment having determined that is least Parse end.

For example, it is assumed that the parsing corresponding number of user equipment in end 1 is 3, and parsing end 2 is right when the user equipment access service end X The number of user equipment answered is 1, and the parsing corresponding number of user equipment in end 3 is 4, then, server-side determines that parsing end 2 is to correspond to The least parsing end of number of user equipment, and by user equipment X distribute to parsing end 2, at this point, parsing the corresponding user in end 2 Number of devices is 2, if server-side is determined there are the initial data that user equipment X is uploaded, which is sent to parsing End 2.

Further, the above-mentioned corresponding least parsing end of number of user equipment is triggered to set the new user received The standby initial data uploaded carries out dissection process, specifically, as shown in fig.3, another big data is handled in the embodiment of the present invention The detailed process of method is as follows:

Step 300: parsing end receives the initial data that the user equipment that server-side is sent uploads, wherein above-mentioned parsing end Above-mentioned server-side determines that corresponding number of users is minimum according to each corresponding number of user equipment in parsing end Parsing end.

In practical application, after server-side has new user equipment access in determination, firstly, it is necessary to from the parsing having connected The corresponding least parsing end of number of user equipment is determined in end;Then, by the corresponding least solution of number of user equipment Analysis end is set as being responsible for parsing the parsing end for the initial data that the new user equipment uploads.So, server-side is receiving this After the initial data that new user equipment uploads, the initial data which uploads is sent to the corresponding use Number of devices least parsing end in family carries out subsequent dissection process.

Step 310: parsing end carries out data conversion treatment to the initial data received, obtains corresponding engineering equipment number According to set state data.

In practical application, the initial data why use parsing end to upload the user equipment received is parsed Processing is because different equipment manufacturers has stringent security requirements, e.g., air-conditioning equipment to the data of the equipment of the equipment manufacturer Initial data be to be generated in strict accordance with the data protocol of setting, be the data that user fails to understand, then, it is necessary to use phase The initial data that the analysis program answered generates air-conditioning equipment carries out data conversion treatment, obtains what corresponding user can understand Data (e.g., user equipment is numbered, the running temperature of user equipment, the parameters such as humidity of user equipment running environment).

Specifically, when executing step 310, since the initial data that parsing termination receives is user equipment in strict accordance with number It is generated according to agreement, then, parsing end is in the initial data that the user equipment for receiving server-side transmission uploads, so that it may so that Data conversion treatment is carried out to the above-mentioned initial data received with corresponding analysis protocol, initial data is converted into accordingly Engineering equipment data and set state data, wherein engineering equipment data include at least: user equipment number, user equipment position The information such as set, and set state data include at least: the running temperature of user equipment, the humidity of user equipment running environment, use The information such as the comprehensive parameters of family equipment operation.

Certainly, parsing end is the prior art to the analysis mode of initial data, and details are not described herein, the embodiment of the present invention In, the specific analysis mode of parsing end parsing initial data is not specifically limited.

For example, parsing end 1 receives the initial data 1 that the user equipment 1 of server-side transmission uploads, and use is set with user The corresponding analysis protocol of data protocol used when standby 1 generation initial data 1 carries out data conversion treatment to initial data 1, will Initial data 1 is converted to engineering equipment data 1: number 0001, the mansion XX Building A Y floor etc.;Set state data 1: user equipment 1 running temperature is 25 degree, and the humidity of 1 running environment of user equipment is 60%, 1 normal operation of user equipment etc..

Step 320: parsing end by above-mentioned set state data publication into Kafka cluster, so that user is from above-mentioned Kafka The set state data subscribed to are obtained in cluster in real time.

So-called Kafka refers to distributed, visible partition a, reproducible message system.Kafka is by message It is concluded as unit of theme, and the object that Kafka theme gives out information is known as the producer, theme will be subscribed to and disappeared The object of breath is known as consumer, and Kafka is run in a manner of cluster, can be made of one or more services.The producer will not For news release with theme to Kafka cluster, Kafka cluster can provide message for consumer.

Specifically, parsing end classifies above-mentioned set state data according to different themes when executing step 320 Processing, and using the set state data of above-mentioned different themes as news release to Kafka cluster, to have subscribed to above-mentioned difference The user of the set state data of theme obtains the set state data of above-mentioned different themes in real time from above-mentioned Kafka cluster.

For example, parsing end as the message producer, by the running temperature of user equipment 1 obtained after parsing, user equipment 1 The humidity of running environment, the information such as comprehensive parameters that user equipment 1 is run are classified according to different type of theme, by user The comprehensive parameters of the running temperature of equipment 1, humidity and user equipment 1 operation of 1 running environment of user equipment are distributed to Kafka collection In group, user can obtain the set state data of the different themes for the user equipment 1 itself subscribed in real time, and e.g., user 1 makees For consumer, the message and theme that the running temperature that theme is user equipment 1 is subscribed to from Kafka cluster are that user equipment 1 is transported The message of the humidity of row environment, then, parsing end is by the wet of 1 running environment of the running temperature of user equipment 1 and user equipment After degree is distributed to Kafka cluster, user 1 can get the running temperature that the theme subscribed to is user equipment 1 in real time Message and theme are the message of the humidity of 1 running environment of user equipment.

Further, parsing end judges memory number after obtaining corresponding engineering equipment data and set state data According in library whether there is above-mentioned engineering equipment data.

If parsing end determines that there are above-mentioned engineering equipment data in memory database, it is determined that created in memory database The engineering of the corresponding user equipment of above-mentioned engineering equipment data;If parsing end determines to set in memory database there is no above-mentioned engineering Standby data then judge in relevant database with the presence or absence of above-mentioned engineering equipment data.

Further, if parsing end determines that there are above-mentioned engineering equipment data in above-mentioned relation type database, it is determined that The engineering of the corresponding user equipment of existing above-mentioned engineering equipment data in above-mentioned relation type database, and by above-mentioned engineering equipment The engineering of the corresponding user equipment of information is added to memory database;If parsing end determines to be not present in above-mentioned relation type database Above-mentioned engineering equipment data, then create the engineering of the corresponding user equipment of above-mentioned engineering equipment data, and by above-mentioned engineering equipment The engineering of the corresponding user equipment of data is added in memory database.

Based on the above embodiment, as shown in fig.4, in the embodiment of the present invention, a kind of big data processing unit (e.g., service End), include at least setting unit 40, determination unit 41 and transmission unit 42, wherein

Setting unit 40, for corresponding number to be arranged for the user equipment when having monitored user equipment access It is stored in the data queue according to queue, and by the initial data that the user equipment uploads;

Determination unit 41 for determining each corresponding number of user equipment in parsing end respectively, and is determined corresponding The least parsing end of number of user equipment;

Transmission unit 42, for the initial data to be sent to the least parsing of corresponding number of user equipment End, and trigger the least parsing end of the corresponding number of user equipment and execute following operate: to the initial data received into Row data conversion treatment, obtains corresponding engineering equipment data and set state data, and by the set state data publication Into Kafka cluster, so that user obtains the set state data that the user has subscribed to from the Kafka cluster.

Optionally, before having monitored user equipment access, setting unit 40 is further used for:

Monitoring that parsing termination is fashionable, it is described for saving according to physical address information creation one of the parsing end The data list of the corresponding number of user equipment in end and each customer equipment identification information is parsed, and is parsing end distribution one A counter, and based on the corresponding number of user equipment in parsing end described in the counter records.

Optionally, the data queue is a kind of linear list of the data structure of fifo fifo.

As shown in fig.5, in the embodiment of the present invention, a kind of big data processing unit (e.g., parsing end) includes at least and receives Unit 50, processing unit 51 and release unit 52, wherein

Receiving unit 50, the initial data that the user equipment for receiving server-side transmission uploads, wherein the big data The processing unit server-side is determined according to each corresponding number of user equipment of big data processing unit, right The least big data processing unit of the number of users answered;

Processing unit 51 obtains corresponding engineering equipment for carrying out data conversion treatment to the initial data received Data and set state data;

Release unit 52, for by the set state data publication into Kafka cluster, so that user is from described The set state data subscribed to are obtained in Kafka cluster in real time.

Optionally, after obtaining corresponding engineering equipment data and set state data, release unit 52 is further used In:

Judge in memory database with the presence or absence of the engineering equipment data;

If it is determined that there are the engineering equipment data in memory database, it is determined that created the work in memory database The engineering of the corresponding user equipment of journey device data.

Optionally, release unit 52 is further used for:

If it is determined that the engineering equipment data are not present in memory database, then judge to whether there is in relevant database The engineering equipment data;

If it is determined that there are the engineering equipment data in the relevant database, it is determined that in the relevant database The engineering of the corresponding user equipment of existing engineering equipment data, and by the corresponding user equipment of the engineering equipment information Engineering be added to memory database;

If it is determined that the engineering equipment data are not present in the relevant database, then the engineering equipment data are created The engineering of corresponding user equipment, and the engineering of the corresponding user equipment of the engineering equipment data is added to memory database In.

Optionally, by the set state data publication into Kafka cluster, so that user is from the Kafka cluster When obtaining the set state data subscribed in real time, release unit 52 is used for:

The set state data are subjected to classification processing according to different themes;

Using the set state data of the different themes as news release to Kafka cluster, so as to subscribed to it is described not Obtain the set state number of the different themes in real time from the Kafka cluster with the user of the set state data of theme According to.

In conclusion during carrying out big data processing, server-side is monitoring user in the embodiment of the present invention When equipment accesses, corresponding data queue, and the initial data that above-mentioned user equipment is uploaded are set for above-mentioned user equipment It is stored in above-mentioned data queue, determines each corresponding number of user equipment in parsing end respectively, and determine corresponding use Above-mentioned initial data is sent to the above-mentioned least parsing of corresponding number of user equipment by the least parsing end of family number of devices End, and trigger the least parsing end of above-mentioned corresponding number of user equipment and execute following operate: to the initial data received into Row data conversion treatment, obtains corresponding engineering equipment data and set state data, and by above-mentioned set state data publication Into Kafka cluster, so that user obtains the set state data that above-mentioned user has subscribed to from above-mentioned Kafka cluster.

Using above-mentioned big data processing method, the dynamic load leveling between each parsing end, further, user are realized The corresponding set state data of each user equipment can also be got in real time, ensure that the timeliness of set state data, And according to the corresponding set state data of each user equipment, detected in each user equipment in advance with the presence or absence of operation exception User equipment improve user equipment after-sale service quality, and then improve user's body to reduce the failure rate of user equipment It tests.

It should be understood by those skilled in the art that, the embodiment of the present invention can provide as method, system or computer program Product.Therefore, complete hardware embodiment, complete software embodiment or reality combining software and hardware aspects can be used in the present invention Apply the form of example.Moreover, it wherein includes the computer of computer usable program code that the present invention, which can be used in one or more, The computer program implemented in usable storage medium (including but not limited to magnetic disk storage, CD-ROM, optical memory etc.) produces The form of product.

The present invention be referring to according to the method for the embodiment of the present invention, the process of equipment (system) and computer program product Figure and/or block diagram describe.It should be understood that every one stream in flowchart and/or the block diagram can be realized by computer program instructions The combination of process and/or box in journey and/or box and flowchart and/or the block diagram.It can provide these computer programs Instruct the processor of general purpose computer, special purpose computer, Embedded Processor or other programmable data processing devices to produce A raw machine, so that being generated by the instruction that computer or the processor of other programmable data processing devices execute for real The device for the function of being specified in present one or more flows of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions, which may also be stored in, is able to guide computer or other programmable data processing devices with spy Determine in the computer-readable memory that mode works, so that it includes referring to that instruction stored in the computer readable memory, which generates, Enable the manufacture of device, the command device realize in one box of one or more flows of the flowchart and/or block diagram or The function of being specified in multiple boxes.

These computer program instructions also can be loaded onto a computer or other programmable data processing device, so that counting Series of operation steps are executed on calculation machine or other programmable devices to generate computer implemented processing, thus in computer or The instruction executed on other programmable devices is provided for realizing in one or more flows of the flowchart and/or block diagram one The step of function of being specified in a box or multiple boxes.

Although preferred embodiments of the present invention have been described, it is created once a person skilled in the art knows basic Property concept, then additional changes and modifications may be made to these embodiments.So it includes excellent that the following claims are intended to be interpreted as It selects embodiment and falls into all change and modification of the scope of the invention.

Obviously, those skilled in the art can carry out various modification and variations without departing from this hair to the embodiment of the present invention The spirit and scope of bright embodiment.In this way, if these modifications and variations of the embodiment of the present invention belong to the claims in the present invention And its within the scope of equivalent technologies, then the present invention is also intended to include these modifications and variations.

Claims (13)

1. a kind of big data processing method characterized by comprising
Server-side is monitoring that parsing termination is fashionable, creates one for saving institute according to the physical address information at the parsing end The data list of the parsing corresponding number of user equipment in end and each customer equipment identification information is stated, and is distributed for the parsing end One counter, and based on the corresponding number of user equipment in parsing end described in the counter records;
Corresponding data queue is arranged when having monitored user equipment access, for the user equipment in server-side, and by institute The initial data for stating user equipment upload is stored in the data queue;
Server-side determines each corresponding number of user equipment in parsing end respectively, and determines corresponding number of user equipment most Few parsing end;
The initial data is sent to the least parsing end of the corresponding number of user equipment by server-side, and trigger it is described right The least parsing end of the number of user equipment answered executes following operation: data conversion treatment is carried out to the initial data received, Corresponding engineering equipment data and set state data are obtained, and by the set state data publication into Kafka cluster, with Just user obtains the set state data that the user has subscribed to from the Kafka cluster.
2. the method as described in claim 1, which is characterized in that the data queue is a kind of data knot of fifo fifo The linear list of structure.
3. a kind of big data processing method characterized by comprising
Parsing, which is terminated, triggers the following operation of server-side execution into server-side: being created according to the physical address information at the parsing end One is used to save the data list of the parsing corresponding number of user equipment in end and each customer equipment identification information, and is A counter is distributed at the parsing end, and based on the corresponding number of user equipment in parsing end described in the counter records;
It parses end and receives the initial data that the user equipment that server-side is sent uploads, wherein the parsing end is the server-side According to each parsing end, corresponding number of user equipment is determined, the corresponding least parsing end of number of users;
Parsing end carries out data conversion treatment to the initial data received, obtains corresponding engineering equipment data and set state Data;
End is parsed by the set state data publication into Kafka cluster, so that user obtains in real time from the Kafka cluster Take the set state data subscribed to.
4. method as claimed in claim 3, which is characterized in that parsing end is obtaining corresponding engineering equipment data and unit shape After state data, further comprise:
Parsing end judges in memory database with the presence or absence of the engineering equipment data;
If parsing end determines that there are the engineering equipment data in memory database, it is determined that created in memory database described The engineering of the corresponding user equipment of engineering equipment data.
5. method as claimed in claim 4, which is characterized in that further comprise:
If parsing end to determine that the engineering equipment data are not present in memory database, judge whether deposit in relevant database In the engineering equipment data;
If parsing end determines that there are the engineering equipment data in the relevant database, it is determined that the relevant database In the corresponding user equipment of the existing engineering equipment data engineering, and the corresponding user of the engineering equipment information is set Standby engineering is added to memory database;
If parsing end to determine that the engineering equipment data are not present in the relevant database, the engineering equipment number is created Internal storage data is added to according to the engineering of corresponding user equipment, and by the engineering of the corresponding user equipment of the engineering equipment data In library.
6. such as the described in any item methods of claim 3-5, which is characterized in that parsing end by the set state data publication extremely In Kafka cluster, so that user obtains the set state data subscribed in real time from the Kafka cluster, specifically include:
It parses end and the set state data is subjected to classification processing according to different themes;
End is parsed using the set state data of the different themes as news release to Kafka cluster, described in having subscribed to The user of the set state data of different themes obtains the set state number of the different themes in real time from the Kafka cluster According to.
7. a kind of big data processing unit characterized by comprising
Setting unit creates a use according to the physical address information at the parsing end for monitoring that parsing termination is fashionable It in the data list for saving the parsing corresponding number of user equipment in end and each customer equipment identification information, and is the solution It analyses end and distributes a counter, and based on the corresponding number of user equipment in parsing end, Yi Ji described in the counter records When having monitored user equipment access, corresponding data queue is set for the user equipment, and will be on the user equipment The initial data of biography is stored in the data queue;
Determination unit for determining each corresponding number of user equipment in parsing end respectively, and determines that corresponding user sets The standby least parsing end of number;
Transmission unit for the initial data to be sent to the least parsing end of the corresponding number of user equipment, and touches To send out described corresponding, and the least parsing end of number of user equipment executes following operation: carrying out data to the initial data received and turns Processing is changed, obtains corresponding engineering equipment data and set state data, and by the set state data publication to Kafka collection In group, so that user obtains the set state data that the user has subscribed to from the Kafka cluster.
8. device as claimed in claim 7, which is characterized in that the data queue is a kind of data knot of fifo fifo The linear list of structure.
9. a kind of big data processing unit characterized by comprising
Receiving unit is used for access service end, and triggering server-side executes following operation: being believed according to the physical address at the parsing end Data column of the breath creation one for saving the parsing corresponding number of user equipment in end and each customer equipment identification information Table, and a counter is distributed for the parsing end, and set based on the corresponding user in parsing end described in the counter records Standby number, and receive the initial data that the user equipment that server-side is sent uploads, wherein the big data processing unit is institute State what server-side was determined according to each corresponding number of user equipment of big data processing unit, corresponding number of users Least big data processing unit;
Processing unit, for carrying out data conversion treatment to the initial data that receives, obtain corresponding engineering equipment data and Set state data;
Release unit, for by the set state data publication into Kafka cluster, so that user is from the Kafka cluster In obtain the set state data subscribed in real time.
10. device as claimed in claim 9, which is characterized in that obtaining corresponding engineering equipment data and set state number According to later, the release unit is further used for:
Judge in memory database with the presence or absence of the engineering equipment data;
If it is determined that there are the engineering equipment data in memory database, it is determined that created the engineering in memory database and set The engineering of the standby corresponding user equipment of data.
11. device as claimed in claim 10, which is characterized in that the release unit is further used for:
If it is determined that the engineering equipment data are not present in memory database, then judge in relevant database with the presence or absence of described Engineering equipment data;
If it is determined that there are the engineering equipment data in the relevant database, it is determined that deposited in the relevant database In the engineering of the corresponding user equipment of the engineering equipment data, and by the work of the corresponding user equipment of the engineering equipment information Journey is added to memory database;
If it is determined that the engineering equipment data are not present in the relevant database, then it is corresponding to create the engineering equipment data User equipment engineering, and the engineering of the corresponding user equipment of the engineering equipment data is added in memory database.
12. such as the described in any item devices of claim 9-11, which is characterized in that extremely by the set state data publication In Kafka cluster, when obtaining the set state data subscribed in real time from the Kafka cluster so as to user, the publication Unit is used for:
The set state data are subjected to classification processing according to different themes;
It is described different main to have subscribed to using the set state data of the different themes as news release to Kafka cluster The user of the set state data of topic obtains the set state data of the different themes in real time from the Kafka cluster.
13. a kind of big data processing system, which is characterized in that server-side, several parsing ends and several user equipmenies are included at least, Wherein,
Server-side creates one according to the physical address information at the parsing end and is used for for monitoring that parsing termination is fashionable The data list of the parsing corresponding number of user equipment in end and each customer equipment identification information is saved, and is the parsing One counter of end distribution, and based on the corresponding number of user equipment in parsing end described in the counter records, monitoring When having user equipment access, corresponding data queue, and the original that the user equipment is uploaded are set for the user equipment Beginning data are stored in the data queue, are determined each corresponding number of user equipment in parsing end respectively, are determined to correspond to The least parsing end of number of user equipment, and it is least that the initial data is sent to the corresponding number of user equipment End is parsed, the least parsing end of corresponding number of user equipment is triggered and executes following operation: to the initial data received Data conversion treatment is carried out, obtains corresponding engineering equipment data and set state data, and the set state data are sent out Cloth is into Kafka cluster, so that user obtains the set state data that the user has subscribed to from the Kafka cluster;
End is parsed, the initial data that the user equipment for receiving server-side transmission uploads, wherein the parsing end is the clothes Business end parses what the corresponding number of user equipment in end was determined according to each, the corresponding least parsing of number of users End carries out data conversion treatment to the initial data received, obtains corresponding engineering equipment data and set state data, will The set state data publication is into Kafka cluster, so that user obtains the machine subscribed to from the Kafka cluster in real time Group status data.
CN201710178126.1A 2017-03-23 2017-03-23 A kind of big data processing method, apparatus and system CN107423316B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710178126.1A CN107423316B (en) 2017-03-23 2017-03-23 A kind of big data processing method, apparatus and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710178126.1A CN107423316B (en) 2017-03-23 2017-03-23 A kind of big data processing method, apparatus and system

Publications (2)

Publication Number Publication Date
CN107423316A CN107423316A (en) 2017-12-01
CN107423316B true CN107423316B (en) 2019-02-15

Family

ID=60424070

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710178126.1A CN107423316B (en) 2017-03-23 2017-03-23 A kind of big data processing method, apparatus and system

Country Status (1)

Country Link
CN (1) CN107423316B (en)

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4157403B2 (en) * 2003-03-19 2008-10-01 株式会社日立製作所 Packet communication device
CN104536405A (en) * 2014-12-16 2015-04-22 珠海格力电器股份有限公司 Remote monitoring system of air conditioning unit

Also Published As

Publication number Publication date
CN107423316A (en) 2017-12-01

Similar Documents

Publication Publication Date Title
US7975186B2 (en) Operations management apparatus, operations management system, data processing method, and operations management program
US9270521B2 (en) Provisioning and managing a cluster deployed on a cloud
US6295557B1 (en) Apparatus for simulating internet traffic
US8861397B2 (en) Apparatus and method for analyzing a network
US20140281308A1 (en) Storage unit selection for virtualized storage units
US20120222029A1 (en) Method of monitoring performance of virtual computer and apparatus using the method
US9053240B2 (en) Computer program testing
CN101707632A (en) Method for dynamically monitoring performance of server cluster and alarming real-timely
JPH077518A (en) Method for analyzing network
US8209511B2 (en) Storage management apparatus, a storage management method and a storage management program
CN105378669A (en) Virtual machine resource management system and method thereof
WO2001041362A2 (en) Load balance apparatus and method
EP2523115A1 (en) Operation management device, operation management method, and program storage medium
CN103460203B (en) cluster unique identifier
US10140142B2 (en) Grouping and placement of virtual machines based on similarity and correlation of functional relations
CN1992635A (en) Method of simulating SNMP network element and performing network management system test with the network element
JP2012079242A (en) Composite event distribution device, composite event distribution method and composite event distribution program
CN102158370B (en) Automated testing method and system
EP2095236B1 (en) Method, system and computer program for testing software applications based on multiple data sources
US10313183B2 (en) Network function virtualization NFV fault management apparatus, device, and method
US9331932B2 (en) Network system
CN102868736B (en) A kind of cloud computing Monitoring framework design basis ground motion method and cloud computing treatment facility
US20060277440A1 (en) Method, system, and computer program product for light weight memory leak detection
CN102037681A (en) Method and apparatus for managing computing resources of management systems
US20160248858A1 (en) Method and Apparatus for Virtualized Network Function Chaining Management

Legal Events

Date Code Title Description
PB01
GR01