CN104133834A - Designated area microblog data collecting and processing method - Google Patents

Abstract

The invention discloses a designated area microblog data collecting and processing method. According to the method, firstly, GEO geographic information seed point selection is carried out; then, microblog data is obtained; and finally, the microblog data is processed. The designated area microblog data collecting and processing method has the advantages that a parallel multi-user calling mode is adopted for increasing the data collecting flow rate; and multi-information-point coverage is adopted for searching and collecting the microblog data, and the requirements of designated area microblog data collection and processing can be met.

Description

Specify region microblogging Data Collection and disposal route

Technical field

The present invention relates to microblogging data processing method field, specifically a kind of region microblogging Data Collection and disposal route of specifying.

Background technology

Along with the rise of microblogging, this rapid enrichment of short text that has comprised a large amount of microcosmic points and be inclined to emotion, microblogging text analyzing becomes popular research direction.

In microblogging data collection process, a large amount of microblogging data collection strategies adopt crawler capturing method conventionally, and the method grasp speed is fast, efficiency is high, but the noise data capturing is large, although reduced the time of data collection, but increase at double obtain the pretreatment time of precise information; And reptile is unstable, usually face the danger of being closed by Sina.A small amount of microblogging data generally adopt the microblogging third party API of Sina to call collection, and noise data that the method is collected is few, region is obvious, but has comprised a large amount of advertisements, has additionally increased again gibberish ratio.

No matter be reptile method or the traditional third party of Sina API Calls, all cannot obtain in a large number the microblogging data under specified domain, particularly specify the processing of microblogging data under region, reptile method and the third party of Sina API Calls all cannot be suitable for.

Summary of the invention

The object of this invention is to provide a kind of region microblogging Data Collection and disposal route of specifying, cannot obtain in a large number the problem of specifying microblogging data under region to solve prior art reptile method or third party's API Calls.

In order to achieve the above object, the technical solution adopted in the present invention is:

Specify region microblogging Data Collection and disposal route, it is characterized in that: comprise the following steps:

(1), GEO geography information Seed Points is chosen:

If the target seed amount of counting is N, region, given city is used to rectangle cutting, determine urban fringe; Do rectangular area diagonal line, do parallel lines with 10 kilometers of spacing of map scale length, cut apart rectangular area; Cut apart on parallel lines at each, cover successively rectangular area taking 5 kilometers of map scale length as radius does border circular areas, each border circular areas is not overlapping; On separator bar, the region of 5 kilometers of less thaies adopts suitable border circular areas to cover by actual conditions; For each border circular areas junction, cover this region taking engineer's scale radius as the border circular areas of R kilometer, R≤5, require to accomplish that overlapping region is no more than 3%; The center of circle that covers each border circular areas of specifying region is candidate GEO geography information Seed Points, and candidate GEO geography information Seed Points total quantity is designated as N', determines last Seed Points quantity according to formula (1):

$f=\left\{\begin{array}{cc}{N}^{\&prime;},& N^{\&prime;}<N\\ N,& N\&le;N^{\&prime;}\end{array}\right.---\left(1\right)$

In formula (1), f represents Seed Points quantity, in the time that candidate GEO geography information Seed Points quantity N' is less than the target seed amount of counting N, gets candidate GEO geography information Seed Points as final Seed Points; In the time that candidate GEO geography information Seed Points quantity N' is greater than the target seed amount of counting N, adjust position and the radius size of candidate GEO geography information Seed Points border circular areas, the border circular areas quantity that makes to cover rectangular area be N with interior integer, what now select border circular areas after adjusting is final Seed Points region;

According to the final Seed Points of above-mentioned acquisition, i.e. the Seed Points of f representative, positioning map information, derives longitude and latitude data, can obtain the GEO geography information of Seed Points;

(2), microblogging data acquisition;

The Seed Points GEO geographic information data obtaining according to step (1), calls microblogging third party api interface, obtains the microblogging data in appointed area; Microblogging data comprise microblogging creation-time, micro-blog information content, geographical information field; The microblogging data of obtaining are kept at this locality by the TXT text of UTF-8 form, are designated as D _gEO;

(3), microblogging data processing:

By the microblogging text D obtaining in step (2) _gEOextract in pairs according to microblogging creation-time and corresponding geography information, and be kept at this locality with the text of UTF_8 form, be designated as D _{t × geo}; From microblogging text D _gEOin extract micro-blog information content, and the local text with UTF-8 form preserves, and is designated as D _cont.

The present invention has improved the third party API of Sina, adopts parallel multi-user's method of calling to increase data collection flow; Adopt many information points to cover and collect microblogging data, obtain the deficiency of data accuracy to make up microblogging interface, can meet the requirement of specifying microblogging Data Collection and processing under region.

Embodiment

Specify region microblogging Data Collection and disposal route, region is the region that has microblogging user issuing microblog, and border, region is divided with Administrative boundaries; All microbloggings that region microblogging sends for the microblog users appearing in appointment region.Comprise the following steps:

(1), GEO geography information Seed Points is chosen:

If the target seed amount of counting is N, region, given city is used to rectangle cutting, determine urban fringe; Do rectangular area diagonal line, do parallel lines with 10 kilometers of spacing of map scale length, cut apart rectangular area; Cut apart on parallel lines at each, cover successively rectangular area taking 5 kilometers of map scale length as radius does border circular areas, each border circular areas is not overlapping; On separator bar, the region of 5 kilometers of less thaies adopts suitable border circular areas to cover by actual conditions; For each border circular areas junction, cover this region taking engineer's scale radius as the border circular areas of R kilometer, R≤5, require to accomplish that overlapping region is no more than 3%; The center of circle that covers each border circular areas of specifying region is candidate GEO geography information Seed Points, and candidate GEO geography information Seed Points total quantity is designated as N', determines last Seed Points quantity according to formula (1):

$f=\left\{\begin{array}{cc}{N}^{\&prime;},& N^{\&prime;}<N\\ N,& N\&le;N^{\&prime;}\end{array}\right.---\left(1\right)$

In formula (1), f represents Seed Points quantity, in the time that candidate GEO geography information Seed Points quantity N' is less than the target seed amount of counting N, gets candidate GEO geography information Seed Points as final Seed Points; In the time that candidate GEO geography information Seed Points quantity N' is greater than the target seed amount of counting N, adjust position and the radius size of candidate GEO geography information Seed Points border circular areas, the border circular areas quantity that makes to cover rectangular area be N with interior integer, what now select border circular areas after adjusting is final Seed Points region;

According to the final Seed Points of above-mentioned acquisition, i.e. the Seed Points of f representative, positioning map information, derives longitude and latitude data, can obtain the GEO geography information of Seed Points;

(2), microblogging data acquisition;

The Seed Points GEO geographic information data obtaining according to step (1), calls microblogging third party api interface, obtains the microblogging data in appointed area; Microblogging data comprise microblogging creation-time, micro-blog information content, geographical information field; The microblogging data of obtaining are kept at this locality by the TXT text of UTF-8 form, are designated as D _gEO;

(3), microblogging data processing:

By the microblogging text D obtaining in step (2) _gEOextract in pairs according to microblogging creation-time and corresponding geography information, and be kept at this locality with the text of UTF_8 form, be designated as D _{t × geo}; From microblogging text D _gEOin extract micro-blog information content, and the local text with UTF-8 form preserves, and is designated as D _cont.

Claims (1)

1. specify region microblogging Data Collection and disposal route, it is characterized in that: comprise the following steps:

(1), GEO geography information Seed Points is chosen:

If the target seed amount of counting is N, region, given city is used to rectangle cutting, determine urban fringe; Do rectangular area diagonal line, do parallel lines with 10 kilometers of spacing of map scale length, cut apart rectangular area; Cut apart on parallel lines at each, cover successively rectangular area taking 5 kilometers of map scale length as radius does border circular areas, each border circular areas is not overlapping; On separator bar, the region of 5 kilometers of less thaies adopts suitable border circular areas to cover by actual conditions; For each border circular areas junction, cover this region taking engineer's scale radius as the border circular areas of R kilometer, R≤5, require to accomplish that overlapping region is no more than 3%; The center of circle that covers each border circular areas of specifying region is candidate GEO geography information Seed Points, and candidate GEO geography information Seed Points total quantity is designated as N', determines last Seed Points quantity according to formula (1):

$f=\left\{\begin{array}{cc}{N}^{\&prime;},& N^{\&prime;}<N\\ N,& N\&le;N^{\&prime;}\end{array}\right.---\left(1\right)$

In formula (1), f represents Seed Points quantity, in the time that candidate GEO geography information Seed Points quantity N' is less than the target seed amount of counting N, gets candidate GEO geography information Seed Points as final Seed Points; In the time that candidate GEO geography information Seed Points quantity N' is greater than the target seed amount of counting N, adjust position and the radius size of candidate GEO geography information Seed Points border circular areas, the border circular areas quantity that makes to cover rectangular area be N with interior integer, what now select border circular areas after adjusting is final Seed Points region;

According to the final Seed Points of above-mentioned acquisition, i.e. the Seed Points of f representative, positioning map information, derives longitude and latitude data, can obtain the GEO geography information of Seed Points;

(2), microblogging data acquisition;

The Seed Points GEO geographic information data obtaining according to step (1), calls microblogging third party api interface, obtains the microblogging data in appointed area; Microblogging data comprise microblogging creation-time, micro-blog information content, geographical information field; The microblogging data of obtaining are kept at this locality by the TXT text of UTF-8 form, are designated as D _gEO;

(3), microblogging data processing:

By the microblogging text D obtaining in step (2) _gEOextract in pairs according to microblogging creation-time and corresponding geography information, and be kept at this locality with the text of UTF_8 form, be designated as D _{t × geo}; From microblogging text D _gEOin extract micro-blog information content, and the local text with UTF-8 form preserves, and is designated as D _cont.