KR20190130216A - method of generating demand forecast data by online bigdata analysis - Google Patents

Abstract

The present invention relates to a technique for forecasting demand by normally analyzing various data. In particular, provided is a technique for finding places where actual data exist, collecting the same everyday in real time, and applying various environment variables, thereby providing an advanced demand forecast algorithm and a result value. Further, the present invention relates to a technique for adjusting a web crawling weight value in generating demand forecast data through big data analysis. In particular, provided is a big data analysis modeling and demand forecast processing technique which can increase big data analysis performance and effectively perform demand forecasting by classifying data in a big data inflow route by a data classification method and feedback-adjusting a weight value for each classification in gathering large amounts of web data through web crawler, which is a computer program exploring the world wide web by an organized and automated method in internet marketing.

Description

Method of generating demand forecast data by online big data analysis

The present invention generally relates to techniques for predicting demand by analyzing various data. In particular, it provides advanced technology for predicting demand and finding results by finding the place where real data exists and applying daily environment variables collected daily and in real time.

The present invention also relates to a technique for adjusting web crawling weights in generating demand forecast data through big data analysis. In particular, in collecting large-scale web data through web crawling, a computer program that browses the World Wide Web in an organized and automated way in Internet marketing, data is classified and classified according to data classification in the big data funnel. It provides big data analysis modeling and demand forecasting technology that can improve big data analysis performance and efficiently demand forecasting by adjusting weights by classification.

Recently, due to the development of smart devices and networks and the development of various network services, the amount of open and accumulated data has been greatly increased, and all institutions, companies, and individuals have analyzed the vast amount of data to provide convenience and improve service levels. Diversified research and development.

However, the data that can be used is inevitably relying on public data held by specific SNS, news and public institutions. However, there are many problems to utilize it. Even though the data is needed, there are many problems such as access to the data in order to use it, problems with the data provider's data disclosure level, and usability due to different formats of the data.

Accordingly, there is a need for a data collection and analysis processing technology that can solve the problems of the prior art by improving a method of collecting and analyzing and processing a large amount of data published on the Internet.

It is generally an object of the present invention to provide a technique for predicting demand by analyzing various data. In particular, it seeks to provide an advanced demand forecasting algorithm and technology that provides real-time demand prediction algorithms by finding various places where real data exists and applying them in real time every day.

It is also an object of the present invention to provide a technique for adjusting web crawling weights in generating demand forecast data through big data analysis. In particular, in collecting large-scale web data through web crawling, a computer program that browses the World Wide Web in an organized and automated way in internet marketing, data is classified by the data classification method in the big data funnel and weighted by the classification. By adjusting the feedback, the company aims to provide big data analysis modeling and demand forecasting techniques that can improve big data analysis performance and efficiently perform demand forecasting.

In order to achieve the above object, the present invention provides a method for generating demand forecast data through online big data analysis.

In particular, the present invention proposes a web crawl weight adjustment technique for generating demand forecast data through online big data analysis.

On the other hand, the computer program according to the present invention is stored in the medium in order to execute the method for generating demand forecast data through on-line big data analysis combined with hardware.

According to the present invention, rather than analyzing closed, processed, and ineffective data in the prior art, data obtained in real time from the currently operating market is analyzed at that time, thereby predicting the market demand, and thus the utilization as a market analysis model is high. There is an advantage.

In addition, the big data analysis module (BAFP) presented in the present invention is designed to utilize the open and shared large amounts of data in the big data era, thereby enabling accurate analysis and because of the target scalability of big data analysis, various industries And country can be analyzed.

In addition, according to the present invention, it is possible to analyze by generation and by country according to consumption patterns, and thus, there is an advantage of predicting demand for each target through analysis of behavior patterns between generations and overseas consumption patterns.

1 is a diagram conceptually illustrating an operation structure of a web crawler.
FIG. 2 conceptually illustrates a data processing structure associated with a big data analysis module (BAFP) in the present invention. FIG.
3 is a view showing in more detail the data processing structure associated with the Big Data Analysis Module (BAFP) in the present invention.
4 is a diagram conceptually illustrating a process of processing big data by a big data analysis module (BAFP) in the present invention.
5 is a view conceptually illustrating the adjustment of trend weights in a commodity valuation equation in the present invention.
6 is a view showing an example utilizing the product preference analysis in the present invention.
7 is a block diagram conceptually showing a functional module of a big data analysis module (BAFP) in the present invention.
8 is a diagram showing an example in which a functional module of a big data analysis module (BAFP) refers to internal processing in the present invention.
9 is a diagram conceptually illustrating a reporting process according to a big data analysis result in the present invention.
10 is a diagram showing an example of analysis reporting in the present invention.

Hereinafter, with reference to the drawings will be described in detail the present invention.

First, we will look into data collection / inflow and backward search for big data analysis modeling and demand forecast data generation.

Data collection has different approaches depending on the scope of disclosure, and the content of data to be collected depends on the purpose of the data.

In addition, since most of the data collection does not support API (Application Program Interface), it can be extracted through various methods depending on the network structure and security environment of the collection target.

Regarding the collection method, data must be collected in real time every day, and data can be collected according to preset patterns and schedules for accurate analysis. On the other hand, because of the large number of virtualizations required, the collection server requires high specification and network speed. For example, data collection can be done using Python, which has multiple virtualizations.

In the present invention, web crawling and data parsing may be utilized for big data collection and retrieval. Through web crawling, we define classifications based on the targets to be collected and the utilization values of data, and through data parsing, periodically collect data for analysis purposes and make data.

1 is a diagram conceptually showing an operation structure of a web crawler that can be well utilized for big data collection in the present invention.

Web crawlers are computer programs that navigate the World Wide Web in an organized and automated way. In general, what a web crawler does is called web crawling or spidering. Many sites, such as search engines, crawl the web to keep data fresh. The web crawler is used to create a copy of every page of the visited site, and search engines index it to quickly search these generated pages. Crawlers are also used for automated maintenance of Web sites, such as link checking and HTML code validation, and to collect certain types of information on Web pages, such as automatic email collection.

Web crawlers are a form of bot or software agent. Web crawlers usually start with a list of URLs called seeds, which recognize all hyperlinks on the page and update the list of URLs. The updated list of URLs recursively visits again.

The working structure of a computer program search engine with the ability to navigate the World Wide Web in an organized and automated way works in three stages: Web crawling, indexing, and searching.

First, the web crawling process copies all the pages of your site. This takes precedence over the latest data. The indexing process collects, stores, and analyzes the data for accurate and fast retrieval. Finally, the extraction process extracts a title, a subject, and the like including words constituting the search word.

This series of processes leads to a search. The search ability of a search engine is determined by the relationship between the search term and the data. Among the searched web pages, there are pages that are more relevant to the search keywords, and the search engine continues to provide feedback through indexing analysis of the most relevant results. Adjust with

Next, the data collection and inflow (log analysis), the creation and retrieval of a cloud database (Cloud DB), and the continuous feedback adjustment configuration of weight values are described in the big data analysis module (BAFP).

2 and 3 are diagrams conceptually showing a data processing structure related to a big data analysis module (BAFP) in the present invention. In order to analyze a large amount of data collected online through web crawling, indexing, extraction, etc. according to the analysis purpose (that is, demand prediction data generation) according to the present invention, the present invention includes a big data analysis module (BAFP). ) Can be used to derive a result that is consistent with the purpose of the analysis.

First, a plurality of tasks are provided to cope with various environments in collecting and introducing big data online. For each task, individual information collection is performed by third-party homepage. Since the type of collected data and the collection environment are different for each information collection site, a module suitable for the environment is configured as a task.

Next, the Big Data Analysis Module (BAFP) describes the process of processing the big data collected and introduced.

First, the Big Data Analysis Module (BAFP) classifies different variables collected in different environments according to an analysis purpose, stores them in a log data database, and performs comparative analysis. The comparatively analyzed data is processed into a modular Cloud Data set to store data useful for analysis.

In addition, the Big Data Analysis Module (BAFP) infers the data value by inverting the data not extracted by comparing the elements (variables) that collect the information required by the third-party homepages.

The following is an example of big data analysis. Task 1 ~ Task N After analyzing sales volume, view and review, we infer sales volume and number of inquiry through review of Task-i. At this time, the big data analysis module (BAFP) is an element that collects necessary information from the third-party homepage site, it is possible to infer the correct data value by applying weights such as industry type, classification, price.

Meanwhile, in the present invention, an example of a collection element (variable) for big data analysis in the big data analysis module (BAFP) is as follows.

1) Product category

2) Product name or unique key

3) Price

4) Selling number

5) View count

6) Review count

7) Q & A count

8) Update Time

4 is a diagram conceptually illustrating a process of processing big data by a big data analysis module (BAFP) in the present invention. In the present invention, the Big Data Analysis Module (BAFP) defines and associates the attributes and associations of each of the collected data, and empirically assumes various variables by repeatedly calculating the calculated data values with another parameter. Derive the data value.

In the present invention, data collection / inflow (log data analysis) and cloud database generation for the big data analysis module (BAFP), cloud database generation, and continuous search and weighting values are analyzed to analyze the correlation of elements (variables) that collect information required by third-party homepages. Perform feedback processing.

On the other hand, the collection elements (variables) exemplified above are examined.

1) Product Update Time (Analysis Key Factor): It is a standard to perform demand analysis and forecast by assigning each variable based on the time of product release.

2) Selling count (analytical key factor): This is a value that changes in real time. Real-time collection is required for comparative prediction.

3) View count (Analysis key factor): It is possible to analyze the hit rate of product purchase by calculating with purchase variables and preference for product is the source of evaluation.

4) Price (weighted variable factor): This is a comparison standard of clusters and affects demand forecast analysis. Intermittent changes may be necessary, so periodic collection comparison is necessary.

5) Review count (weighted variable factor): Through evaluation of the product, the direction of improvement and development criteria are prepared.

6) Q & A count (weight factor): It is necessary to analyze the interest index and product characteristics of the product.

7) Product category (weight variables): A detailed definition is needed to analyze market share and form groups.

8) Product name or unique key (reference element): To identify a product, a product name is defined (collected) or distinguished by a unique value.

Next, the present invention describes a product preference analysis method using a big data analysis module (BAFP).

As a preferred embodiment of the present invention through the product evaluation conversion formula of [Equation 1] the effectiveness of the product (evaluation indicators of the product) is analyzed.

In the present invention, the product valuation equation provides a concept of the product valuation equation to numerically represent the product valuation, which is an element (variable) that collects necessary information from a third-party homepage. [Equation 1] is an example. The numerical value of the product evaluation is the actual measurement result that defines the product's productability. So-called psychological support lines are generated by classification and price, and the reference point is measured differently. In the same classification, they show similar or identical patterns and are weighted differently according to price.

In Equation 1, the trend weight is calculated according to the result of the flow of the result obtained by applying the period to several sectors, and is applied as a weight to the product valuation conversion. In the present invention, the product evaluation has a change in flow according to trends and external variables (seasonal, specificity), but generally has a constant change pattern, so that the coefficient value of the trend weight can be extracted.

The use cases of such product evaluation are as follows.

1) Wet tissue (classification of necessities): The measured value is measured very high and weighted according to the price.

2) Luxury goods (classification of luxury goods): The measured value is measured very low and the weight is generated low according to the price.

5 is a view conceptually illustrating the adjustment of the trend weight in the product valuation equation in the present invention. A mathematical approach for weight adjustment is presented. 5GE shown in FIG. 5 is data analyzed for a trend-high product family (clothing), and one feature of the present invention is that the period can be intelligently adjusted by, for example, artificial intelligence (AI) according to the sensitivity of the trend.

6 is a view showing an example using the product preference analysis as shown in [Equation 1] in the present invention. As in the case of FIG. 6, the result reliability may be adjusted according to a selection condition or a selective condition according to the sensitivity of the investigation target. In this case, options such as sales order, sales rate (sales volume / view count), sales trend change order, season, month, week, etc. may be set for the reliability of the analysis result.

Next, big data analysis and demand prediction, data classification method, and correlation weight adjustment for the big data analysis module (BAFP) will be described in the present invention from elements (variables) that collect information required by third-party homepages.

FIG. 7 is a block diagram conceptually illustrating a functional module of a big data analysis module (BAFP) in the present invention, and FIG. 8 is a function module of the big data analysis module (BAFP) in the present invention for reference to internal processing. It is a figure which shows an example.

The data collected for the Big Data Analysis Module (BAFP) is analyzed as the analysis result values are matched to the purpose of the analysis as shown in FIG. 2, and each cloud data set is provided through the Big Data Analysis Module (BAFP) of FIG. It is stored in (Cloud Data Set) and Log Data Set, and then re-analyzed and updated.

The big data analysis module (BAFP) of FIG. 7 defines each correlation of M pieces of data collected in real time and constructs a set of operations, and calculates another association except the calculated data and the analysis day. The data value is derived through N-1 iterations of calculating the coefficients. The derived data values can be predicted by intersecting and sequential calculation process with each module such as Pattern Classification Module, A Correlation Coefficient Module, and Weighted Ratio Module. It is processed into data.

In the present invention, a weight adjustment method of data collection is described. In order to generate accurate and real-time analysis data and to efficiently use system resources, we propose a more efficient algorithm for collection. Scheduling is performed by assigning a weight to each update data, which enables automatic movement to the data store and the collection scheduling store according to the data classification method.

The collection system database structure is described in the present invention. For different data sets, different periods are classified, such as Data 1 (10 minute period), Data 2 (30 minute period), Data 3 (1 hour period), Data 4 (3 hour period), Data 5 ( 6 hour cycle), Data 6 (12 hour cycle), Data 7 (18 hour cycle), Data 8 (24 hour cycle), and Data Free (autonomous collection).

The standard can perform efficient data update by selectively inserting and moving to the database pool (DB.Pool) according to the weight updated in 1 hour. Since there are sequential selects for every 9 levels of DB.Pool, there is an effect that is selected automatically.

Next, the present invention describes the efficiency analysis of collecting variable data from third-party homepages for the big data analysis module (BAFP). For Big Data Analysis Module (BAFP), it is possible to configure factor (variable) weights to collect necessary information from third-party homepages and to report through the analysis results of data. At this time, a change in the degree of rise and the degree of fall can be detected.

FIG. 9 is a diagram conceptually illustrating a reporting process according to a big data analysis result in the present invention, and FIG. 10 is a diagram illustrating an example of analysis reporting derived through big data analysis in the present invention. In FIG. 10, the classification by numerical diagram may be further subdivided according to the number of cases generated while various examples are generated to derive a result value for analysis on the measurement.

As described above, a variable data classification method and a correlation weight adjustment method for collecting information required by third-party homepages may be implemented in various ways. For example, 1) time, day of the week, seasonal distribution analysis, 2) rise and fall change analysis, 3) sporadic rapid change analysis, 4) sales distribution analysis, and 5) market share analysis. Based on the different visual analysis, it is possible to provide an automatic reporting evaluation report on the overall items.

Meanwhile, the present invention may be embodied in the form of computer readable codes on a computer readable nonvolatile recording medium. Such nonvolatile recording media include various types of storage devices, such as hard disks, SSDs, CD-ROMs, NAS, magnetic tapes, web disks, cloud disks, etc., and code is distributed in a plurality of networked storage devices. Forms that are implemented and executed may also be implemented. In addition, the present invention may be implemented in the form of a computer program stored in a medium in combination with hardware to execute a specific procedure.

Claims (2)

Demand forecast data generation method through online big data analysis.
A computer program coupled to hardware and stored in a medium for executing the demand forecast data generation method through on-line big data analysis according to claim 1.

Images