WO2019140567A1 - Procédé et système d'analyse de mégadonnées - Google Patents

Procédé et système d'analyse de mégadonnées Download PDF

Info

Publication number
WO2019140567A1
WO2019140567A1 PCT/CN2018/072989 CN2018072989W WO2019140567A1 WO 2019140567 A1 WO2019140567 A1 WO 2019140567A1 CN 2018072989 W CN2018072989 W CN 2018072989W WO 2019140567 A1 WO2019140567 A1 WO 2019140567A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
control terminal
analysis
big data
analyzed
Prior art date
Application number
PCT/CN2018/072989
Other languages
English (en)
Chinese (zh)
Inventor
张北江
衣佳鑫
Original Assignee
新联智慧信息技术（深圳）有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 新联智慧信息技术（深圳）有限公司 filed Critical 新联智慧信息技术（深圳）有限公司
Priority to PCT/CN2018/072989 priority Critical patent/WO2019140567A1/fr
Publication of WO2019140567A1 publication Critical patent/WO2019140567A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24—Querying
    • G06F16/245—Query processing
    • G06F16/2455—Query execution

Definitions

  • the present invention relates to the field of communications technologies, and in particular, to a method and system for analyzing big data.
  • McKinsey The earliest mention of the "big data” era is the world-renowned consulting firm McKinsey, McKinsey said: "Data has penetrated into every industry and business function field today and has become an important production factor. People are exploring and using massive data. A new wave of productivity growth and the wave of consumer surplus.” “Big data” has existed in the fields of physics, biology, environmental ecology, and military, finance, communications, etc., but because of the Internet in recent years. And the development of the information industry has attracted people's attention.
  • Big Data is often used to describe a large amount of unstructured and semi-structured data created by a company that spends too much time and money when downloaded to a relational database for analysis. Big data analytics is often associated with cloud computing because real-time large dataset analysis requires a framework like MapReduce to distribute work to dozens, hundreds, or even thousands of computers.
  • the existing cloud platform is slow to process big data, which cannot meet the requirements of users and affect the user experience.
  • the embodiment of the present invention provides a method and a system for analyzing big data, and performs processing of big data by configuring a terminal of the cloud platform to improve processing speed of big data and improve user experience.
  • an embodiment of the present application provides a method for analyzing big data, where the method includes the following steps:
  • the control terminal receives the data amount of the big data to be analyzed and the analysis instruction
  • the control terminal parses the analysis instruction to determine whether the big data can be split. If the data can be split, the storage address of the big data to be analyzed is split into multiple partial storage addresses;
  • the control terminal acquires the load value of the analysis terminal and the number of transmission hops
  • the control terminal distributes the plurality of partial storage addresses and the analysis instruction to the analysis terminal according to the transmission hop count and the load value;
  • the analyzing terminal extracts part of the data corresponding to the part of the storage address, and analyzes the part of the data according to the analysis instruction to obtain a partial result, and returns the partial result to the control terminal;
  • the control terminal combines the partial results to obtain the final result of the big data to be analyzed.
  • a second aspect provides an analysis system for big data, the system comprising: a control terminal and an analysis terminal; wherein
  • control terminal configured to receive a data quantity of the big data to be analyzed and an analysis instruction; analyze the analysis instruction to determine whether the big data can be split, and if the data can be split, split the storage address of the big data to be analyzed into a plurality of partial storage addresses; obtaining a load value of the analysis terminal and a transmission hop count; distributing the plurality of partial storage addresses and the analysis instruction to the analysis terminal according to the transmission hop count and the load value;
  • the analysis terminal is configured to extract part of the data corresponding to the part of the storage address, and analyze and process the part of the data according to the analysis instruction to obtain a partial result, and return the partial result to the control terminal;
  • the control terminal is further configured to obtain a final result of the big data to be analyzed according to the partial result.
  • 1 is a schematic flow chart of a method for analyzing big data
  • FIG. 2 is a schematic diagram of a method for analyzing big data provided by an embodiment of the present application
  • FIG. 3 is a schematic structural diagram of a hardware provided by an embodiment of the present application.
  • FIG. 4 is a schematic structural diagram of a network topology according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of an analysis system for big data provided by an embodiment of the present application.
  • FIG. 6 is a schematic structural diagram of another terminal according to an embodiment of the present application.
  • references to "an embodiment” herein mean that a particular feature, result, or characteristic described in connection with the embodiments can be included in at least one embodiment of the invention.
  • the appearances of the phrases in various places in the specification are not necessarily referring to the same embodiments, and are not exclusive or alternative embodiments that are mutually exclusive. Those skilled in the art will understand and implicitly understand that the embodiments described herein can be combined with other embodiments.
  • a mobile terminal also called a User Equipment (UE) is a device that provides voice and/or data connectivity to users, for example, a handheld device with an infinite connection function, an in-vehicle device, and the like.
  • Common terminals include, for example, mobile phones, tablets, laptops, PDAs, mobile internet devices (mobile Internet device, MID), wearable devices such as smart watches, smart bracelets, pedometers, etc.
  • FIG. 1 is a method for analyzing big data.
  • the method for analyzing big data includes the following steps.
  • the method may be performed by a terminal, and the terminal may specifically be: a mobile phone, a tablet computer, or a notebook.
  • MID mobile internet device
  • Step S101 The terminal receives the big data and analyzes the instruction.
  • Step S102 The terminal extracts part of the data of the big data, and then analyzes and processes the part of the data according to the analysis instruction to obtain a partial analysis result;
  • Step S103 The terminal returns the partial analysis result to the control terminal corresponding to the cloud platform.
  • FIG. 2 is a method for analyzing big data. As shown in FIG. 2, the method is executed by any terminal in the cloud platform. The method is shown in the hardware architecture shown in FIG. The network topology is implemented. The method is as shown in FIG. 2, and includes the following steps:
  • Step S201 The control terminal receives the data quantity of the big data to be analyzed and the analysis instruction.
  • Step S202 The control terminal parses the analysis instruction to determine whether the big data can be split. If the data can be split, the storage address of the big data to be analyzed is split into multiple partial storage addresses.
  • Step S203 the control terminal acquires the load value of the analysis terminal and the number of transmission hops
  • Step S204 The control terminal distributes the plurality of partial storage addresses and the analysis instruction to the analysis terminal according to the transmission hop count and the load value.
  • Step S205 The analysis terminal extracts part of the data corresponding to the partial storage address, and analyzes the partial data according to the analysis instruction to obtain a partial result, and returns the partial result to the control terminal.
  • Step S206 The control terminal combines the partial results to obtain a final result of the big data to be analyzed.
  • the technical solution provided by the present application splits the big data to be analyzed into a plurality of analysis processing terminals to perform partial processing results in parallel, and the control terminal combines the partial processing results to obtain a final result, and the processor does not need to control the terminal.
  • the big data to be analyzed is sent, and only the storage address of the big data to be analyzed can be split to realize the splitting of the big data, so that the data transmission between the cloud platforms only needs to transmit the corresponding storage address, without large transmission. Data, reducing the amount of transmission, so it has the advantage of short calculation time and improved user experience.
  • the implementation method of the foregoing step S206 may be specifically:
  • the control terminal determines a splitting order of the plurality of partial storage addresses, and combines the partial results according to the splitting order to obtain a final result.
  • the implementation method of the foregoing step S202 may specifically be:
  • analysis instruction is a matrix multiplication matrix operation, determining that the data to be analyzed is detachable data, if the analysis instruction is an addition operation, determining that the data to be analyzed is non-separable data.
  • the foregoing method may further include:
  • control terminal selects the analysis terminal with the smallest load value to process the big data to be processed.
  • FIG. 5 provides an analysis system for big data, where the system includes: a control terminal 501 and an analysis terminal 502;
  • control terminal configured to receive a data quantity of the big data to be analyzed and an analysis instruction; analyze the analysis instruction to determine whether the big data can be split, and if the data can be split, split the storage address of the big data to be analyzed into a plurality of partial storage addresses; obtaining a load value of the analysis terminal and a transmission hop count; distributing the plurality of partial storage addresses and the analysis instruction to the analysis terminal according to the transmission hop count and the load value;
  • the analysis terminal is configured to extract part of the data corresponding to the part of the storage address, and analyze and process the part of the data according to the analysis instruction to obtain a partial result, and return the partial result to the control terminal;
  • the control terminal is further configured to obtain a final result of the big data to be analyzed according to the partial result.
  • the technical solution provided by the present application splits the big data to be analyzed into a plurality of analysis processing terminals to perform partial processing results in parallel, and the control terminal combines the partial processing results to obtain a final result, and the processor does not need to control the terminal.
  • the big data to be analyzed is sent, and only the storage address of the big data to be analyzed can be split to realize the splitting of the big data, so that the data transmission between the cloud platforms only needs to transmit the corresponding storage address, without large transmission. Data, reducing the amount of transmission, so it has the advantage of short calculation time and improved user experience.
  • FIG. 6 is a block diagram showing a partial structure of a terminal provided by an embodiment of the present application.
  • the server includes: a radio frequency (RF) circuit 910, a memory 920, an input unit 930, a sensor 950, an audio circuit 960, and wireless fidelity (Wireless).
  • RF radio frequency
  • Fidelity, WiFi Fidelity, WiFi
  • application processor AP980 application processor AP980 and power supply 990 and other components.
  • FIG. 6 does not constitute a limitation to the smart device, and may include more or less components than those illustrated, or some components may be combined, or different component arrangements.
  • the input unit 930 can be configured to receive input numeric or character information and to generate key signal inputs related to user settings and function control of the smart device.
  • the input unit 930 may include a touch display screen 933, a stylus 931, and other input devices 932.
  • the input unit 930 can also include other input devices 932.
  • other input devices 932 may include, but are not limited to, one or more of physical buttons, function keys (such as volume control buttons, switch buttons, etc.), trackballs, mice, joysticks, and the like.
  • the AP 980 is a control center for smart devices that connects various portions of the entire smart device using various interfaces and lines, by executing or executing software programs and/or modules stored in the memory 920, and invoking data stored in the memory 920, executing The intelligent device's various functions and processing data, so that the smart device is monitored as a whole.
  • the AP 980 may include one or more processing units; optionally, the AP 980 may integrate an application processor and a modem processor, where the application processor mainly processes an operating system, a user interface, an application, and the like, and the modulation solution The processor mainly handles wireless communication. It can be understood that the above modem processor may not be integrated into the AP 980.
  • the AP980 can be integrated with the face recognition module. Of course, in the actual application, the face recognition module can also be separately set or integrated in the camera 770. The face recognition module shown in FIG. 6 is integrated in the AP980. example.
  • memory 920 can include high speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
  • the RF circuit 910 can be used for receiving and transmitting information.
  • RF circuit 910 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low) Noise Amplifier, LNA), duplexer, etc.
  • RF circuitry 910 can also communicate with the network and other devices via wireless communication.
  • the above wireless communication may use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (General Packet) Radio Service, GPRS), Code Division Multiple Access (Code) Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), E-mail, Short Message Service (Short Messaging) Service, SMS), etc.
  • GSM Global System of Mobile communication
  • General Packet General Packet Radio Service
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • SMS Short Message Service
  • the smart device may also include at least one type of sensor 950, such as a light sensor, a motion sensor, a proximity sensor, and other sensors.
  • the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the touch display screen according to the brightness of the ambient light, and the proximity sensor can turn off the touch display when the mobile phone moves to the ear. And / or backlight.
  • the accelerometer sensor can detect the magnitude of acceleration in all directions (usually three axes). When it is stationary, it can detect the magnitude and direction of gravity.
  • Proximity sensor can be used to detect the distance between the phone and the user.
  • Other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like that can be configured in the mobile phone are not described herein.
  • An audio circuit 960, a speaker 961, and a microphone 962 can provide an audio interface between the user and the smart device.
  • the audio circuit 960 can transmit the converted electrical data of the received audio data to the speaker 961 for conversion to the sound signal by the speaker 961; on the other hand, the microphone 962 converts the collected sound signal into an electrical signal by the audio circuit 960. After receiving, it is converted into audio data, and then the audio data is played by the AP 980, sent to the other mobile phone via the RF circuit 910, or the audio data is played to the memory 920 for further processing.
  • WiFi is a short-range wireless transmission technology
  • the mobile phone can help users to send and receive emails, browse web pages, and access streaming media through the WiFi module 970, which provides users with wireless broadband Internet access.
  • FIG. 6 shows the WiFi module 970, it can be understood that it does not belong to the essential configuration of the smart device, and can be omitted as needed within the scope of not changing the essence of the application.
  • the smart device also includes a power supply 990 (such as a battery or a power module) that supplies power to various components.
  • a power supply 990 such as a battery or a power module
  • the power supply can be logically connected to the AP980 through a power management system to manage charging, discharging, and power management through the power management system.
  • the embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, the computer program causing the computer to perform some or all of the steps of any of the methods described in the foregoing method embodiments.
  • the computer includes a terminal device.
  • the embodiment of the present application further provides a computer program product, comprising: a non-transitory computer readable storage medium storing a computer program, the computer program being operative to cause a computer to perform the operations as recited in the foregoing method embodiments Part or all of the steps of either method.
  • the computer program product can be a software installation package, the computer including a terminal device.
  • the steps of the method or algorithm described in the embodiments of the present application may be implemented in a hardware manner, or may be implemented by a processor executing software instructions.
  • Software instructions can be composed of corresponding software modules, which can be stored in random access memory (Random) Access Memory, RAM), Flash, Read Only Memory (Read Only Memory, ROM), Erasable Programmable Read Only Memory (Erasable) Programmable ROM (EPROM), electrically erasable programmable read only memory (EEPROM), registers, hard disk, removable hard disk, compact disk read only (CD-ROM) or any other form of storage medium known in the art.
  • An exemplary storage medium is coupled to the processor to enable the processor to read information from, and write information to, the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and the storage medium can be located in an ASIC.
  • the ASIC can be located in an access network device, a target network device, or a core network device.
  • the processor and the storage medium may also exist as discrete components in the access network device, the target network device, or the core network device.
  • the functions described in the embodiments of the present application may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • software it may be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer instructions.
  • the processes or functions described in accordance with embodiments of the present application are generated in whole or in part.
  • the computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable device.
  • the computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be from a website site, computer, server or data center By wire (eg coaxial cable, fiber optic, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (eg infrared, wireless, microwave, etc.) to another website, computer, server or data center.
  • the computer readable storage medium can be any available media that can be accessed by a computer or a data storage device such as a server, data center, or the like that includes one or more available media.
  • the available media may be magnetic media (eg, floppy disk, hard disk, magnetic tape), optical media (eg, digital video disc (Digital) Video Disc, DVD)), or semiconductor media (for example, solid state drive (Solid State Disk, SSD)).

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

L'invention concerne un procédé et un système d'analyse de mégadonnées. Le procédé comprend les étapes suivantes : un terminal de commande reçoit une taille de données de mégadonnées à analyser et une instruction d'analyse (S201); le terminal de commande analyse l'instruction d'analyse pour déterminer si les mégadonnées peuvent être divisées et, si tel est le cas, diviser une adresse de stockage des mégadonnées à analyser en de multiples adresses de stockage partielles (S202); le terminal de commande acquiert une valeur de charge et un nombre de sauts de transmission d'un terminal d'analyse (S203); et le terminal de commande distribue, en fonction du nombre de sauts de transmission et de la valeur de charge, les multiples adresses de stockage partielles et l'instruction d'analyse au terminal d'analyse (S204). La solution décrite ci-dessus présente l'avantage d'offrir une expérience utilisateur améliorée.
PCT/CN2018/072989 2018-01-17 2018-01-17 Procédé et système d'analyse de mégadonnées WO2019140567A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/072989 WO2019140567A1 (fr) 2018-01-17 2018-01-17 Procédé et système d'analyse de mégadonnées

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2018/072989 WO2019140567A1 (fr) 2018-01-17 2018-01-17 Procédé et système d'analyse de mégadonnées

Publications (1)

Publication Number Publication Date
WO2019140567A1 true WO2019140567A1 (fr) 2019-07-25

Family

ID=67300930

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/072989 WO2019140567A1 (fr) 2018-01-17 2018-01-17 Procédé et système d'analyse de mégadonnées

Country Status (1)

Country Link
WO (1) WO2019140567A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113630408A (zh) * 2021-08-03 2021-11-09 Oppo广东移动通信有限公司 数据处理方法、装置、存储介质及服务器
CN113821386A (zh) * 2020-06-19 2021-12-21 顺丰科技有限公司 性能测试方法、装置、网络设备及计算机可读存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799808A (zh) * 2009-02-10 2010-08-11 中国移动通信集团公司 一种数据处理方法及其系统
US20110202929A1 (en) * 2010-02-16 2011-08-18 Stephen Schleimer Method and system for parallelizing database requests
US20120167103A1 (en) * 2010-12-23 2012-06-28 Electronics And Telecommunications Research Institute Apparatus for parallel processing continuous processing task in distributed data stream processing system and method thereof
CN104102475A (zh) * 2013-04-11 2014-10-15 腾讯科技（深圳）有限公司 分布式并行任务处理的方法、装置及系统
CN105740085A (zh) * 2014-12-11 2016-07-06 华为技术有限公司 容错处理方法及装置
CN105740063A (zh) * 2014-12-08 2016-07-06 杭州华为数字技术有限公司 一种数据处理方法及装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101799808A (zh) * 2009-02-10 2010-08-11 中国移动通信集团公司 一种数据处理方法及其系统
US20110202929A1 (en) * 2010-02-16 2011-08-18 Stephen Schleimer Method and system for parallelizing database requests
US20120167103A1 (en) * 2010-12-23 2012-06-28 Electronics And Telecommunications Research Institute Apparatus for parallel processing continuous processing task in distributed data stream processing system and method thereof
CN104102475A (zh) * 2013-04-11 2014-10-15 腾讯科技（深圳）有限公司 分布式并行任务处理的方法、装置及系统
CN105740063A (zh) * 2014-12-08 2016-07-06 杭州华为数字技术有限公司 一种数据处理方法及装置
CN105740085A (zh) * 2014-12-11 2016-07-06 华为技术有限公司 容错处理方法及装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113821386A (zh) * 2020-06-19 2021-12-21 顺丰科技有限公司 性能测试方法、装置、网络设备及计算机可读存储介质
CN113630408A (zh) * 2021-08-03 2021-11-09 Oppo广东移动通信有限公司 数据处理方法、装置、存储介质及服务器
CN113630408B (zh) * 2021-08-03 2023-06-16 Oppo广东移动通信有限公司 数据处理方法、装置、存储介质及服务器

Similar Documents

Publication Publication Date Title
US10959124B2 (en) Uplink data transmission method, terminal, network side device and system
US20130110992A1 (en) Electronic device management using interdomain profile-based inferences
CN107204989B (zh) 广告拦截方法、终端、服务器和存储介质
CN110069715B (zh) 一种信息推荐模型训练的方法、信息推荐的方法及装置
CN105740263B (zh) 页面显示方法和装置
WO2018219119A1 (fr) Procédé de commande de fonctions d'application, et produit associé
US20180253450A1 (en) Method and apparatus for processing timedly-published data
CN110781373A (zh) 榜单更新方法、装置、可读介质和电子设备
JP2023512551A (ja) 予測情報を生成する方法、装置、電子機器及びコンピュータ可読媒体
CN113034189A (zh) 归因方法、装置、计算机设备和存储介质
WO2019140567A1 (fr) Procédé et système d'analyse de mégadonnées
CN111359210B (zh) 一种数据处理方法、装置、电子设备以及存储介质
WO2019072179A1 (fr) Procédé et appareil de commande d'exécution d'application
CN108279978A (zh) 大数据的计算方法及系统
CN116596202A (zh) 工单处理方法、相关装置及存储介质
CN109739409B (zh) 一种批处理的方法、装置以及终端设备
CN110019648B (zh) 一种训练数据的方法、装置及存储介质
WO2019140565A1 (fr) Procédé et système de gestion d'instructions de stockage en nuage
CN115589432A (zh) 消息推送管理方法、装置、介质及设备
WO2019140577A1 (fr) Procédé et système de calcul de mégadonnées
US20220197939A1 (en) Image-based search method, server, terminal, and medium
CN110322039B (zh) 一种点击率预估方法、服务器及计算机可读存储介质
WO2019140578A1 (fr) Procédé et système de gestion d'instruction de sauvegarde de données
CN110619079A (zh) 用于推送信息的方法和装置
CN115801308B (zh) 数据处理方法、相关装置及存储介质

Legal Events

Date Code Title Description
NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 19/11/2020)

122 Ep: pct application non-entry in european phase

Ref document number: 18901075

Country of ref document: EP

Kind code of ref document: A1