CN112769711A - Data processing method and system - Google Patents

Data processing method and system Download PDF

Info

Publication number
CN112769711A
CN112769711A CN201911062020.0A CN201911062020A CN112769711A CN 112769711 A CN112769711 A CN 112769711A CN 201911062020 A CN201911062020 A CN 201911062020A CN 112769711 A CN112769711 A CN 112769711A
Authority
CN
China
Prior art keywords
file data
data
offline
remote network
file
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911062020.0A
Other languages
Chinese (zh)
Inventor
虞陆平
顾晓倩
蔡益平
左仁珂
谭易炳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Strong Education Technology Co ltd
Original Assignee
Hangzhou Strong Education Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Strong Education Technology Co ltd filed Critical Hangzhou Strong Education Technology Co ltd
Priority to CN201911062020.0A priority Critical patent/CN112769711A/en
Publication of CN112769711A publication Critical patent/CN112769711A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/24Traffic characterised by specific attributes, e.g. priority or QoS
    • H04L47/2408Traffic characterised by specific attributes, e.g. priority or QoS for supporting different services, e.g. a differentiated services [DiffServ] type of service
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • G09B5/06Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • G09B5/065Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
    • GPHYSICS
    • G09EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09BEDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00Electrically-operated educational appliances
    • G09B5/08Electrically-operated educational appliances providing for individual presentation of information to a plurality of student stations
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00Traffic control in data switching networks
    • H04L47/10Flow control; Congestion control
    • H04L47/24Traffic characterised by specific attributes, e.g. priority or QoS
    • H04L47/2416Real-time traffic
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/01Protocols
    • H04L67/06Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/56Provisioning of proxy services
    • H04L67/568Storing data temporarily at an intermediate stage, e.g. caching
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00Network arrangements or protocols for supporting network services or applications
    • H04L67/50Network services
    • H04L67/60Scheduling or organising the servicing of application requests, e.g. requests for application data transmissions using the analysis and optimisation of the required network resources
    • H04L67/63Routing a service request depending on the request content or context

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention provides a data processing method and a data processing system in a weak network environment. And by adding an offline cache server specially processing the file data, loading part of data in advance. The concurrency control is added when the same file data is downloaded, and the concurrency uploading is controlled through queuing when the file is uploaded, so that the bandwidth requirement is reduced, and more online users can be supported to browse and watch the application data at the same time by using a small bandwidth.

Description

Data processing method and system
Technical Field
The invention relates to the technical field of network data processing, in particular to a data processing method and system in a weak network environment.
Background
Education informatization refers to a process of comprehensively and deeply applying modern information technology to promote education reform and development in the education field (education management, education teaching and education scientific research). The technology is characterized by digitalization, networking, intellectualization and multimedia, and the basic characteristics are opening, sharing, interaction and cooperation. The education modernization is promoted by the education informatization, the traditional mode is changed by the information technology, and most of the Internet education systems in the market at present are provided with real-time live broadcast classrooms by carrying software application platforms by hardware equipment such as computers, mobile phones or flat plates and the like. One of the live scenes is a flat classroom scene: the teacher and the students attend class in a flat classroom, the teacher starts live broadcast in the classroom, uploads teaching files and initiates real-time questions and answers or instant tasks, the students download the teaching files through a flat computer, receive the questions and answers tasks to answer instantly, and upload answer files.
The following problems arise in such teaching activities:
1. the problem of slow downloading caused by large downloading flow is as follows:
the PPT or the video needs to be played when a teacher live broadcasts courseware, student terminals need to be watched synchronously, when the teacher pages courseware or plays the video, a student end is triggered to download files simultaneously, the instantaneous flow is high, for example, one file is generally 10KB, 1000 students are 100MB, the corresponding bandwidth requirement is 800Mbps, however, most schools must use an education network, the private installation of broadband is not allowed, the broadband cost of part of areas is high (such as coastal cities like Shenzhe, Guangzhou and the like), therefore, the bandwidth of the schools is very limited, generally about 100 Mbps, the bandwidth is insufficient, the download failure or the download is slow, and the overall class effect is influenced.
2. Too much uploading flow leads to slow uploading and unstable classroom:
teachers can initiate some tasks of doing questions when they are in class, students can upload files such as answers (picture formats) after completing the tasks, and due to the fact that completion time is almost short, broadband uplink flow of schools is small, class connection is easily interrupted, and class taking is affected.
3. The problem of bandwidth waste due to repeated downloading of files is as follows:
as the answers uploaded by the N students are in the classroom, when the teacher cries, the teacher needs to check the answers of the students and needs to download N files, and at the moment, the students also need to synchronize the pictures of the teacher, so that the N x N files can be downloaded. The number of active files is only N, but the traffic of N x (N +1) files is used.
The main solutions to solve the above problems in the current market are 3:
1. the 4G card is used for increasing school bandwidth or mobile equipment. According to the scheme, more cost is required to be invested in bandwidth by a school side, and the threshold of the school hardware environment using the live broadcast platform is improved.
2. The file compression algorithm is researched, so that the file transmission bandwidth cost is reduced: however, file compression algorithms are not uniform, the file compression effect is limited finally, the smaller the compression is, the harder the compression is, and the higher the cost is, and when the number of files is too large, even if the compression algorithm is good, if hundreds of people access together, the flow rate is inevitably increased, the compression needs calculation, the calculation needs hardware support, and the cost is increased.
3. Regionalized deployment of a whole set of live broadcast system: enterprises need to deploy a plurality of servers in each city or area, and the cost investment of hardware, personnel, maintenance, management and the like is greatly increased.
Disclosure of Invention
In order to solve the above problems, the present invention discloses a data processing method and system, which is faster in uploading or downloading link, greatly reduces the requirement of bandwidth, reduces the traffic cost, reduces the operation cost, and makes the real-time live broadcast more convenient for popularization, and the invention is realized as follows:
a data processing method, comprising:
dividing application data into three data types of service data, signaling data and file data, wherein the service data and the signaling data are managed by a remote network platform, and the file data is managed by the remote network platform and an offline cache server together;
when the user is online, the application selects different management modules to respond to the application data request of the user according to the network condition, the data type and the access priority of the user in the running process.
And the user network condition comprises that the user network is an internal network or an external network.
Further, when a user operates file data in an intranet, the file data operation request is processed preferentially through the offline cache server, and if the offline cache server cannot meet the file data operation request, the file data operation request is transmitted to the remote network platform through the offline cache server; when the user network is an external network, all types of data are responded directly through the remote network platform.
Further, the file data operation request includes uploading file data and downloading file data.
Furthermore, when a user downloads file data of a remote network platform in an intranet, the offline cache server performs concurrency control on the file data, wherein the concurrency control means that each piece of file data is downloaded from the remote network platform to the offline cache server only once; when a plurality of downloading requests simultaneously require to download the same file data, only one request is effective, and other requests directly use the cache after one-time downloading is completed.
Further, when a user requests the remote network platform to download intranet file data from the extranet and the remote network platform does not retrieve the file data, the remote network platform notifies the offline penetration program client deployed in the intranet to upload the file data through the offline penetration program server deployed in the extranet, and after the file data is uploaded, the offline penetration program server transmits the file data to the remote network platform and the remote network platform transmits the file data to an extranet user.
Furthermore, the offline cache server downloads the file data to be used in a certain time from the remote network platform at regular time, and skips the downloaded file data.
The invention also discloses a data processing system, which is characterized by comprising:
the platform data preprocessing module is used for preparing file data required to be used within the stage time in advance; the offline cache agent module is used for downloading and caching the file data, and simultaneously provides a concurrency control function to prevent the same file data from being downloaded for multiple times; the offline uploading module is used for recording the content of the uploaded file data, continuously uploading the file data to a remote network platform at a low flow rate, and uploading the file data on a specified path by using the copy characteristic of the FASTDFS; the offline download management module is used for regularly pulling the prepared data in the platform data preprocessing module and downloading the file data in the data; the offline penetration program server is used for providing service for the offline penetration program client, and the service comprises the step of informing the offline penetration program client to upload specified file data; and the offline penetration program client module actively uploads the file data when receiving the notification of the offline penetration program server.
Furthermore, the offline uploading module records a file data list uploaded by a user, and uploads file data by using a thread pool background with a small number of threads, so that the situation that the file data uploading process does not occupy too much bandwidth is ensured.
The invention has the beneficial effects that: the method and the system for managing the live course application data in the weak network environment are used for classifying the application data, setting response rules, configuring the cache server and concurrently controlling to avoid the waste of bandwidth due to repeated downloading of files, so that a live course is more stable, the hardware cost is reduced and the practical effect is good.
Drawings
FIG. 1 is a schematic flow chart of a data processing method disclosed in the present invention;
FIG. 2 is a block diagram of a data processing system according to the present disclosure;
fig. 3 is a schematic diagram of a file data transmission flow in the embodiment of the present invention.
Detailed Description
So that the manner in which the above recited objects, features and advantages of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to the appended drawings
The present invention will be described in further detail with reference to the following embodiments. While specific use scenarios are set forth in the following description to provide a thorough understanding of the present invention, the present invention may be practiced in other scenarios than those described herein and similar generalization may be made by those skilled in the art without departing from the spirit of the present invention and, therefore, the present invention is not limited to the specific embodiments disclosed below.
As shown in FIG. 2, a schematic representation of a data processing system includes: the platform data preprocessing module is used for preparing file data required to be used within the stage time in advance; the offline cache agent module is used for downloading and caching the file data, and simultaneously provides a concurrency control function to prevent the same file data from being downloaded for multiple times; the offline uploading module is used for recording the content of the uploaded file data, continuously uploading the file data to a remote network platform at a low flow rate, and uploading the file data on a specified path by using the copy characteristic of the FASTDFS; the offline download management module is used for regularly pulling the prepared data in the platform data preprocessing module and downloading the file data in the data; the offline penetration program server is used for providing service for the offline penetration program client, and the service comprises the step of informing the offline penetration program client to upload specified file data; and the offline penetration program client module actively uploads the file data when receiving the notification of the offline penetration program server.
The file data transmission flow diagram shown in fig. 3 is explained in combination with the following scenarios:
scene one, teachers and students use the school online live-broadcast lessons:
1. teachers and students log on a remote network platform and select courses to enter a live classroom.
2. In a live classroom, when a teacher operates courseware, two parts of data are mainly used: file data and file data addresses. The file data address is returned through the remote network platform, the file data address is accessed through the offline cache server to download the file data, and when the file data does not exist in the offline cache server, the offline cache agent module can actively request the remote network platform to start downloading the file data.
3. After a teacher uploads ppt courseware (belonging to file data), fifty students can download the same ppt file data at the same time, in order to avoid the waste of downloading flow, the offline cache agent module only executes one operation of requesting a remote network platform to download the file data, other requests are set to be waiting, and after the offline cache agent module finishes downloading, the file data is cached, and the cached file data is responded to all requests.
4. The teacher client side can generate handwriting data in the blackboard writing process, the handwriting data can be sent to the remote network platform, the platform is sent to all the student client sides in a signaling data broadcasting mode, the student client sides receive the signaling data and reproduce the blackboard writing process of the teacher after analyzing the handwriting data, and therefore the purpose of synchronizing the desktop is achieved.
5. When the students submit exercises, file data uploading operation can be triggered, and some answers need to be stored in the form of pictures. When the students upload the pictures, the students firstly upload the pictures to the offline uploading module, and after receiving the uploaded pictures, the offline uploading module returns a picture address to the students and records the picture address. After receiving the picture address, the student forwards the picture address to the teacher through the remote network platform along with the submitted answer signaling data. In addition, after the offline uploading module records the picture addresses, the uploading task in the offline uploading module monitors the record change of the file data, and then the file data can be actively uploaded to the remote network platform, so that the offline uploading module is convenient for students outside schools to use.
6. In the above steps, the user name and password used during login, the course list displayed in the school timetable page, the courseware data when the teacher operates courseware, and the like all belong to the service data; page turning signaling data, handwriting signaling data and the like which are synchronously operated belong to the signaling data; the picture, ppt, word, etc. are file data.
Scene two, based on scene one, but several students are in class in the extranet environment outside the school:
1. enter the live classroom along with the teacher and the students in the school.
2. After receiving courseware page turning signaling data (managed by a remote network platform), school students directly download file data by accessing the remote network platform if the courseware data in the page turning signaling has a file data address, when the remote network platform does not have the file data, the remote network platform informs an external network offline penetration program server terminal to inform an internal network offline penetration program client terminal to upload the file data, and the internal network offline penetration program client terminal executes file uploading operation. And responding to the remote network platform through the extranet offline penetration program server, and returning the file to the out-of-school students by the remote network platform so as to realize synchronous in-school teacher client desktops of the out-of-school students. Other related signaling data are circulated through a remote network platform, and finally, teachers and students using an internal network in a school and students using an external network outside the school can synchronously live courses.

Claims (9)

1. A data processing method, comprising:
dividing application data into three data types of service data, signaling data and file data, wherein the service data and the signaling data are managed by a remote network platform, and the file data is managed by the remote network platform and an offline cache server together;
when the user is online, the application selects different management modules to respond to the file data request of the user according to the network condition, the data type and the access priority of the user in the running process.
2. The data processing method according to claim 1, wherein the user network condition comprises whether the user network is an intranet or an extranet.
3. The data processing method according to claim 2, wherein when the user operates the file data in the intranet, the file data operation request is preferentially processed through the offline cache server, and if the offline cache server cannot meet the file data operation request, the file data operation request is transmitted to the remote network platform by the offline cache server; when the user network is an external network, all types of data are responded directly through the remote network platform.
4. A data processing method according to claim 3, wherein the file data operation request includes uploading file data and downloading file data.
5. The data processing method according to claim 4, wherein when the user downloads the file data of the remote network platform in the intranet, the offline cache server performs concurrency control on the file data, wherein the concurrency control means that each file data is downloaded from the remote network platform to the offline cache server only once; when a plurality of downloading requests simultaneously require to download the same file data, only one request is effective, and other requests directly use the cache after one-time downloading is completed.
6. The data processing method as claimed in claim 5, wherein when the user requests the remote network platform from the external network to download the file data from the internal network, and the remote network platform does not retrieve the file data, the remote network platform will notify the offline client deployed in the internal network to upload the file data through the offline server deployed in the external network, and after uploading, the offline server will transmit the file data to the remote network platform, and the remote network platform will transmit the file data to the external network user.
7. A data processing method according to claim 1, wherein the offline cache server periodically downloads the file data to be used for a certain period of time from the remote network platform, and skips the downloaded file data.
8. A data processing system, comprising:
the platform data preprocessing module is used for preparing file data required to be used within the stage time in advance; the offline cache agent module is used for downloading and caching the file data, and simultaneously provides a concurrency control function to prevent the same file data from being downloaded for multiple times; the offline uploading module is used for recording the content of the uploaded file data, continuously uploading the file data to a remote network platform at a low flow rate, and uploading the file data on a specified path by using the copy characteristic of the FASTDFS; the offline download management module is used for regularly pulling the prepared data in the platform data preprocessing module and downloading the file data in the data; the offline penetration program server is used for providing service for the offline penetration program client, and the service comprises the step of informing the offline penetration program client to upload specified file data; and the offline penetration program client module actively uploads the file data when receiving the notification of the offline penetration program server.
9. The data processing system of claim 8, wherein the offline uploading module records a file data list uploaded by a user, and uploads the file data in the background of the thread pool using a small number of threads, so as to ensure that the process of uploading the file data does not occupy too much bandwidth.
CN201911062020.0A 2019-11-01 2019-11-01 Data processing method and system Pending CN112769711A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911062020.0A CN112769711A (en) 2019-11-01 2019-11-01 Data processing method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911062020.0A CN112769711A (en) 2019-11-01 2019-11-01 Data processing method and system

Publications (1)

Publication Number Publication Date
CN112769711A true CN112769711A (en) 2021-05-07

Family

ID=75692090

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911062020.0A Pending CN112769711A (en) 2019-11-01 2019-11-01 Data processing method and system

Country Status (1)

Country Link
CN (1) CN112769711A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114500514A (en) * 2022-02-14 2022-05-13 京东科技信息技术有限公司 File transmission method and device, electronic equipment and computer readable storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102739712A (en) * 2011-04-08 2012-10-17 中国移动通信集团贵州有限公司 Method and system for network cache data distribution
CN103118049A (en) * 2011-11-17 2013-05-22 中兴通讯股份有限公司 Method and system downloading files through network caching
CN104539436A (en) * 2014-12-22 2015-04-22 杭州施强网络科技有限公司 Lesson content real-time live broadcasting method and system
CN104780226A (en) * 2015-05-06 2015-07-15 成都麦子信息技术有限公司 Online education platform
WO2016115957A1 (en) * 2015-01-19 2016-07-28 张维加 Method and device for accelerating computers and intelligent devices for users and applications

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102739712A (en) * 2011-04-08 2012-10-17 中国移动通信集团贵州有限公司 Method and system for network cache data distribution
CN103118049A (en) * 2011-11-17 2013-05-22 中兴通讯股份有限公司 Method and system downloading files through network caching
CN104539436A (en) * 2014-12-22 2015-04-22 杭州施强网络科技有限公司 Lesson content real-time live broadcasting method and system
WO2016115957A1 (en) * 2015-01-19 2016-07-28 张维加 Method and device for accelerating computers and intelligent devices for users and applications
CN104780226A (en) * 2015-05-06 2015-07-15 成都麦子信息技术有限公司 Online education platform

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114500514A (en) * 2022-02-14 2022-05-13 京东科技信息技术有限公司 File transmission method and device, electronic equipment and computer readable storage medium
CN114500514B (en) * 2022-02-14 2023-12-12 京东科技信息技术有限公司 File transmission method and device for cloud storage, electronic equipment and storage medium

Similar Documents

Publication Publication Date Title
US10586296B2 (en) Facilitating diagnosis and correction of operational problems
CN111611434B (en) Online course interaction method and interaction platform
US9462017B1 (en) Meeting collaboration systems, devices, and methods
CN105047033A (en) Classroom cloud and education cloud collaborative teaching interaction system and method
CN104540026A (en) Lesson content video recoding and replaying method and system
CN104539436A (en) Lesson content real-time live broadcasting method and system
CN105306861A (en) Online teaching recording and playing method and system
CN205158690U (en) Teaching interaction system of classroom cloud and cooperation of education cloud
CN103473297A (en) Multimedia question-answering system and method based on intelligent mobile terminal
CN108737372B (en) Streaming media playback method, computer equipment and readable medium
CN108538118A (en) A kind of on-line education system
WO2019196577A1 (en) Streaming media playback method, server, client and computer device
US11575528B1 (en) Dynamic virtual meeting, online webinar and distance-based education systems and methods
CN112769711A (en) Data processing method and system
CN110730377A (en) Short video recording method and device based on interactive classroom
CN112216311B (en) Online classroom reappearing method, system, equipment and medium
CN110650159A (en) Multi-person interactive online teaching method, device and system
TWM631602U (en) Live teaching system
KR20220067402A (en) Interactive internet real-time lecture system and lecture method using the same
CN112150870A (en) Cloud classroom online education interaction system
CN114554239B (en) Teaching live broadcast system and method
US20120301863A1 (en) Interactive Learning Network
CN117221619B (en) Video caching method for examination video paper reading
Pullen Scaling up a distance education program in computer science
Syaifudin et al. The design of foreign language teaching software in school computer laboratory

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20210507

WD01 Invention patent application deemed withdrawn after publication