CN112905332A - Method for realizing English PDF online rapid translation based on LVS load balancing Django architecture - Google Patents

Abstract

The invention discloses a method for realizing on-line quick translation of English PDF based on an LVS load balancing Django architecture, which comprises the following steps: s10, building a load balancing server architecture by using LVS and NGINX technologies, and realizing user access limitation by using Lua glue language on NGINX; s20, building a Django application server framework to receive user data and return the user data; and S30, storing and extracting the master-slave cluster of the Redis database, storing data by the Redis, and returning data required by the user. The invention can realize load balance and easily realize more than fifty thousand concurrent users, thereby not only meeting the requirements of a large number of users, but also accelerating the translation speed of the users and improving the user experience.

Description

Method for realizing English PDF online rapid translation based on LVS load balancing Django architecture

Technical Field

The invention belongs to the field of databases, and particularly relates to a method for realizing on-line quick translation of English PDF (portable document format) based on an LVS (virtual server) load balancing Django architecture.

Background

With the advent of the 5G era, more and more network users are available, and the pressure resistance of the server becomes more and more important, for example, a 12306 ticket purchasing website and a Taobao Shuang11 with hundreds of thousands of millions of users online at the same time, under the high concurrency condition, one server cannot meet all requirements, and no host bears the great pressure, no matter the hardware and software conditions are not met.

It is very important to select a good server architecture, and LVS (Linux virtual server) + NGINX (engine x, high performance HTTP and reverse proxy web server) architecture is used, where LVS is used as a load balancing server and NGINX is used as a reverse proxy server. In order to reduce unnecessary resource overhead and have a forwarding function, a Lua language is used, the Lua is a lightweight and small scripting language, the Lua language is written in a standard C language and is opened in a source code form, the Lua language can be embedded into a scripting language in NGINX, complex control can be realized by combining with the NGINX, and the running speed is greatly improved. The Python language has recently become more popular, and as a mature application server framework for Python, Django is a good choice, and Django is an open source Web application framework. The data interaction can not be carried out without databases, such as relational databases of MySQL, Oracle and the like, and non-relational databases of Redis, Memcached and the like. Combining the above can solve some bottleneck limitations and solve some specific problems.

Disclosure of Invention

In view of the technical problems, the invention provides a method for realizing on-line quick translation of English PDF based on an LVS load balancing Django architecture.

The method comprises the following steps:

s10, building a load balancing server architecture by using LVS and NGINX technologies, and realizing user access limitation by using Lua glue language on NGINX;

s20, building a Django application server framework to receive user data and return the user data;

and S30, storing and extracting the master-slave cluster of the Redis database, storing data by the Redis, and returning data required by the user.

Preferably, the S10 includes the steps of:

s11, building an LVS server and synchronizing a request forwarding strategy;

s12, building an NGINX server cluster, binding the IP operated by NGINX with the virtual IP of the LVS server, and utilizing Lua to carry out restriction operation on specific IP or user access in a configuration file, thereby realizing the complex control of NGINX forwarding.

Preferably, the S20 includes the steps of:

s21, layout front end interface: selecting HTML or PHP, beautifying the interface of a user for uploading files, selecting local PDF and uploading to a background server; the user response is rendered on a webpage and is stored to the local of the user through the webpage or directly to finish the response;

s22, connecting a Redis (Remote Dictionary Server) cluster with Django, and configuring interactive communication between Django and Redis;

s23, configuring uWSGI, realizing the connection of NGINX and uWSGI, and communicating the NGINX and uWSGI by using soket and a specific port; uWSGI is a Web server that implements WSGI protocols, uWSGI, http and other protocols.

S24, the crawler program crawls the English word library and stores high-frequency words into a Redis database according to the occurrence frequency.

Preferably, the S30 includes the steps of:

s31, storing data, wherein after connection with Redis test is successfully completed, the Redis storage structure is in a key/value format, English vocabulary is stored as key, and Chinese translation is stored as value;

and S32, data extraction, namely finding a specific value Chinese from a database according to a word requested by a user, responding to the data and rendering the data on a page, and Redis realizes different functions according to needs through a sentinel model.

The invention has the following beneficial effects:

1. a translation server is built by using a Python web framework Django and can be used with other cache systems, so that the loading speed of the page is increased. The advantages of Python in artificial intelligence and statistics are utilized, data can be optimized, data screening is facilitated, a voice reading library is integrated with Python, and one-time reading and one-time translation can be realized;

2. the bottleneck of NGINX load balancing is broken, concurrency is improved, millions of users can access concurrency by utilizing the advantages of the LVS, and the pressure of a single host is greatly reduced;

3. list storage and memory running speed are realized by using a Redis storage structure, and traditional I/O interaction is greatly saved;

4. redis can build a cluster and master-slave structure, so that the defect that a server cannot work due to the fact that a database is hung is avoided, and the provided persistence can greatly reduce the harm caused by the downtime of the server.

Drawings

Fig. 1 is a flowchart illustrating steps of a method for implementing online quick translation of an english PDF based on an LVS load balancing Django architecture according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating steps of a method for implementing online quick translation of an english PDF based on an LVS load balancing Django architecture according to another embodiment of the present invention;

fig. 3 is a load balancing server architecture diagram for implementing an online quick english PDF translation method based on an LVS load balancing Django architecture according to an embodiment of the present invention;

fig. 4 is a database cluster architecture diagram of a server framework connection based on the LVS load balancing Django architecture for implementing the english PDF online fast translation method according to an embodiment of the present invention;

fig. 5 is a communication architecture diagram of a server and a server application framework for implementing an online quick translation method of an english PDF based on an LVS load balancing Django architecture according to an embodiment of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Example 1

Referring to fig. 1, a flow chart of steps of a method for realizing on-line quick translation of an English PDF based on an LVS load balancing Django architecture, S10, a load balancing server architecture is built by using LVS and NGINX technologies, and Lua glue language is applied to NGINX to realize user access limitation;

s20, building a Django application server framework to receive user data and return the user data;

and S30, storing and extracting the master-slave cluster of the Redis database, storing data by the Redis, and returning data required by the user.

Detailed description of the preferred embodiments

S10 includes the steps of:

s11, building an LVS server and synchronizing a request forwarding strategy; in the Linux system, a Virtual IP (VIP) is established, and in a specific Nginx server, an IP to be looped is set, namely, the address of the VIP is set on a lo network card. Adding the virtual IP to the route, and deploying the connected server cluster for the back according to specific requirements;

the LVS selects a DR (direct routing) mode, the LVS can use two network cards to configure different IPs, the IP load balancing technology of the LVS is realized through an IPVS module, the IPVS is core software of the LVS cluster system, and the LVS mainly has the following functions: and installing the Server on the local Server, and simultaneously virtually simulating an IP address on the local Server, wherein the user has to access the Server through the virtual IP address. This Virtual IP is commonly referred to as the VIP of the LVS, the Virtual IP. The local IP is connected to the following NGINX, which is connected to the following Django. The local IP and the NGINX are necessarily in the same network segment because a direct routing mode is adopted, namely the local IP and the NGINX directly communicate through a Mac address without passing through the IP;

s12, building an NGINX server cluster, binding the IP operated by NGINX with the virtual IP of the LVS server, and utilizing Lua to carry out restriction operation on specific IP or user access in a configuration file, thereby realizing the complex control of NGINX forwarding.

Configuring the route with the local IP at the host stage of the NGINX server, and proxying the IP address of the following Django in the NGINX configuration file, wherein the uWSGI protocol is used, and the user is limited to be accessed or the content is returned in the configuration file through the Lua language. Lua as a scripting language in NGINX may act as glue to join other content. Lua can also configure databases such as Redis, MySQL and the like, so that the efficiency is greatly improved to a certain extent. NGINX can realize that different agents return the dynamic files directly through NGINX and Lua by accessing the static files and the dynamic files through users, and the efficiency is greatly improved by proxying the static files to Django.

S20 includes the steps of:

s21, layout front end interface: selecting HTML or PHP, beautifying the interface of a user for uploading files, selecting local PDF and uploading to a background server; the user response is rendered on a webpage and is stored to the local of the user through the webpage or directly to finish the response;

since the server is definitely divided into the front end and the back end, the front end directly interacts with the user, so the quality of the front end page layout directly influences the user experience. The method mainly realizes uploading of the PDF file and saving of the file to the local after the translation is finished. Django needs to judge whether the PDF file is a PDF file when receiving a user request, if not, the page is returned to 404, if the PDF file is an English file, then, whether the PDF file is an English file is judged, and if not, the user is reminded of uploading errors. In the Django mode, the MVT mode, the relative path of a front-end template must be added in a setting file, and the path is rewritten correctly for a jump URL;

s22, connecting the Redis cluster with Django, and configuring interactive communication between Django and Redis; django can connect various databases of MySQL, Memcached and the like. And IP and port number are started through Redis by modifying the configuration file. Redis is chosen here because it is an in-memory database, which interacts differently with the I/O interface. And Redis provides persistence, and Redis data can be stored in a hard disk, so that the next use is facilitated. Redis realizes session sharing, because the HTTP protocol is stateless, the session of the user is also stored in the database in consideration of safety and convenience, and the user information is available no matter which server is allocated;

s23, configuring uWSGI, realizing the connection of NGINX and uWSGI, and communicating the NGINX and uWSGI by using soket and a specific port; ensuring that this port is not adopted by other programs. This may send a request to access NGINX to Django via the uWSGI protocol. The dynamic request is used for receiving the dynamic request forwarded by the front-end server, processing the dynamic request and then sending the dynamic request to the web application program. After receiving the request, the uWSGI server analyzes and calls back Django application according to the wsgi protocol, the Django application performs tasks such as Redis reading and the like according to the request, and then reversely returns NGINX and LVS;

s24, crawling an English word library by a crawler program, storing high-frequency words into a Redis database according to occurrence frequency, wherein the high-frequency words are used as a server which must have a data source for a user to access, and if the speed of calling other api interfaces is too slow and is not suitable for being used as a high concurrency server, I select to crawl public data through the crawler and store the public data into the Redis. By virtue of persistence, the latest data can be updated in real time, so that a large number of data sources are obtained.

In order to provide a rich data source for the local, a Python crawler is required to obtain a large number of english words and determine whether to repeat, and if so, one is reserved. The case of word-ambiguity is left with the download in the form of an array list. This is also one of the advantages of Redis, which can store list form.

S30 includes the steps of:

s31, storing data, wherein after connection with Redis test is successfully completed, the Redis storage structure is in a key/value format, English vocabulary is stored as key, and Chinese translation is stored as value;

and S32, data extraction, namely finding a specific value Chinese from a database according to a word requested by a user, responding to the data and rendering the data on a page, and Redis realizes different functions according to needs through a sentinel model. Three Redis database clusters are built, one master database and two slave databases are built, and the three databases can normally communicate.

Referring to fig. 2, NGINX is used alone as a load balancing server, and although the internal scheme adopts epoll to adopt asynchronous non-blocking I/O multiplexing, the load balancing server also has a limit value, so that ten thousand concurrences are reached on a host computer, packet loss occurs, and webpage loading is slow. In order to solve the problem of realizing hundreds of thousands of concurrences, a load balancing server is built by matching with the LVS, and the limitation on the number of the concurrences of the users is greatly reduced by using NGINX as a server cluster. NGINX can also be used as a load balancing server, and a Django server cluster is used later, so that the pressure of a single server can be greatly reduced, and the work is not influenced when one server cluster is down, because the server clusters do the same work and have the same function.

Referring to fig. 3, each NGINX is followed by a web framework to provide services, where one NGINX is followed by a plurality of frameworks, which can be balanced to reduce the burden and avoid the consequence that an error overall fails to work.

Referring to fig. 4, a Redis database cluster is used, stored values are distributed to a specific database through a hash slot special algorithm, so that the access pressure of one database can be reduced, the Redis cluster needs at least three, and can be optimized into a master-slave structure, so that six Redis databases are needed, each master database is matched with one slave database, data is written into the master database, data is read from the slave database, the read-write separation improves the efficiency, and when the master server makes a mistake, the slave server can be lifted to the master server without influencing the work.

Referring to fig. 5, which is a server and server framework communication principle and protocol, uWSGI acts like a tie to connect the two. Therefore, selection is provided for structure diversification, and developers can select different frameworks according to different advantages of different frameworks, so that the efficiency is greatly improved.

It is to be understood that the exemplary embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the present invention have been described with reference to the accompanying drawings, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims.

Claims (4)

1. A method for realizing on-line quick translation of English PDF based on LVS load balancing Django architecture is characterized by comprising the following steps:

s10, building a load balancing server architecture by using LVS and NGINX technologies, and realizing user access limitation by using Lua glue language on NGINX;

s20, building a Django application server framework to receive user data and return the user data;

and S30, storing and extracting the master-slave cluster of the Redis database, storing data by the Redis, and returning data required by the user.

2. The method for realizing online rapid translation of English PDF based on LVS load balancing Django architecture as claimed in claim 1, wherein said S10 comprises the following steps:

s11, building an LVS server and synchronizing a request forwarding strategy;

s12, building an NGINX server cluster, binding the IP operated by NGINX with the virtual IP of the LVS server, and utilizing Lua to carry out restriction operation on specific IP or user access in a configuration file, thereby realizing the complex control of NGINX forwarding.

3. The method for realizing online rapid translation of English PDF based on LVS load balancing Django architecture as claimed in claim 1, wherein said S20 comprises the following steps:

s21, layout front end interface: selecting HTML or PHP, beautifying the interface of a user for uploading files, selecting local PDF and uploading to a background server; the user response is rendered on a webpage and is stored to the local of the user through the webpage or directly to finish the response;

s22, connecting the Redis cluster with Django, and configuring interactive communication between Django and Redis;

s23, configuring uWSGI, realizing the connection of NGINX and uWSGI, and communicating the NGINX and uWSGI by using soket and a specific port;

s24, the crawler program crawls the English word library and stores high-frequency words into a Redis database according to the occurrence frequency.

4. The method for realizing online rapid translation of English PDF based on LVS load balancing Django architecture as claimed in claim 1, wherein said S30 comprises the following steps:

s31, storing data, wherein after connection with Redis test is successfully completed, the Redis storage structure is in a key/value format, English vocabulary is stored as key, and Chinese translation is stored as value;

and S32, data extraction, namely finding a specific value Chinese from a database according to a word requested by a user, responding to the data and rendering the data on a page, and Redis realizes different functions according to needs through a sentinel model.

Images