CN111459686B - Queue message storing and forwarding method, system and computer device with operating system - Google Patents
Queue message storing and forwarding method, system and computer device with operating system Download PDFInfo
- Publication number
- CN111459686B CN111459686B CN202010186264.6A CN202010186264A CN111459686B CN 111459686 B CN111459686 B CN 111459686B CN 202010186264 A CN202010186264 A CN 202010186264A CN 111459686 B CN111459686 B CN 111459686B
- Authority
- CN
- China
- Prior art keywords
- topic
- information table
- user
- storage device
- access request
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2282—Tablespace storage structures; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/548—Queue
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Transfer Between Computers (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method, a system and a computer device with an operating system for storing and forwarding queue messages, wherein the method for storing and forwarding the queue messages is executed by a computer or software and comprises the steps of receiving an access request for appointed contents and storing a Topic carried by the access request into a user access information table; the user access information table is used for modifying the priority of the Topic of the specified content in the user Topic priority table, and the specified content corresponding to the access request is fetched from the third storage device for storing the specified content by using the memory queue to respond to the main body initiating the access request to the specified content. By the queue message storing and forwarding method, the system and the computer device with the operating system, high concurrency response of a user to massive entity data is met, delay phenomena of massive Topic messages in the storing and forwarding process are eliminated, and calculation cost to physical equipment is reduced.
Description
Technical Field
The present invention relates to the field of computer data processing technologies, and in particular, to a method and a system for storing and forwarding queue messages, and a computer device with an operating system.
Background
Table storage (Tablestore) is a NoSQL multimodal database that is capable of providing massive structured data storage as well as fast query and analysis services. The distributed storage of the table store and the powerful indexing engine are capable of supporting PB level storage, tens of millions of TPS, and millisecond delayed service capabilities. In the big data age, the number of topics in the consumer queue model based on traditional producers becomes a sea level. In the scenes of popup window, streaming media file pushing, commodity recommendation webpage pushing, class generation, function generation, thread generation or process generation in a computer hardware device/software system and the like, each message type can be a Topic, and a plurality of sub-classifications and grandchild classifications exist in each Topic, and even more deep classifications exist.
For example, in a streaming media file pushing scenario, when a user performs video-on-demand on a video client, two categories of sports and variety are displayed in a visual interface of the video client, and often the sports category further includes sub-categories such as basketball and tennis, and the basketball sub-category further includes grandchild categories such as NBA and CBA, and the grandchild category of NBA may further include multiple lower categories such as NBA highlights and decade. Then there will be a particular producer and consumer for the queue task for which each hierarchical class corresponds. The module/body that generates the data is the Producer; while the module/body that processes the data is the Consumer (Consumer). A subscription schema refers to a message producer publishing a message into topic while multiple message consumers consume the message, and messages published to topic will be consumed by all subscribers. At present, in a transaction processing system or a computer program based on a message processing model, certain delay exists in the process of storing and forwarding a queue message, and the technical requirements on physical equipment are severe. More importantly, the queue message storing and forwarding method in the prior art cannot reasonably and accurately respond to the personalized requirements of the user.
Disclosure of Invention
The invention aims to disclose a queue message storing and forwarding method, a system and a computer device with an operating system, which are used for solving the defects of massive high-concurrency access requests in the prior art, and particularly for reducing delay existing in the process of storing and forwarding massive Topic messages in a transaction processing system or a computer program based on a message processing model, reducing the calculation cost of physical equipment and meeting the high-concurrency response of massive users to entity data corresponding to the Topic messages with different priorities.
To achieve the first object, the present invention provides a queue message store-and-forward method,
executed by a computer or software, comprising:
receiving an access request for specified content, and storing a Topic carried by the access request into a user access information table;
the user access information table is used for modifying the priority of the Topic of the specified content in the user Topic priority table, and the specified content corresponding to the access request is fetched from the third storage device for storing the specified content by using the memory queue to respond to the main body initiating the access request to the specified content.
As a further improvement of the invention, the user access information table is associated with a full-volume information table, so that specified content corresponding to the access request is determined according to the Topic carried by the access request through the full-volume information table and/or the user access information table.
As a further improvement of the invention, the user Topic priority table is associated with a Topic information table, so that task sequences with priority in the user Topic priority table are adjusted through the Topic information table;
the user access information table and/or the total information table are stored in a first storage device in a table storage form.
As a further improvement of the present invention, there is also included: pre-configuring recommended content and storing the recommended content in a Topic information table;
the Topic information table and/or the user Topic priority table are stored in a second storage means in a table storage form.
As a further improvement of the invention, the first storage device and the second storage device are mutually independent and controlled by the response end, and global maintenance is carried out on the full-volume information table in the first storage device and the Topic information table in the second storage device through the response end.
As a further improvement of the present invention, there is also included: the user access information table updates the recommended content based on the received access request to the specified content.
As a further refinement of the present invention, the memory queue forms a plurality of task sequences responsive to a body initiating an access request to specified content and comprising different priority orders.
As a further improvement of the present invention, the Topic information table and the user Topic priority table are based on a multithreaded timing task scheduling mechanism, and obtain multiple access requests concurrently, so as to capture the specified content corresponding to the access requests from the storage device storing the specified content through the memory queue through the multiple access requests and respond to the consumer.
As a further improvement of the present invention, there is also included:
the method comprises the steps that based on a mobile terminal, an access request for specified content is sent and displayed, a producer and a consumer are deployed, the producer issues the access request for the specified content to a user access information table, the consumer receives a Topic returned by a total information table or the user access information table in a first storage device to a main body initiating the access request for the specified content, and the specified content related to the returned Topic is based on the specified content;
the mobile terminal comprises an APP or a computer device embedded in a user interactive interface.
The specified content includes a streaming media file, an audio file, a text file, or a program.
As a further improvement of the present invention, the Topic information table includes: primary key ID, priority, load balancing partition key, topic code, topic name, user ID, topic information table creation time, and Topic information table update time; the user Topic priority table includes: primary key ID, priority, load balancing partition key, topic ID, user ID, message ID, user Topic priority table creation time, and user Topic priority table update time.
As a further improvement of the present invention, the user access information table includes: a primary key ID, a Topic ID, a user ID, a message ID, a user access information table creation time and a user access information table update time; the full amount information table includes: primary key ID, topic ID, message name, message content, full information table creation time, and full information table update time.
As a further improvement of the invention, the memory queue holds concurrent consumption of different priorities configured for the same body and/or different bodies of the specified content initiated access request, the concurrent consumption comprising at least one prioritized task sequence.
Based on the same inventive concept, the application also discloses a queue message store-and-forward system, comprising:
a first storage device for configuring a user access information table, a second storage device for configuring a user Topic priority table, a third storage device, a memory queue and a response end;
when an access request for the specified content is received, the Topic carried by the access request is stored in a user access information table, the priority of the Topic for the specified content in a user Topic priority table is modified through the user access information table, and the specified content corresponding to the access request is fetched from a third storage device for storing the specified content by using a memory queue to respond to a main body initiating the access request for the specified content.
As a further improvement of the present invention, there is also included: a recommendation template generation module;
the recommendation template generation module pre-configures recommendation contents and stores the recommendation contents in a Topic information table;
the Topic information table and/or the user Topic priority table are stored in a second storage means in a table storage form.
As a further improvement of the present invention, the first storage device is further configured with a full-volume information table, and the user access information table is associated to the full-volume information table, so that specified content corresponding to the access request is determined according to Topic carried by the access request through the full-volume information table and/or the user access information table; the second storage device is further configured with a Topic information table, and the user Topic priority table is associated to the Topic information table, so that task sequences with priorities in the user Topic priority table are adjusted through the Topic information table.
As a further improvement of the invention, the first storage device and the second storage device are mutually independent and controlled by the response end, and global maintenance is carried out on the full-volume information table in the first storage device and the Topic information table in the second storage device through the response end.
Finally, the present application also discloses a computer device with an operating system, including:
one or more processors;
a memory for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of queue message store-and-forward as disclosed in any of the inventions above.
As a further refinement of the invention, the computer device with operating system is configured as a computer cluster, a data center or a cloud computing platform.
Compared with the prior art, the invention has the beneficial effects that:
by the queue message storing and forwarding method, the system and the computer device with the operating system, high concurrency response of a user to massive entity data and high concurrency response to entity data corresponding to the Topic messages with different priorities are met, delay of the massive Topic messages in the storing and forwarding process is reduced, and calculation cost of physical equipment is reduced.
Drawings
FIG. 1 is a flow chart of a method for storing and forwarding queue messages according to the present invention;
FIG. 2 is a topology diagram of a cloud computing platform deploying a queue message store-and-forward system of the present invention;
FIG. 3 is a modification of the response end included in the queue message store-and-forward system shown in FIG. 2;
FIG. 4 is another modification of the response side included in the queue message store-and-forward system shown in FIG. 2;
FIG. 5 is a complete flow diagram of the response of the queue message store-and-forward system shown in FIG. 2 to a user-initiated access request;
FIG. 6 is a schematic diagram of concurrent consumption of different priority configurations for loading different recommendation templates to a second storage device and initiating access requests to specified content based on a body held by a memory queue;
FIG. 7 is a schematic diagram of a user access information table modifying the priorities of Topic for specified content in a user Topic priority table to form a task sequence of different priorities;
FIG. 8 is a diagram of an example user interface of recommended content displayed to a user in a mobile terminal;
FIG. 9 is a diagram of an example user interface of modified recommended content displayed to a user in a mobile terminal;
FIG. 10 is a schematic diagram of a deployment producer and consumer in a mobile terminal;
FIG. 11 is a schematic diagram of a user and a first storage device sending an access request sent by the user and recommending updated recommended content to the user;
FIG. 12 is a topology diagram of a computer device with an operating system according to the present invention.
Detailed Description
The present invention will be described in detail below with reference to the embodiments shown in the drawings, but it should be understood that the embodiments are not limited to the present invention, and functional, method, or structural equivalents and alternatives according to the embodiments are within the scope of protection of the present invention by those skilled in the art.
The term'Logic for logic control"includes any physical and tangible function for performing a task. For example, each operation shown in the flowchart corresponds to a logic component for performing the operation. Operations may be performed using, for example, software running on a computer device, hardware (e.g., chip-implemented logic functions), etc., and/or any combination thereof. When implemented by a computing device, the logical components represent electrical components that are physical portions of the computer system, regardless of how they are implemented.
Phrase'Is configured as"or phrase"Is configured to "includes any manner in which any kind of physical and tangible function may be constructed to perform the identified operations. The functions may be configured to perform operations using, for example, software running on a computer device, hardware (e.g., chip-implemented logic functions), etc., and/or any combination thereof.
The term "entity data" refers broadly to streaming media files, photo files, audio files, installation files, txt documents, or computer system readable files with the suffix dat, as opposed to Topic messages.
The applicant first briefly outlines the queue message store-and-forward method according to the present application and the core inventive concept included in a queue message store-and-forward system 100 based on the foregoing queue message store-and-forward method, and details the various inventions according to the present application by the following embodiments. The applicant notes that the various systems, devices, and response ends disclosed by the queue message store-and-forward system 100 are a logical overview of the queue message store-and-forward method in a computer program formed by computer executable code and can be reasonably combined and split based on different programming languages.
As shown in fig. 1, 2 and 5, the queue message store-and-forward method is executed by a computer or software and includes the following steps S1 to S2.
Step S1, an access request for the specified content is received, and the Topic carried by the access request is saved in the user access information table 201. In step S2, the user access information table 201 uses the memory queue 70 to grasp the specified content corresponding to the access request from the third storage device 50 storing the specified content, and responds to the body initiating the access request to the specified content, for modifying the priority of the Topic to the specified content in the user Topic priority table 302.
The access request may be initiated by user 1 and/or user 2, by a robot program containing the code, or even by a hardware device containing the robot program. The access request triggers a computer operation such as invoking, exposing, modifying, storing, deleting, etc., the specified content. "designated content" is understood in this application as the aforementioned entity data, and such designated content is a process of an access request event initiated by a mobile terminal 10 (or a mobile terminal 11 in fig. 2) through a body (e.g., a user 1) that initiates an access request, and encapsulating the entity data with a Topic message memory queue 70 under the control of a responding end 60, and feeding back to the user 1. The specified content is typically stored in the third storage means 50. The third storage device 50 is composed of servers-1 to-n, and the servers may be physical servers. At the same time, topic messages exhibiting different priorities can also be visualized in the mobile terminal 10 to the body initiating the access request to the specified content.
The user access information table 201, the full-scale information table 202, the Topic information table, and the user Topic priority table 302 referred to in the various embodiments of the present application are all table stores. The table storage supports the characteristics of high concurrency of cross partitions and self-increasing of primary keys, and can be perfectly suitable for storing and scheduling massive Topic messages. Meanwhile, the access request in the application can be browsing behavior, file downloading behavior, file uploading behavior, live broadcasting behavior of streaming media file/audio file, on-demand behavior of streaming media file/audio file, program uploading or downloading behavior and the like of the webpage
Hereinafter, the present invention will be described in detail by way of a plurality of examples.
Embodiment one:
referring to fig. 1 to 9, this embodiment discloses a specific implementation of a queue message store-and-forward method (hereinafter or simply referred to as "method").
Referring to fig. 5, a user 1 sends an access request to a producer 101 in a mobile terminal 10, and the producer 101 sends user access information included in the access request, which includes one or more Topic messages, to a user access information table 201. Meanwhile, in the present embodiment, the producer 101 refers to a body or a program that creates entity data (e.g., streaming media data), and the consumer 102 is a body or a program that accesses entity data. Thus, the producer 101 and the consumer 102 shown in this embodiment are just one functional role embedded in the mobile terminal 10 running on the computer system level, and can be connected as an account or some kind of program or interface capable of distributing or receiving entity data in the actual running environment. It should be noted that the producer 101 and the consumer 102 may be omitted (see fig. 10 and 11), and replaced by a complete executable program page.
When the user 1 uses the mobile terminal 10 for the first time, the user access information table 201 may transmit user access information to the recommendation template generation module 40 to generate a preconfigured recommendation content through the recommendation template generation module 40 and save the preconfigured recommendation content to the Topic information table 301; when the user 1 uses the mobile terminal 10 later, the user access information may be directly transmitted to the user Topic priority table 302, so that the recommended content is updated based on the received access request for the specified content through the user access information table 201, thereby updating the user Topic in the user Topic priority table 302. The user access information carries one or more Topic messages.
The Topic information table 301 deployed in the second storage device 30 globally controls the user Topic priority table 302. The Topic information table 301 and/or the user Topic priority table 302 are stored in a table storage form in the second storage means 30. The second storage device 30 further deploys a task scheduling cluster 33 including task schedulers 321 to 32i, where the parameter i is a positive integer greater than or equal to 2. The Topic information table 301 periodically pulls data (i.e., topic messages) from the Topic information table 301 according to a time period set by a Timer (Timer) set by each task scheduler in the task scheduling cluster 33. The user Topic priority table 302 pulls the specified content from the corresponding task scheduler 321-32 i and updates the cursor. Reference is made in detail to the description below regarding the completion cursor P1 and the grasping cursor P2.
The user access information table 201 responds to the modification of the priority of Topic for the specified content in the user Topic priority table 302 by using the memory queue 70 to fetch the specified content corresponding to the access request from the third storage 50 storing the specified content to the body that initiated the access request for the specified content. Specifically, the user Topic priority table 302 sends the priority-adjusted Topic message queue to the memory queue 70 in a multithreaded manner. The memory queue 70 informs the responder 60 that the responder 60 encapsulates the specified content associated with the Topic message from the third storage 50 in the memory queue 70, and the memory queue 70 sends the specified content to the consumer 102 to eventually push the response request to the user 1 by the consumer 102.
In this embodiment, the user access information table 201 is associated with the full-volume information table 202, so that the specified content corresponding to the access request is determined according to the Topic carried by the access request through the full-volume information table 202 and/or the user access information table 201. The first storage device 20 and the second storage device 30 are independent from each other and are controlled by the response end 60, and global maintenance is performed on the full-size information table 202 in the first storage device 20 and the Topic information table 301 in the second storage device through the response end 60.
The Topic information table 301 contains: primary key ID, priority, load balancing partition key, topic code, topic name, user ID, topic information table creation time, and Topic information table update time. The user Topic priority table 302 contains: primary key ID, priority, load balancing partition key, topic ID, user ID, message ID, user Topic priority table creation time, and user Topic priority table update time. The user access information table 201 contains: a primary key ID, a Topic ID, a user ID, a message ID, a user access information table creation time, and a user access information table update time. The full amount information table 202 includes: primary key ID, topic ID, message name, message content, full information table creation time, and full information table update time. Specifically, the Topic information table 301 and the user Topic priority table 302 establish an association relationship by the primary key IDs each contained in each other. The user access information table 201 and the full-volume information table 202 also establish an association relationship by the primary key IDs each contained in each other.
The Topic information table 301 and the user Topic priority table 302 obtain multiple access requests concurrently based on a multithreaded timed task scheduling mechanism, so as to fetch the specified content corresponding to the access requests from the storage device storing the specified content through the memory queue 70 through the multiple access requests to respond to the consumer 102. In the present embodiment, the storage device storing the specified content is generally referred to as the third storage device 50, but when the data size of the entity data corresponding to the access request is small, the entity data may be grasped from the first storage device 20 and fed back to the mobile terminal 10. At this point, consumer 102 may query the specified entity data directly from Topic. Consumer 102 receives the Topic returned by the full size information table 202 (shown by reference arrow 7 a) or the Topic returned by the user access information table 201 (shown by reference arrow 7 b) to the body initiating the access request to the specified content in the first storage device 20 and the specified content associated based on the returned Topic. Meanwhile, the response procedure of the entity data shown by the arrow 7a and the arrow 7b is optional, and the entity data with smaller data volume is usually returned to the consumer 102 by the user access information table 201, but when the user access information table 201 cannot respond, the user access information table 201 may be replaced by the full information table 202, so that the entity data with smaller data volume is returned to the consumer 102 through the full information table 202. In the present embodiment, the third storage device 50 is mounted on the first storage device 20, and the user access information table 201 or the full-volume information table 202 can directly access the third storage device 50 to directly call entity data from one or more servers in the third storage device 50 to form specified contents in response to an access request initiated by the user 1 when the recommended contents are not updated.
In addition, entity data corresponding to the access request initiated by the main body (e.g., user 1) capable of initiating the access request includes, but is not limited to, a streaming media file, an audio file, a text file, or a program; meanwhile, the "specified content" is based on the service content corresponding to the Topic message, and changes in the service content are caused along with changes in the content included in the access request initiated by the user, and finally the service content is displayed in the mobile terminal 10 as shown in fig. 8 and 9.
Referring to fig. 8, the mobile terminal 10 is shown in the form of a visual interface 103, which visual interface 103 may be understood as a page formed by software (e.g. "jittering" APP) or a web page (e.g. a homepage of a video playing website) or a computer device embedded in a user interactive interface (e.g. a mobile phone or tablet computer in which the jittering APP is installed) for responding to a user initiated access request. Fig. 8 shows main menus such as "sports", "finance", and "variety". The visual interface 103 is jumped to display the next-level menus such as "basketball", "gymnastics", "snooker" when the user 1 clicks "sports", and the visual interface 103 is jumped to display the further next-level menus such as "Ying jin Sai", "Shi Cai", "China Cai Chi", "TPO10" when the user 1 further clicks "snooker". If user 1 clicks "sports", "snooker" and "inflorescence", respectively, that particular user is shown to be interested in the "world tournament" comparison. Therefore, when the response end 60 controls the memory queue 70 to assemble and play the entity data (e.g. "video file") corresponding to the Topic message of "world tournament" to the user 1, the response end 60 may issue an instruction for updating the recommended content to the recommendation template generating module 40. The user Topic priority table 302 is associated to the Topic information table 301 to adjust the task sequence with priority in the user Topic priority table 302 by means of said Topic information table 301. The user access information table 201 and/or the total amount information table 202 are stored in the first storage device 20 in a table storage form. Thereby eventually forming a visual interface 103a as shown in fig. 9. Meanwhile, the response end 60 can intervene in the event of generating the recommended content by the recommended template generating module 40 based on the advertisement putting behaviors of the publishers of different entity data, so that a business effect of targeted and accurate marketing is generated.
Referring to fig. 9, at this point, the "world tournament" has been taken as an option in the main menu, and user 1 can click on the "world tournament" directly to enter the video playing interface. The priority of the world championship is improved. Meanwhile, based on the mobile terminals 10 controlled by different users, the main menu interface presented in the mobile terminal 10 or the mobile terminal 11 controlled by each user may be different, and the priorities of the various levels of menus formed in the main menu interface of each user may also be different; it should be noted that, the priority in the main menu interface may be understood as not only the priority defined by the nested relationship, but also the logical relationship on the page layout, for example, the priority corresponding to the top-set message (or link) is higher than the priority defined by the message (or link) under the top-set message (or link). Therefore, personalized access requirements of different users on the specified content are met, and page arrangement can be carried out according to historical data and customization of access requests of the users. Such page arrangement does not require intervention in the spontaneous behavior of the user and can thus be adapted to the actual needs of the user, resulting in a good user experience. The ability of taking a partition key as a queue in the form storage nature enables the queue message storing and forwarding method disclosed by the embodiment to realize the queue storage of massive Topic messages, and can also assign priority, thereby ensuring the stability and reliability of the queue messages.
Meanwhile, the user access information table 201 updates the recommended content based on the received access request for the specified content. Specifically, when the user access information table 201 receives the user access request for the specified content, the Topic message carried by the access request may update the recommended content already existing in the user Topic priority table 302 and configured by the recommendation template generating module 40.
In this embodiment, the queue message store-and-forward method further includes: based on the mobile terminal 10 sending an access request for a specified content and displaying the specified content, the mobile terminal deploys the producer 101 and the consumer 102, the producer 101 issues an access request for the specified content into the user access information table 201, the consumer 102 receives a Topic returned by the full-size information table 202 or the user access information table 201 in the first storage device 20 to the main body initiating the access request for the specified content, and the specified content associated based on the returned Topic. The mobile terminal 10 includes an APP or computer device embedded in a user-interactive interface. The specified content includes a streaming media file, an audio file, a text file, or a program.
The memory queue 70 holds concurrent consumption, namely concurrent consumption 71, concurrent consumption 72 and concurrent consumption 73, of different priorities configured for the same principal and/or different principals of the specified content initiated access request, the concurrent consumption comprising at least one prioritized task sequence. Referring to fig. 6, the recommended templates a, B, C, and D are loaded into the database 32 by the recommended template generation module 40 under the control of the response end 60. The second storage device sends the Topic to the memory queue 70 in a multithreading manner, and notifies the response end 60 to pull the data source corresponding to the Topic (corresponding to the specified content corresponding to the Topic carried by the access request initiated by the user) from the third storage device 50, and encapsulates the data source in the memory queue 70 to form a plurality of concurrent consumption processes, where each concurrent consumption process has a different priority. The concurrent consumption 71, the concurrent consumption 72 and the concurrent consumption 73 are respectively sent to the recommendation template generation module 40 to update the recommended content caused by the access request to the specified content initiated by the same user or different users, and are displayed on the visual interface 103 of the mobile terminal 10.
As shown in connection with fig. 6, the memory queue 70 forms a plurality of task sequences that are responsive to the subject of initiating an access request to specified content and that include different priority orders. The Topic information table 301, the user Topic priority table 302 and the task scheduling cluster 33 may be deployed and run in the database 32, and the recommendation template generation module 40 may be deployed and run in the second storage device 30, or may be deployed and run in the responder 60, and is most preferably deployed and run in the database 32 of the second storage device 30.
The concurrent consumption 71 includes task sequences with priority order, i.e. task 1a, task 2a to task 19a; the concurrent consumption 72 includes a task sequence with a priority order, i.e. task 3a, task 4a to task 21a; the concurrent consumption 73 includes a task sequence with a priority order, i.e., task 2a, task 3a to task 20a. Each task corresponds to a Topic message with a different priority. Different concurrent consumption is based on one or more Topic messages contained in the access request received by each mobile terminal 10 from user 1 and forms a message repository based on the Timeline model. The user synchronizes the access behavior of streaming media files and the like to the specified content. The consumption messages 71 to 73 may be obtained concurrently, and a message distribution mechanism based on Kafka may be adopted, and a producer (producer 101) -consumer (consumer 102) asynchronous request processing mechanism may be adopted, where the producer and the consumer may be deployed in the same mobile terminal, or may belong to two different mobile terminals (for example, the mobile terminal 10 and the mobile terminal 11 shown in fig. 2). Producer 101 is responsible for crawling URL queue messages of web pages by crawler technology, and consumer 102 is responsible for retrieving URL queue messages from the queue to be crawled and storing them in the crawled queue. Producer 101 and consumer 102 may produce and consume Topic messages on multiple broaders for the purpose of accommodating high concurrent access requests and improving the stability and throughput of the queue message store-and-forward system 100 disclosed herein.
Referring to fig. 7, each concurrent consumption includes multiple Topic messages, each of which has a sequential ID (SeqId), the SeqId of the message at the back of the queue must be greater than the SeqId of the message at the front, i.e., it is guaranteed that the SeqId must be incremented, but is not required to be strictly incremented. New Topic messages are always added at the tail, ensuring that the SeqId of the new message is always larger than the messages already in the queue. The message may be randomly located to a specific message according to the SeqId for reading, or all messages within a certain given range may be read at will, for example, one or more Topic messages between the completed cursor P1 and the fetched cursor P2 in fig. 7.
In addition to task crawling, each mobile terminal 10 may have a less frequent task for task completion scanning, which task is used to most recently complete task cursors. The maximum value of the task scheduler in the scanning is the current pulled starting position, and the scanning logic is divided into the following three types:
(1) Scanning that the concurrent consumption has been updated to completion, at which time the cursor may be moved directly downward;
(2) Scanning to a task or an initial state, wherein a task is not set as running by any person but is pulled, because the task is a repeatedly grabbed task, at the moment, whether the value exists or not can be checked in a URL table, and direct skipping exists;
(3) And when the task is running, if the task is not overtime, the task is determined to be still executing, and the round scanning is ended. If the check time stamp times out, the result is checked, and if the content already exists, it is possible that the update state returns to the task table to fail, and the cursor continues to move down. If the content is not present either,
in this embodiment, each concurrent consumption has two cursors, namely, a completion cursor P1 and a capture cursor P2, and each priority of each Topic message is corresponding to the completion cursor P1 and the capture cursor P2. The definition of the grabbing cursor P2 refers to the scanned maximum displacement position under the current priority level of the current task. The definition of the completion cursor P1 indicates that the maximum grabbing completion displacement point is reached under the current priority level of the task, and all previous tasks have completed grabbing.
In this embodiment, by completing the cursor P1 and the grabbing cursor P2, the task scanning cursor can quickly walk down after each round of scanning, and the long-tail task can not obstruct the scanning of the new task. On the other hand, the task can preempt on the large URL table, so that unnecessary repeated grabbing is avoided, and the high throughput performance of a system using the queue message storing and forwarding method is improved.
At the same time, the self-increment column stored based on the table ensures that new tasks can be ranked at the end of the current queue with larger values. In addition, there is a complete scanning thread, which ensures that the new task is updated after all completed, and the cursor represents whether the whole task is completed or not. This cursor also ensures that tasks are not lost. The task can reconstruct a task with a long tail and insert the task into a queue, the new task can be triggered again by a new crawler, the problem of loss of the consumption messages 71-73 caused by the blocking or the dead halt of one mobile terminal 10 (or the mobile terminal 11) is avoided, zero loss of the consumption messages 71-73 is realized, reliable response to a plurality of access requests initiated by a user is ensured, and good user experience can be obtained.
Embodiment two:
based on the method disclosed in the first embodiment, based on the same inventive concept, this embodiment also discloses a specific implementation of a queue message store-and-forward system 100.
In this embodiment, the queue message store-and-forward system 100 includes:
the first storage device 20 for configuring the user access information table 201, the second storage device 30 for configuring the user Topic priority table 302, the third storage device 50, the memory queue 70 and the responding end 60. When an access request to the specified content is received, the Topic carried by the access request is stored in the user access information table 201, and the specified content corresponding to the access request is fetched from the third storage device 50 storing the specified content by using the memory queue 70 to respond to the consumer (namely, the user 1 or the user 2) through the priority modification of the Topic to the specified content in the user Topic priority table 302 by the user access information table 201.
Preferably, in this embodiment, the first storage device 20 further configures a full-volume information table 202, and the user access information 201 table is associated with the full-volume information table 202, so as to determine, through the full-volume information table 202 and/or the user access information table 201, specified content corresponding to the access request according to the Topic carried by the access request. The second storage means 30 is further configured with a Topic information table 301, and a user Topic priority table 302 is associated to the Topic information table 301 for adjusting the task sequence with priority in the user Topic priority table 302 by means of said Topic information table 301.
The queue message store-and-forward system 100 further comprises: the recommendation template generation module 40. The recommendation template generation module 40 pre-configures the recommendation content and saves it to the Topic information table 301. The Topic information table 301 and/or the user Topic priority table 302 are stored in the second storage means 30 in the form of a table store (TableStore). More specifically, the second storage device 30 may be understood as a storage device capable of configuring and running the database 32 in an actual configuration environment, and is preferably logically independent from the first storage device 20, and independent from the memory queue 70, so as to ensure that the Topic information table 301 and the user Topic priority table 302 configured in the second storage device 30 do not interfere with the full information table 202 and the user access information table 201 configured in the first storage device 20, and ensure the reliability and disaster tolerance in the process of calling the Topic message while ensuring the mutual data independence. Meanwhile, referring to fig. 5, in the present embodiment, the first storage device 20 and the second storage device 30 are independent from each other and are controlled by the response end 60, and global maintenance is performed on the full-size information table 202 in the first storage device 20 and the Topic information table 301 in the second storage device 30 through the response end 60. Specifically, in the present embodiment, the database 32 is preferably a distributed time series database.
In particular, in the present embodiment, the Topic information table 301, the user Topic priority table 302, the total information table 202, and the user access information table 201 are all stored in the form of a table store (table store) and forward Topic messages. The multi-element index (SearchIndex) function stored in the form can easily meet the functional requirements of multidimensional query, GEO retrieval, statistics and the like of users; a distributed NoSQL database constructed based on an LSM tree can easily support high-volume concurrent reading and writing, and zero operation and maintenance can easily cope with continuous expansion of data volume without upper limit in theory. The table storage is charged by volume, providing two examples of capacity type and high performance type, the capacity type is more suitable for cold data (cold data refers to data which is rarely accessed after being generated), and lower storage cost is provided. More importantly, the form storage has perfect calculation ecology, provides full and incremental channel service, provides a calculation system integrating stream calculation and batch calculation, and provides a channel for the value mining of future monitoring data. Therefore, the use of the table storage can meet the use requirement of the user for high concurrency response to the massive entity data, and can meet the high concurrency response to the entity data corresponding to the Topic messages with different priorities, thereby reducing the delay existing in the storing and forwarding process of the massive Topic messages, reducing the calculation overhead to the physical devices (such as a plurality of servers in the third storage device 50 and the response end 60), and simultaneously, as the response end 6, without configuring a CPU with too high specification, only supporting network card multi-queues and higher I/O capability is needed, and the task of storing the massive entity data can be completed by the third storage device 50 or executed by a hosting server or a data center (IDC) controlled by the response end 60.
As a reasonable modification of the present embodiment, as shown in fig. 3 and fig. 4, the second storage device 30 and the recommendation template generation module 40 may be logically disposed at the response end 60; meanwhile, the first storage device 20, the second storage device 30, the recommendation template generation module 40 and the memory queue 70 may be logically disposed at the response end 60. The responder 60 may be understood as a type of computer device (or server or cluster of computers or data center) that holds specified content corresponding to an access request initiated by user 1 (or user 2).
Meanwhile, the full-volume information table 202 in the first storage device 20 is globally maintained by the responding end 60, the Topic information table 301 in the second storage device 30 may also be globally maintained by the responding end 60, or the Topic information table 301 may be globally maintained by the responding end 60 sending an instruction to update the recommended content to the recommendation template generating module 40.
The technical solutions of the same parts of the queue message store-and-forward system 100 disclosed in the present embodiment and the queue message store-and-forward method disclosed in the first embodiment are described in the first embodiment, and are not repeated here.
Embodiment III:
Referring to FIG. 12, a specific implementation of a computer device 400 with an operating system is disclosed in accordance with one or both of the first and second embodiments.
In this embodiment, the "operating system" in the computer device 400 includes, but is not limited to, IOS operating system, windows operating system, linux operating system, unix operating system, android operating system, ubuntu operating system, cloud host operating system, and any other program capable of running independently on physical and/or virtual media (kernel space), or desktop virtual machines (e.g., VMware workstation).
In this embodiment, the computer device 400 with an operating system includes:
one or more processors 41;
a memory 42 for storing one or more programs,
the one or more programs, when executed by the one or more processors 41, cause the one or more processors 41 to perform the queue message store-and-forward method as disclosed in embodiment one.
Specifically, the computer device 400 with an operating system may be configured as a computer cluster, a data center, or a cloud computing platform, and may even be a mobile electronic device with a built-in large-scale integrated circuit or a very large-scale integrated circuit, a wearable smart device, a smart phone, a tablet computer, or the like. The cloud computing platform includes a public cloud, a private cloud, or a hybrid cloud. The specific technical scheme of the queue message store-and-forward method disclosed in this embodiment is described in the first embodiment, and is not repeated here.
In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above list of detailed descriptions is only specific to practical embodiments of the present invention, and they are not intended to limit the scope of the present invention, and all equivalent embodiments or modifications that do not depart from the spirit of the present invention should be included in the scope of the present invention.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. The present embodiments are, therefore, to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Furthermore, it should be understood that although the present disclosure describes embodiments, not every embodiment is provided with a separate embodiment, and that this description is provided for clarity only, and that the disclosure is not limited to the embodiments described in detail below, and that the embodiments described in the examples may be combined as appropriate to form other embodiments that will be apparent to those skilled in the art.
Claims (18)
1. A method for storing and forwarding a queue message,
executed by a computer or software, comprising:
receiving an access request for specified content, and storing a Topic carried by the access request into a user access information table;
the user access information table is used for modifying the priorities of the topics of the specified contents in the user Topic priority table, the specified contents corresponding to the access requests are grabbed from the third storage device for storing the specified contents by using the memory queue, and the user Topic priority table is associated with the Topic information table so as to adjust the task sequences with the priorities in the user Topic priority table.
2. The method according to claim 1, wherein the user access information table is associated to a full-volume information table, so as to determine, according to Topic carried by an access request, specified content corresponding to the access request through the full-volume information table and/or the user access information table.
3. The queue message store-and-forward method of claim 1, wherein the user access information table and/or the full-size information table is stored in a table storage form in the first storage device.
4. A method of queue message store-and-forward as claimed in claim 3, further comprising: pre-configuring recommended content and storing the recommended content in a Topic information table;
the Topic information table and/or the user Topic priority table are stored in a second storage means in a table storage form.
5. The method for storing and forwarding queue messages according to claim 4, wherein the first storage device and the second storage device are independent from each other and are controlled by the response end, and global maintenance is performed on the full-size information table in the first storage device and the Topic information table in the second storage device by the response end.
6. The method of queue message store-and-forward of claim 4, further comprising: the user access information table updates the recommended content based on the received access request to the specified content.
7. The method of claim 1, wherein the memory queues form a plurality of task sequences that are responsive to a body initiating an access request to specified content and that include different priority orders.
8. The method according to claim 7, wherein the Topic information table and the user Topic priority table are based on a multithreaded timed task scheduling mechanism, and a plurality of access requests are obtained concurrently, so that the specified content corresponding to the access requests is fetched from a storage device storing the specified content through a memory queue through the plurality of access requests to respond to the consumer.
9. The queue message store-and-forward method of claim 1, further comprising:
the method comprises the steps that based on a mobile terminal, an access request for specified content is sent and displayed, a producer and a consumer are deployed, the producer issues the access request for the specified content to a user access information table, the consumer receives a Topic returned by a total information table or the user access information table in a first storage device to a main body initiating the access request for the specified content, and the specified content related to the returned Topic is based on the specified content;
the mobile terminal comprises an APP or a computer device embedded in a user interactive interface;
the specified content includes a streaming media file, an audio file, a text file, or a program.
10. The method for storing and forwarding queue messages according to claim 3, wherein said Topic information table comprises: primary key ID, priority, load balancing partition key, topic code, topic name, user ID, topic information table creation time, and Topic information table update time; the user Topic priority table includes: primary key ID, priority, load balancing partition key, topic ID, user ID, message ID, user Topic priority table creation time, and user Topic priority table update time.
11. The queue message store-and-forward method according to any one of claims 2-6 or 9, wherein the user access information table comprises: a primary key ID, a Topic ID, a user ID, a message ID, a user access information table creation time and a user access information table update time; the full amount information table includes: primary key ID, topic ID, message name, message content, full information table creation time, and full information table update time.
12. A method according to claim 7 or 8, wherein the memory queue holds concurrent consumption of different priorities for the same body and/or different bodies of a given content-initiated access request, the concurrent consumption comprising at least one task sequence with a priority order.
13. A queue message store-and-forward system, comprising:
a first storage device for configuring a user access information table, a second storage device for configuring a user Topic priority table, a third storage device, a memory queue and a response end;
when an access request for specified contents is received, the Topic carried by the access request is stored in a user access information table, the priority of the Topic for the specified contents in a user Topic priority table is modified through the user access information table, the specified contents corresponding to the access request are grabbed from a third storage device for storing the specified contents by using a memory queue, the second storage device responds to a main body for initiating the access request for the specified contents, the second storage device is further provided with a Topic information table, and the user Topic priority table is associated with the Topic information table so as to adjust the task sequence with the priority in the user Topic priority table through the Topic information table.
14. The queue message store-and-forward system of claim 13, further comprising: a recommendation template generation module;
the recommendation template generation module pre-configures recommendation contents and stores the recommendation contents in a Topic information table;
the Topic information table and/or the user Topic priority table are stored in a second storage means in a table storage form.
15. The queue message store-and-forward system of claim 14, wherein the first storage device is further configured with a full-size information table, and wherein the user access information table is associated with the full-size information table, so that specified content corresponding to the access request is determined according to Topic carried by the access request through the full-size information table and/or the user access information table.
16. The queue message store-and-forward system of any one of claims 13-15, wherein the first storage device and the second storage device are independent of each other and are both controlled by a response end, and global maintenance is performed on a full-volume information table in the first storage device and a Topic information table in the second storage device by the response end.
17. A computer device having an operating system, comprising:
one or more processors;
A memory for storing one or more programs,
the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the queue message store-and-forward method of any one of claims 1-12.
18. The computer device with an operating system of claim 17, wherein the computer device with an operating system is configured as a computer cluster, a data center, or a cloud computing platform.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010186264.6A CN111459686B (en) | 2020-03-17 | 2020-03-17 | Queue message storing and forwarding method, system and computer device with operating system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010186264.6A CN111459686B (en) | 2020-03-17 | 2020-03-17 | Queue message storing and forwarding method, system and computer device with operating system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111459686A CN111459686A (en) | 2020-07-28 |
| CN111459686B true CN111459686B (en) | 2023-06-27 |
Family
ID=71682087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010186264.6A Active CN111459686B (en) | 2020-03-17 | 2020-03-17 | Queue message storing and forwarding method, system and computer device with operating system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111459686B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115134419A (en) * | 2022-06-24 | 2022-09-30 | 杭州涂鸦信息技术有限公司 | Data transmission method, device, equipment and medium |
| CN116048819A (en) * | 2023-03-30 | 2023-05-02 | 杭州西软计算机工程有限公司 | High concurrency data storage method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2005081136A1 (en) * | 2004-01-20 | 2005-09-01 | Cedars-Sinai Medical Center | Data repository system |
| CN107465735A (en) * | 2017-07-31 | 2017-12-12 | 杭州多麦电子商务股份有限公司 | Distributed information system |
| CN109271417A (en) * | 2018-11-23 | 2019-01-25 | 福建榕基软件股份有限公司 | A kind of lightweight messages queue implementing method and storage equipment based on database |
| CN109462546A (en) * | 2018-12-28 | 2019-03-12 | 苏州思必驰信息科技有限公司 | A kind of voice dialogue history message recording method, apparatus and system |
| CN110113420A (en) * | 2019-05-08 | 2019-08-09 | 重庆大学 | Distributed Message Queue management system based on NVM |
-
2020
- 2020-03-17 CN CN202010186264.6A patent/CN111459686B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2005081136A1 (en) * | 2004-01-20 | 2005-09-01 | Cedars-Sinai Medical Center | Data repository system |
| CN107465735A (en) * | 2017-07-31 | 2017-12-12 | 杭州多麦电子商务股份有限公司 | Distributed information system |
| CN109271417A (en) * | 2018-11-23 | 2019-01-25 | 福建榕基软件股份有限公司 | A kind of lightweight messages queue implementing method and storage equipment based on database |
| CN109462546A (en) * | 2018-12-28 | 2019-03-12 | 苏州思必驰信息科技有限公司 | A kind of voice dialogue history message recording method, apparatus and system |
| CN110113420A (en) * | 2019-05-08 | 2019-08-09 | 重庆大学 | Distributed Message Queue management system based on NVM |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111459686A (en) | 2020-07-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11422853B2 (en) | Dynamic tree determination for data processing | |
| CN109902220B (en) | Webpage information acquisition method, device and computer readable storage medium | |
| US9996593B1 (en) | Parallel processing framework | |
| TWI703463B (en) | Information display method, device and intelligent terminal | |
| KR102056881B1 (en) | Navigating among content items in a set | |
| US9367211B1 (en) | Interface tab generation | |
| US10007711B2 (en) | Data rendering optimization | |
| US20140047323A1 (en) | Systems and methods for alert management | |
| CN111666490A (en) | Information pushing method, device, equipment and storage medium based on kafka | |
| US20080065602A1 (en) | Selecting advertisements for search results | |
| KR20180073644A (en) | Mobile user interface | |
| CN111459686B (en) | Queue message storing and forwarding method, system and computer device with operating system | |
| CN103699674A (en) | Webpage storing method, webpage opening method, webpage storing device, webpage opening device and webpage browsing system | |
| CN103034518A (en) | Method and browser for loading browser control tool | |
| US10942948B2 (en) | Cloud-based pluggable classification system | |
| US10567845B2 (en) | Embeddable media content search widget | |
| CN113608737A (en) | Page generation method, device, equipment and medium | |
| CN116546091B (en) | Recommendation method, device, equipment and storage medium of streaming content | |
| US11954164B2 (en) | Asynchronous content switching | |
| AU2021297136B2 (en) | Dynamic trigger of web beacons | |
| Chen et al. | Research on Data Storage and Processing Optimization Based on Federation HDFS and Spark | |
| CN108319621B (en) | Media content processing method, device and platform | |
| CN113901271A (en) | Method, device, storage medium and equipment for configuring system based on electronic form | |
| CN117170666A (en) | Search page generation method and device, storage medium and terminal equipment | |
| Akritidis et al. | Modern web technologies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: No. 6 Science and Education Software Park, Binhu District, Wuxi City, Jiangsu Province Applicant after: Huayun data holding group Co.,Ltd. Address before: No. 6 Science and Education Software Park, Binhu District, Wuxi City, Jiangsu Province Applicant before: WUXI CHINAC DATA TECHNICAL SERVICE Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |