CN116385116A

CN116385116A - Method for acquiring big data of multiple household appliances by utilizing priority queue and computing equipment

Info

Publication number: CN116385116A
Application number: CN202310398916.6A
Authority: CN
Inventors: 李洪世; 徐博
Original assignee: Shenzhen Zhige Digital Technology Co ltd
Current assignee: Shenzhen Zhige Digital Technology Co ltd
Priority date: 2023-04-04
Filing date: 2023-04-04
Publication date: 2023-07-04

Abstract

The application provides a method and computing equipment for acquiring big data of multiple home appliances by utilizing a priority queue. The method comprises the following steps: setting a queue for storing the online store ID, wherein the queue comprises a non-execution regular queue, an execution regular queue, a non-execution priority queue and an execution priority queue; updating the online store ID in the queue at regular time; preferentially finishing requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the unexecuted priority queue; and when no data exists in the non-execution priority queue and the execution priority queue, requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the non-execution conventional queue. According to the technical scheme, the priority queue is set, so that the data of newly added online stores can be acquired preferentially in a simple mode.

Description

Method for acquiring big data of multiple household appliances by utilizing priority queue and computing equipment

Technical Field

The application relates to the technical field of machine learning and business big data, in particular to a method and computing equipment for acquiring multi-household appliance business big data by utilizing a priority queue.

Background

With the development of cross-border electronic commerce, more and more cross-border electronic commerce begin to present financing requirements. When a financial institution carries out cross-border e-commerce financing business, the rating of credit risk depends on the down-line adjustment, and the time and labor cost are relatively high. This results in a predominance of mortgage loans, such as unrooted water, over-border electronic commerce, such that many electronic commerce operations that require financing but lack mortgages are limited in their development.

Conventionally, financial institutions typically conduct risk rating predictions through human experience or by way of creating risk scoring cards. However, the conventional method is not high in accuracy and requires a lot of manpower, and fails to effectively utilize a lot of commercial big data generated in the e-commerce activity. Compared with the traditional industry, the electronic commerce can generate massive and available original data on the ecological value chain. The acquisition, processing, or efficient use of such data may provide assistance to the business operations or support for business decisions.

Some e-commerce platforms have opened a data interface. Therefore, a solution for efficiently acquiring big data of an electronic commerce needs to be developed to acquire complete and accurate mass data from an electronic commerce platform as timely as possible.

The above information disclosed in the background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art.

Disclosure of Invention

The application aims to provide a method and computing equipment for acquiring big data of multiple home appliances by utilizing a priority queue, and the priority acquisition of data of newly added online stores, for example, can be realized in a simple manner by setting the priority queue.

The user characteristics and advantages of the present application will become apparent from the detailed description set forth below, or may be learned in part by practice of the application.

According to an aspect of the present application, there is provided a method for acquiring big data of multiple home appliances using a priority queue, including: setting a queue for storing the online store ID, wherein the queue comprises a non-execution regular queue, an execution regular queue, a non-execution priority queue and an execution priority queue; updating the online store ID in the queue at regular time; preferentially finishing requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the unexecuted priority queue; and when no data exists in the non-execution priority queue and the execution priority queue, requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the non-execution conventional queue.

According to another aspect of the present application, there is provided a computing device comprising: a processor; a memory having a computer program stored thereon; the aforementioned method is implemented when the processor executes the computer program.

According to another aspect of the present application, there is provided a computer readable medium having stored thereon a computer program which, when executed by a processor, implements the method as described above.

According to some embodiments, by setting the priority queue, priority acquisition of data of, for example, newly added online stores can be achieved in a simple manner.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings.

Fig. 1 shows a schematic diagram of an application scenario of the technical solution of the present application.

Fig. 2 shows a flowchart of a method for acquiring multi-home vendor big data according to an example embodiment of the present application.

FIG. 3 illustrates an operational flow diagram for obtaining online store IDs in a thread-safe manner and requesting first type data for different online stores from an e-commerce platform, respectively, using a plurality of first threads, according to an embodiment.

FIG. 4 illustrates an operational flow diagram for obtaining online store IDs in a thread-safe manner and requesting first type data for different online stores from an e-commerce platform, respectively, using a plurality of first threads in accordance with another embodiment.

Fig. 5 illustrates a flowchart of a method for acquiring multi-home merchant big data using a priority queue according to an example embodiment.

FIG. 6 illustrates a flowchart of a method for prioritizing completion of a request for corresponding e-commerce data to an e-commerce platform based on an online store ID in a non-execution priority queue in accordance with an example embodiment.

FIG. 7 illustrates a flowchart of a method for requesting corresponding e-commerce data from an e-commerce platform based on an online store ID in a non-execution regular queue in accordance with an example embodiment.

Fig. 8 illustrates a flowchart of a method for acquiring multi-vendor big data using multi-threading and priority queues according to an example embodiment.

Fig. 9 shows a schematic diagram of a system for acquiring multi-home vendor big data according to another embodiment of the present application.

FIG. 10 illustrates a block diagram of a computing device according to an example embodiment of the present application.

Detailed Description

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted.

Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present application. One skilled in the relevant art will recognize, however, that the aspects of the application can be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the application.

The block diagrams depicted in the figures are merely functional entities and do not necessarily correspond to physically separate entities. That is, the functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different networks and/or processor devices and/or microcontroller devices.

The flow diagrams depicted in the figures are exemplary only, and do not necessarily include all of the elements and operations/steps, nor must they be performed in the order described. For example, some operations/steps may be decomposed, and some operations/steps may be combined or partially combined, so that the order of actual execution may be changed according to actual situations.

The terms first, second and the like in the description and in the claims of the present application and in the above-described figures, are used for distinguishing between different objects and not for describing a particular sequential order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may include other steps or elements not listed or inherent to such process, method, article, or apparatus.

Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art will appreciate that the embodiments described herein may be combined with other embodiments.

The online operation mode of the electronic commerce generates a large amount of raw electronic commerce data of a repository, and some electronic commerce platforms open a data interface to acquire data for the electronic commerce. However, the e-commerce platform is generally request limited to interface access to the same user, e.g., requests within one second cannot exceed a predetermined value, etc. When there is a newly added online store authorized to access the data interface, an efficient special handling is required because the newly authorized online store has a large amount of history data.

Therefore, the embodiment of the application provides a method for acquiring large data of multiple home appliances, which can efficiently acquire the large data of the electric appliances from the electric appliance platform through processing modes such as priority queues and the like.

The technical scheme of the present application will be described in detail with reference to examples.

Referring to fig. 1, in an e-commerce system, data generated on an e-commerce value chain may be deposited in a database of an e-commerce platform. In order to acquire the precipitation data, the data can be authorized to be used by the data processing system according to the embodiment of the application through an API interface of the e-commerce platform in an e-commerce authorization mode. In addition, the system can also obtain the original electronic commerce data of multiple dimensions (sales, stock, flow, policy violations, logistics, settlement and the like) of the user in real time by carrying out data docking in the form of interfaces and the like with other main participants (including third party payments, logistics merchants and warehouse service merchants) of the electronic commerce, and can use a distributed technical means for storage and calculation.

After receiving the authorization, the data processing system according to the embodiment of the application pulls the original electronic commerce data of the corresponding electronic commerce in the electronic commerce platform to a storage system associated with the data processing system. According to some embodiments, the storage system may be a distributed storage system.

The data processing system according to the embodiment of the application processes the data, for example, through standardization processing, so as to obtain data which can be used subsequently; and then, combining industry experience, business model, financial model and/or statistical model, and the like, obtaining a risk rating result and a risk portrait of the e-commerce enterprise by means of machine learning and the like, providing the risk rating result and the risk portrait to financial institutions such as banks and the like, and taking the risk rating result and the risk portrait as reliable financing basis, and the like.

Referring to fig. 2, at S201, a plurality of first threads of N number are simultaneously turned on.

According to an example embodiment, the N first threads are more capable of requesting data from the e-commerce platform than the electricity Shang Ping requests for data of the first type, and the N-2 first threads are less capable of requesting data from the e-commerce platform than the electricity Shang Ping requests for data of the first type. Thus, by controlling the number of threads, the interface capability provided by the e-commerce platform can be maximized. In addition, by controlling the number of threads, the computational resources and network resources are not wasted as much as possible, and excessive thread idling is avoided.

At S203, first type data of different online stores are respectively requested from the e-commerce platform using a plurality of first threads.

After obtaining the internet shop authorization, the data interface of the e-commerce platform can be accessed. For example, after confirming the authorization of the customer, the e-commerce platform returns a token (token) of the online store, saves the token in the background, and all interfaces need to carry the token to request.

The interfaces of the e-commerce platform may include an order data interface, a sales data interface, a report data interface, a document data interface, and the like. The interfaces provided by different e-commerce platforms may vary. The first type of data may be data acquired from one of these interfaces, which is not limited in this application. It is to be understood that the first type of interface is generally referred to herein, and is intended to indicate that the technical solution of the present application may acquire data of any type of interface, and certainly may also acquire data of these interfaces through different threads (e.g., the 2 nd, … th, and n th threads) at the same time.

According to some embodiments, further, the N-1 first threads have less capability to request data from the e-commerce platform than the request limit of electricity Shang Ping for the first type of data.

According to some embodiments, the request responses received by the plurality of first threads may also be monitored, thereby dynamically adjusting the number of first threads, and thus making a more accurate dynamic adjustment of the number of threads.

According to some embodiments, the plurality of first threads may run on a plurality of different servers when performance of a single server is not satisfactory.

At S205, the first type of data is stored in the distributed file system.

According to the example embodiment, the acquired first type data is stored in the distributed file system, so that the data acquisition efficiency is improved, and the data security is improved.

For example, the distributed file system may include the FastDFS system, but the present application is not limited thereto.

At S207, the first type of data in the file system is stored in the distributed database system.

According to an example embodiment, further, the first type of data in the file system may be stored in a distributed database system. Stored in the file system are a large number of small text files that are retrieved. For the convenience of statistical processing and utilization, a large number of small text files are subjected to analysis processing, and data in the small text files are stored in a distributed database system.

For example, the distributed database system may include a TiDB system, but the present application is not limited thereto.

According to the above-described embodiments, by employing multiple threads whose request capabilities are well beyond the interface limitations of the e-commerce platform, the interface capabilities provided by the e-commerce platform can be maximized while minimizing the waste of computing resources and network resources.

Referring to fig. 3, at S301, each first line Cheng Huoqu is a first distributed lock.

According to some embodiments, a distributed lock Redison implemented with an open source may be employed, but the application is not limited thereto.

At S303, the first thread that acquired the first distributed lock acquires the online store ID and releases the first distributed lock.

According to some embodiments, the online store ID (identifier) of the online store that needs to obtain the data is stored in the unexecuted queue.

The first thread obtains the online store ID from the unexecuted queue and then the first thread places the online store ID into the in-progress queue. In this way, since the first thread has acquired the first distributed lock, the problem of the online store ID being repeatedly acquired does not occur.

According to some embodiments, a list of online stores authorized by a client may be queried periodically, a set of online store IDs may be obtained, and the obtained set of online store IDs may be added to the unexecuted queue after de-duplication. For example, deduplication may be achieved by removing the online store IDs of the unexecuted queue and the in-execution queue from the acquired set of online store IDs.

At S305, the first thread that acquired the online store ID requests corresponding first type data from the e-commerce platform.

After the online store ID is acquired, the first thread may acquire the token according to the online store ID and parameters such as a time range of data to be acquired, and request first type data of the corresponding online store, such as a single data interface, a sales data interface, a report data interface, or a document data interface, from the e-commerce platform.

According to an example embodiment, after the first thread completes requesting the first type of data corresponding to the online store ID from the e-commerce platform, the online store ID may be removed from the in-execution queue and another online store ID may be obtained from the un-execution queue, and then the above operations may be repeated.

Referring to fig. 4, at S401, a plurality of first threads acquire online store IDs from unexecuted queues, respectively.

According to some embodiments, the online store ID (identifier) of the online store that needs to obtain the data is placed into an unexecuted queue, which is a queue based on a single-threaded memory storage system, which may include, for example, a Redis system. However, the present application is not limited thereto, and other thread-safe queues may be used to store the online store IDs, so as to ensure that the problem of repeatedly acquiring the online store IDs does not occur.

In S403, the first thread that acquired the online store ID puts the acquired online store ID in the in-progress queue.

Thus, through the queue in execution, the visualization and real-time monitoring of the online store in the data grabbing process can be realized.

At S405, the first thread that acquired the online store ID requests corresponding first type data from the e-commerce platform.

Referring to fig. 5, at S501, a queue for storing online store IDs is set. According to an example embodiment, the queues include a normal queue not executing, a normal queue in execution, a priority queue not executing, a priority queue in execution.

By setting different queues, a basis can be provided for preferentially acquiring certain online store data. In addition, through the arrangement of the queues in execution, the visualization and real-time monitoring of online stores in the data grabbing process can be realized, and the preferential acquisition of newly-added online store data is further ensured.

At S503, the online store ID in the queue is updated regularly.

According to some embodiments, customer-authorized online store names may be queried periodically, such as from a database, to obtain an online store ID set. The online store IDs in the queue are then excluded from the set of online store IDs. And adding the reserved online store ID of the newly added online store to the non-execution priority queue, and adding the reserved online store ID of the non-new online store to the non-execution conventional queue.

According to some embodiments, the acquired set of online store IDs may include a first set including online store IDs of newly added online stores and a second set including online store IDs of non-new online stores. Thus, when the online store IDs in the queue are excluded from the online store ID set, the online store IDs of the non-execution priority queue and the in-execution priority queue are excluded from the first set, and the online store IDs of the non-execution regular queue and the in-execution regular queue are excluded from the second set.

In S505, the request for the corresponding e-commerce data from the e-commerce platform is completed preferentially according to the online store ID in the unexecuted priority queue.

When there is a newly added online store authorized to access the data interface, since the newly authorized online store has a large amount of history data, a priority is required to acquire the data of the newly added online store.

According to an example embodiment, requesting the corresponding e-commerce data from the e-commerce platform is accomplished by processing the online store ID in the unexecuted priority queue.

In S507, when there is no data in the priority queue in execution and the priority queue in execution, corresponding e-commerce data is requested to the e-commerce platform according to the online store ID in the regular queue in non-execution.

According to an example embodiment, the acquisition of the data of the online stores in the regular queue is started after the completion of the data acquisition of the online stores in the priority queue. For example, the data of the stock online store is acquired only after the data of the newly added online store is acquired from the e-commerce platform. In this way, not only can the preferential acquisition of specific data be ensured, but also the efficient acquisition of data can be realized in a simple manner, and the switching or waiting process in the data acquisition process is reduced as much as possible.

According to some embodiments, the acquired data may be stored in a distributed file system, improving data acquisition efficiency and increasing data security. For example, the distributed file system may include the FastDFS system, but the present application is not limited thereto.

According to some embodiments, further, the obtained data in the file system may be stored in a distributed database system. Stored in the file system are a large number of small text files that are retrieved. For the convenience of statistical processing and utilization, a large number of small text files are subjected to analysis processing, and data in the small text files are stored in a distributed database system. For example, the distributed database system may include a TiDB system, but the present application is not limited thereto.

According to some embodiments, S505 and S507 may be performed in multiple threads. The unexecuted regular queue and unexecuted priority queue are queues based on a single-threaded memory storage system, which may include a Redis system. However, the present application is not limited thereto, and other thread-safe queues may be used to store the online store ID, or the online store ID may be acquired in other thread-safe manners, so as to ensure that the problem that the online store ID is repeatedly acquired does not occur.

Referring to fig. 6, in S601, the online store ID is acquired from the unexecuted priority queue.

According to an example embodiment, each time an online store ID is acquired to request corresponding data from an e-commerce platform, the online store ID is first acquired from an unexecuted priority queue. In this way, the data of some online stores can be preferably acquired, for example, the data of newly added online stores is acquired first.

In S603, the acquired online store ID is put in the execution priority queue.

According to the embodiment, the acquired online store ID is put into the priority queue in execution, so that the online store in the data grabbing process can be visually and real-time monitored. In addition, preferential acquisition of newly added online store data, for example, can be further ensured, see the description below.

At S605, after the request for the corresponding e-commerce data from the e-commerce platform is completed according to the acquired online store ID, the online store ID is removed from the in-execution priority queue.

After the request of the data corresponding to the online store ID from the e-commerce platform is completed, the online store ID may be removed from the in-process priority queue, and then the above operations may be repeatedly performed.

Referring to fig. 7, in S701, the online store ID is acquired from the unexecuted priority queue.

According to an example embodiment, each time an online store ID is acquired to request corresponding data from an e-commerce platform, the online store ID is first acquired from an unexecuted priority queue. In this way, it is ensured that when the newly added online store ID is present in the non-execution priority queue, the online store ID can be processed with priority.

If the online store ID is not acquired from the unexecuted priority queue, but the in-execution priority queue is not empty, the process returns to S701 after a delay. In this way, it is ensured that the data of the online store in the conventional queue starts to be acquired from the e-commerce platform after the data of the online store in the priority queue is acquired from the e-commerce platform is preferentially completed.

If the online store ID is not acquired from the unexecuted priority queue and the in-execution priority queue is empty, the flow goes to S703.

At S703, the online store ID is acquired from the non-execution regular queue.

Since the online store that is preferentially handled is not already needed at this time, the online store in the regular queue, for example, an on-store online store, starts to be handled.

In S705, the acquired online store ID is put in the in-progress regular queue.

As before, by placing the acquired online store ID into the executing regular queue, the online store of the data grabbing process can be visually and real-time monitored.

At S707, after the request for the corresponding e-commerce data from the e-commerce platform is completed according to the acquired online store ID, the online store ID is removed from the in-process regular queue.

After the request of the data corresponding to the online store ID from the e-commerce platform is completed, the online store ID may be removed from the in-process regular queue and then the above operations may be repeatedly performed.

Referring to fig. 8, at S801, a queue for storing online store IDs is set. According to an example embodiment, the queues include a normal queue not executing, a normal queue in execution, a priority queue not executing, a priority queue in execution.

At S803, a number N of first threads are simultaneously started.

According to the example embodiment, the data is acquired in a multithreading mode, so that the data acquisition efficiency is further improved.

According to some embodiments, the ability of the N first threads to request data from the e-commerce platform is greater than the request limit of the electricity Shang Ping platform for the first type of data. Further, according to some embodiments, it may be determined that the ability of N-2 or N-1 first threads to request data from the e-commerce platform is less than the request limit of electricity Shang Ping for the first type of data.

Thus, by controlling the number of threads, the interface capability provided by the e-commerce platform can be maximized. In addition, by controlling the number of threads, computing resources and network resources are not wasted as much as possible.

At S805, the online store ID in the queue is updated regularly.

In S807, first class data of different online stores are respectively requested from the e-commerce platform according to the online store IDs in the unexecuted priority queue using the plurality of first threads.

According to an example embodiment, requesting the corresponding first type of data from the e-commerce platform is accomplished by processing the online store ID in the unexecuted priority queue. By performing the acquisition of the first type of data in a multithreading manner, the data acquisition efficiency can be improved.

Furthermore, by controlling the number of threads, the interface capabilities provided by the e-commerce platform may be maximized and no computing and network resources are wasted as much as possible, according to some embodiments.

In S809, when there is no data in the non-execution priority queue and the in-execution priority queue, the first threads are utilized to request the first type data of different online stores from the e-commerce platform according to the online store IDs in the non-execution regular queue.

As before, by performing the acquisition of the first type of data in a multithreading manner, the data acquisition efficiency may be improved.

The method according to the embodiment shown in fig. 8 may otherwise employ a similar process to the other embodiments described above, and will not be described in detail here.

Referring to fig. 9, according to an example embodiment, a plurality of threads for acquiring online store data are executed on at least two servers a and B.

As shown in fig. 9, the data to be acquired includes four types, which are order data, sales data, report data, and document data, respectively. Each server a or server B creates 3 threads on each class of data to acquire data from the e-commerce platform for online stores. As previously described, the number of threads may be adjusted based on the status of the server and network resources and the limitations of the e-commerce platform on the interface requests.

Referring to fig. 9, for each thread, the online store ID is first acquired, and then the data of the corresponding online store is acquired from the e-commerce platform.

According to an embodiment, each thread first obtains the online store ID from the unexecuted priority queue, and places the obtained online store ID into the in-execution priority queue. And then, according to the acquired online store ID, completing the request of corresponding e-commerce data from the e-commerce platform, and removing the online store ID from the in-process priority queue. Then, another online store ID is acquired from the non-execution priority queue, and the process of acquiring corresponding data is continued.

If the online store ID is not acquired from the non-execution priority queue, but the execution priority queue is not empty, the online store ID is returned after delay, and the online store ID acquisition operation is continued. In this way, it is ensured that the data of the online store in the conventional queue starts to be acquired from the e-commerce platform after the data of the online store in the priority queue is acquired from the e-commerce platform is preferentially completed.

If no online store ID is obtained from the unexecuted priority queue and the in-execution priority queue is empty, indicating that no online stores need to be prioritized at this time, processing of online stores in the regular queue is started, and the online store ID is obtained from the unexecuted regular queue. And putting the acquired online store ID into an executing conventional queue, and completing the request of corresponding e-commerce data from the e-commerce platform according to the acquired online store ID. Then, the online store ID is removed from the executing regular queue, another online store ID is tried to be acquired from the non-executing priority queue, and the process of acquiring data is continued.

As shown in fig. 10, the computing device 30 includes a processor 12 and a memory 14. Computing device 30 may also include a bus 22, a network interface 16, and an I/O interface 18. The processor 12, memory 14, network interface 16, and I/O interface 18 may communicate with each other via a bus 22.

The processor 12 may include one or more general purpose CPUs (Central Processing Unit, central processing units), microprocessors, or application specific integrated circuits, etc. for executing associated program instructions.

Memory 14 may include machine-system-readable media in the form of volatile memory, such as Random Access Memory (RAM), read Only Memory (ROM), and/or cache memory. Memory 14 is used to store one or more programs including instructions as well as data. The processor 12 may read instructions stored in the memory 14 to perform the methods described above in accordance with embodiments of the present application.

Computing device 30 may also communicate with one or more networks through network interface 16. The network interface 16 may be a wired network interface or a wireless network interface, or may be a virtual network interface.

Computing device 30 may also communicate with one or more external devices (e.g., audio input devices, audio output devices, cameras, keyboards, mice, displays, various types of sensors, etc.) through input/output (I/O) interface 18.

Bus 22 may include an address bus, a data bus, a control bus, and the like. Bus 22 provides a path for exchanging information between the components.

It should be noted that, in the implementation, the computing device 30 may further include other components necessary to achieve normal operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.

The present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of the above method. The computer readable storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, DVDs, CD-ROMs, micro-drives, and magneto-optical disks, ROM, RAM, EPROM, EEPROM, DRAM, VRAM, flash memory devices, magnetic or optical cards, nanosystems (including molecular memory ICs), network storage devices, cloud storage devices, or any type of media or device suitable for storing instructions and/or data.

The present application also provides a computer program product comprising a non-transitory computer readable storage medium storing a computer program operable to cause a computer to perform part or all of the steps of any one of the methods described in the method embodiments above.

It will be clear to a person skilled in the art that the solution of the present application may be implemented by means of software and/or hardware. "Unit" and "module" in this specification refer to software and/or hardware capable of performing a specific function, either alone or in combination with other components, where the hardware may be, for example, a field programmable gate array, an integrated circuit, or the like.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of action combinations, but it should be understood by those skilled in the art that the present application is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required in the present application.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative, such as the division of the units, merely a logical function division, and there may be additional manners of dividing the actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some service interface, device or unit indirect coupling or communication connection, electrical or otherwise.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a memory, including several instructions for causing a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application.

The embodiments of the present application have been described and illustrated in detail above. It should be clearly understood that this application describes how to make and use particular examples, but is not limited to any details of these examples. Rather, these principles can be applied to many other embodiments based on the teachings of the present disclosure.

Those skilled in the art will readily appreciate from the description of example embodiments that the risk rating prediction method according to embodiments of the present application has at least one or more of the following advantages.

According to some embodiments, by controlling the number of threads, the interface capabilities provided by the e-commerce platform may be maximized.

According to some embodiments, by controlling the number of threads, computing resources and network resources are not wasted as much as possible.

According to some embodiments, by employing a thread-safe queue to store online store IDs, it may be ensured that the problem of online store IDs being repeatedly acquired does not occur.

According to some embodiments, the in-execution queue is set, so that visualization and real-time monitoring of an online store in the data grabbing process can be realized.

According to some embodiments, by setting an in-execution priority queue, priority acquisition of newly added online store data, for example, may be further ensured.

According to some embodiments, each time the online store ID is acquired to request corresponding data from the e-commerce platform, the online store ID is acquired from the unexecuted priority queue, so that the unexecuted priority queue can be ensured to be processed preferentially when the online store ID is newly added.

According to some embodiments, when the priority queue is not empty in execution, the online store ID is not acquired from the regular queue, so that it may be ensured that the online store data in the regular queue starts to be acquired from the e-commerce platform after the online store data in the priority queue is preferentially acquired from the e-commerce platform.

According to some embodiments, through setting different queues and combining multithreading to acquire data, priority of acquiring certain online store data can be realized, and meanwhile, the data acquisition efficiency is improved.

The foregoing may be better understood in light of the following clauses:

1. a method for acquiring big data of multiple home appliances by using a priority queue, comprising:

Setting a queue for storing the online store ID, wherein the queue comprises a non-execution regular queue, an execution regular queue, a non-execution priority queue and an execution priority queue;

updating the online store ID in the queue at regular time;

preferentially finishing requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the unexecuted priority queue;

and when no data exists in the non-execution priority queue and the execution priority queue, requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the non-execution conventional queue.

2. The method of clause 1, wherein the periodically updating the online store ID in the queue comprises:

inquiring a list of online stores authorized by a client at regular time to acquire an online store ID set;

excluding the online store IDs in the queue from the set of online store IDs;

adding the reserved online store ID of the newly added online store to an unexecuted priority queue;

the reserved online store ID of the non-new online store is added to the unexecuted regular queue.

3. The method of clause 2, wherein,

the online store ID set comprises a first set and a second set, wherein the first set comprises online store IDs of newly added online stores, and the second set comprises online store IDs of non-new online stores;

Excluding the online store IDs in the queue from the set of online store IDs, comprising:

excluding from the first set the online store IDs of the unexecuted priority queue and the in-execution priority queue;

the online store IDs of the unexecuted regular queue and the in-execution regular queue are excluded from the second set.

4. The method of clause 1, wherein requesting the corresponding e-commerce data from the e-commerce platform is preferably accomplished according to the online store ID in the unexecuted priority queue, comprising repeatedly performing the operations of:

acquiring an online store ID from the unexecuted priority queue;

placing the acquired online store ID into the execution priority queue;

and after the corresponding electronic commerce data is requested to the electronic commerce platform according to the acquired online store ID, removing the online store ID from the execution priority queue.

5. The method of clause 4, wherein when there is no data in the non-executing priority queue and the executing priority queue, requesting corresponding e-commerce data from the e-commerce platform according to the online store ID in the non-executing regular queue, including repeatedly executing the following operations:

acquiring an online store ID from the unexecuted priority queue;

If the online store ID is not acquired from the non-execution priority queue and the execution priority queue is not empty, returning to acquiring the online store ID from the non-execution priority queue after delay;

if the online store ID is not acquired from the unexecuted priority queue and the in-execution priority queue is empty, acquiring the online store ID from the unexecuted regular queue;

placing the acquired online store ID into the executing routine queue;

and after the corresponding electronic commerce data is requested to the electronic commerce platform according to the acquired online store ID, removing the online store ID from the executing conventional queue.

6. The method of clause 1, further comprising: and storing the electronic commerce data in a distributed file system.

7. The method of clause 6, wherein the distributed file system comprises a FastDFS system.

8. The method of clause 6, further comprising: and storing the e-commerce data in the distributed file system in a distributed database system.

9. The method of clause 8, wherein the distributed database system comprises a TiDB system.

10. The method of clause 1, wherein the following operations are performed in a plurality of threads:

Preferentially finishing requesting corresponding E-commerce data from the E-commerce platform according to the online store ID in the unexecuted priority queue; and

11. The method of clause 10, wherein the unexecuted regular queue and the unexecuted priority queue are single threaded memory storage system based queues.

12. The method of clause 11, wherein the single threaded memory storage system comprises a Redis system.

13. A computing device, comprising:

a processor;

a memory having a computer program stored thereon;

the method of any one of clauses 1-12 being implemented when the processor executes the computer program.

Exemplary embodiments of the present application are specifically illustrated and described above. It is to be understood that this application is not limited to the details of construction, arrangement or method of implementation described herein; on the contrary, the intention is to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

updating the online store ID in the queue at regular time;

2. The method of claim 1, wherein the periodically updating the online store IDs in the queue comprises:

excluding the online store IDs in the queue from the set of online store IDs;

3. The method of claim 2, wherein,

4. The method of claim 1, wherein prioritizing completion of the request for the corresponding e-commerce data from the e-commerce platform based on the online store ID in the unexecuted priority queue comprises repeating the operations of:

acquiring an online store ID from the unexecuted priority queue;

placing the acquired online store ID into the execution priority queue;

5. The method of claim 4, wherein requesting corresponding e-commerce data from the e-commerce platform based on the online store ID in the unexecuted regular queue when there is no data in both the unexecuted priority queue and the in-execution priority queue comprises repeating the operations of:

acquiring an online store ID from the unexecuted priority queue;

placing the acquired online store ID into the executing routine queue;

6. The method as recited in claim 1, further comprising: and storing the electronic commerce data in a distributed file system.

7. The method of claim 6 wherein the distributed file system comprises a FastDFS system.

8. The method as recited in claim 6, further comprising: and storing the e-commerce data in the distributed file system in a distributed database system.

9. The method of claim 8, wherein the distributed database system comprises a TiDB system.

10. The method of claim 1, wherein the following operations are performed in a plurality of threads:

11. The method of claim 10, wherein the unexecuted regular queue and the unexecuted priority queue are single threaded memory storage system based queues.

12. The method of claim 11, wherein the single-threaded memory storage system comprises a dis system.

13. A computing device, comprising:

a processor;

a memory having a computer program stored thereon;

the method of any of claims 1-12 being implemented when the processor executes the computer program.