CN112231311A

CN112231311A - Method, system and computer readable medium for integrating cross-platform commodity data

Info

Publication number: CN112231311A
Application number: CN202011147716.6A
Authority: CN
Inventors: 王泰舟
Original assignee: Shiheng Shanghai Technology Service Co ltd
Current assignee: Shiheng Shanghai Technology Service Co ltd
Priority date: 2020-10-23
Filing date: 2020-10-23
Publication date: 2021-01-15

Abstract

The invention provides a method, a system and a computer readable medium for integrating cross-platform commodity data. The method comprises the following steps: pulling commodity list data from a plurality of commodity platforms and storing the commodity list data into a stream processing platform; monitoring commodity list data pulled by the stream processing platform, and storing the commodity list data into an original database; performing data cleaning on the commodity list data to obtain structured platform metadata, and storing the structured platform metadata in a structured database; and associating the platform metadata with the merchant data to obtain mapping metadata with a mapping relation, and storing the mapping metadata in a heterogeneous result database.

Description

Method, system and computer readable medium for integrating cross-platform commodity data

Technical Field

The present invention relates generally to the field of data processing, and more particularly to a method, system, and computer readable medium for integrating cross-platform merchandise data.

Background

With the increase of the number of commodity order platforms, the same merchant may need to operate on a plurality of commodity order platforms, each platform having its own commodity system. The data for the system of these commodities is heterogeneous and difficult to unify. This makes it difficult for the merchant to process its own merchandise data.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a method, a system and a computer readable medium for integrating cross-platform commodity data, which can integrate the commodity data of a plurality of commodity platforms so as to carry out analysis uniformly.

In order to solve the technical problem, the invention provides a method for integrating cross-platform commodity data, which comprises the following steps: pulling commodity list data from a plurality of commodity platforms and storing the commodity list data into a stream processing platform; monitoring commodity list data pulled by the stream processing platform, and storing the commodity list data into an original database; performing data cleaning on the commodity list data to obtain structured platform metadata, and storing the structured platform metadata in a structured database; and associating the platform metadata with the merchant data to obtain mapping metadata with a mapping relation, and storing the mapping metadata in a heterogeneous result database.

In an embodiment of the present invention, the step of performing data cleansing on the commodity list data includes: removing special characters in the commodity list data; removing repeated data in the commodity list data; carrying out consistency check on the commodity inventory data; filtering invalid data in the commodity list data according to business requirements; and analyzing the special format in the commodity list data to obtain the structured platform metadata.

In an embodiment of the present invention, the step of associating the platform metadata with the merchant data to obtain mapping metadata with mapping relationship includes: a. judging whether the current commodity has the mapped local commodity identification, if so, ending, otherwise, entering the step b; b. judging whether the current commodity supports an external commodity identifier, if so, entering the step c, otherwise, entering the step d; c. judging whether the current commodity is configured with an external commodity identifier, if so, entering a step e, and otherwise, entering a step d; and d, searching related local commodity identifications according to the keywords, and associating the current commodity with the searched local commodity identifications.

In an embodiment of the present invention, the step of associating the platform metadata with the merchant data to obtain mapping metadata with mapping relationship further includes: when the relevant local commodity identification is not searched in the step d, generating an abnormal event comprising unmapped commodities; and reminding a user of the abnormal event.

In an embodiment of the present invention, the step of associating the platform metadata with the merchant data to obtain mapping metadata with mapping relationship further includes: and receiving manual binding of the unmapped commodities and local commodity identifications by the user.

In one embodiment of the invention, the method for searching for the relevant local commodity identification according to the keywords comprises determining a finite automaton algorithm.

In an embodiment of the present invention, the step of checking the consistency of the merchandise list data includes: and checking whether the commodity list data meet the requirements or not according to the reasonable value range and the mutual relation of each variable.

In an embodiment of the present invention, the method further includes performing aggregation analysis on the commodity list data by using the mapping metadata in the heterogeneous result database.

Another aspect of the present invention provides a system for integrating cross-platform merchandise data, comprising: a memory for storing instructions executable by the processor; and a processor for executing the instructions to implement the method as described above.

Another aspect of the invention proposes a computer-readable medium having stored thereon computer program code which, when executed by a processor, implements the method as described above.

Compared with the prior art, the method and the system have the advantages that after the commodity data of the plurality of commodity platforms are obtained, a unified commodity integration system is abstracted through the steps of data cleaning, mapping relation establishment and the like, so that unified analysis can be carried out on the basis, and the data processing efficiency of merchants is greatly improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the application and together with the description serve to explain the principle of the invention. In the drawings:

FIG. 1 is a block diagram of a system for integrating cross-platform merchandise data according to an embodiment of the present application.

Fig. 2 is a schematic diagram of a method for integrating cross-platform commodity data according to an embodiment of the present application.

Fig. 3 is a schematic diagram of a relationship mapping process according to an embodiment of the present application.

FIG. 4 is a diagram illustrating a process for processing unmapped data according to an embodiment of the present application.

Fig. 5 is a schematic diagram of a commodity data analysis process according to an embodiment of the present application.

FIG. 6 is a system hardware implementation environment for integrating cross-platform merchandise data according to an embodiment of the present application.

Detailed Description

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. It is obvious that the drawings in the following description are only examples or embodiments of the application, from which the application can also be applied to other similar scenarios without inventive effort for a person skilled in the art. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.

As used in this application and the appended claims, the terms "a," "an," "the," and/or "the" are not intended to be inclusive in the singular, but rather are intended to be inclusive in the plural unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.

The relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present application unless specifically stated otherwise. Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description. Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate. In all examples shown and discussed herein, any particular value should be construed as merely illustrative, and not limiting. Thus, other examples of the exemplary embodiments may have different values. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

In the description of the present application, it is to be understood that the orientation or positional relationship indicated by the directional terms such as "front, rear, upper, lower, left, right", "lateral, vertical, horizontal" and "top, bottom", etc., are generally based on the orientation or positional relationship shown in the drawings, and are used for convenience of description and simplicity of description only, and in the case of not making a reverse description, these directional terms do not indicate and imply that the device or element being referred to must have a particular orientation or be constructed and operated in a particular orientation, and therefore, should not be considered as limiting the scope of the present application; the terms "inner and outer" refer to the inner and outer relative to the profile of the respective component itself.

Flow charts are used herein to illustrate operations performed by systems according to embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, various steps may be processed in reverse order or simultaneously. Meanwhile, other operations are added to or removed from these processes.

FIG. 1 is a logic block diagram of a system for integrating cross-platform merchandise data according to an embodiment of the present application. Referring to FIG. 1, system 100 is coupled to a plurality of merchandise systems 200 and clients 300 for interacting with the devices. The plurality of merchandise platforms 200 may be, for example, various commercial platforms such as Mei Tuo, hungry, Jingdong, hungry, and so on. System 100 may include a data layer module 110, an application module 120, and a configuration center 130. The data layer module 120 can pull the commodity list data from the commodity system 200 and perform data processing internally, so as to obtain structured commodity data, and store the structured commodity data in the database. The application module 120 includes a commodity sorting module 121 and a business module 122. The commodity sorting module 121 sorts the structured commodity data to obtain mapped commodity data, and stores the mapped commodity data in the database. The business module 122 can utilize the mapped commodity data for various types of analysis with valuable results. These results may be queried by clients 300 by way of web service 123. The configuration center 130 can configure the operation of the data layer module 110 and the application module 120. In one embodiment of the present invention, the commodity is, for example, dishes.

Fig. 2 is a schematic diagram of a method for integrating cross-platform commodity data according to an embodiment of the present application. According to this method, the stream processing platform 211, the raw database 212, the structured database 212, and the heterogeneous result database 213 are configured to facilitate the exchange of data between the various steps.

Referring to fig. 2, a method for integrating cross-platform commodity data of the present embodiment includes the following steps:

in step 201, commodity inventory data is pulled from a plurality of commodity systems and stored in a stream processing platform.

The plurality of merchandise platforms 200 may be, for example, various commercial platforms such as Mei Tuo, hungry, Jingdong, hungry, and so on. Each commodity system comprises business data such as commodity list data, order data and the like. Generally, data in each commodity system is inconsistent in terms of field naming, data format and the like, namely heterogeneous data.

Stream processing platform 211 may store the pulled inventory data for use in subsequent steps of the process. Stream processing platform 211 may relieve the performance pressure of the system and decouple commodity system 200 from integrated system 100. For example, the stream processing platform 211 may be a kafka platform.

In step 202, the monitoring stream processing platform pulls the commodity list data and stores the commodity list data in the original database.

In this step, the system consumes the inventory data captured by the monitoring stream processing platform and stores the inventory data as raw data in the raw database 212. The commodity list data is stored as original data, so that data problems can be conveniently traced.

In step 203, the commodity inventory data is data-cleaned to obtain structured platform metadata, and the structured platform metadata is stored in a structured database.

In this step, in order to ensure the stability of the data, the original data is cleaned and the cleaned data is stored. The goals of data cleansing are as follows: clearing special characters which may affect the operation and accuracy of the program, clearing invalid data and error data, and checking consistency.

In one embodiment, the step of data cleansing includes removing special characters from the inventory data. Some special characters, such as \ emoji and other special characters, are destructive to system stability, which may cause system errors and affect the comparison of some data. It is therefore necessary to remove these special characters.

In one embodiment, the step of data cleansing includes removing duplicate data from the inventory data. Specifically, the repeated data caused by the situations of re-running, run-up and the like in the data acquisition process is deleted, otherwise, the calculated data is inaccurate. It is noted that different service deduplication logic is different, and therefore the deduplication logic may be set according to the service.

In one embodiment, the step of data cleansing includes performing a consistency check on the inventory data to prevent undesirable data conditions caused by platform updates or occasional errors. The consistency check mainly comprises checking whether the commodity list data meets the requirements according to the reasonable value range and the mutual relation of each variable. The case of inconsistency includes: data out of the normal range, data logically unreasonable, data contradiction, and the like.

In one embodiment, the step of data cleansing includes filtering invalid data in the inventory data based on business requirements. Specifically, unwanted data is filtered out according to traffic requirements. If the meal boxes are not commodities but are collected into the integrated system 100 during the subsequent commodity analysis process, the statistics are affected, and the filtering operation is directly performed here.

In one embodiment, the step of data cleansing includes parsing a special format in the inventory data to obtain structured platform metadata. Specifically, the raw data from the commodity system may have various formats such as xml, json, character strings, files, and the like, and if the business cannot be used directly, the raw data needs to be analyzed to obtain structured data.

The structured data processed as described above is stored in the structured database 212 as platform metadata. The structured database 212 may be used while the integration system 100 is offline.

In step 204, the platform metadata is associated with the merchant data to obtain mapping metadata with mapping relationships, and the mapping metadata is stored in the heterogeneous result database.

In this step, relational mapping is performed, that is, the cleaned platform metadata is associated with the original merchant data in the integration system 100 to be converted into mapping metadata with mapping relationships, so that data analysis is facilitated.

Fig. 3 is a schematic diagram of a relationship mapping process according to an embodiment of the present application. Referring to fig. 3, after obtaining the heterogeneous relationship data, first, in step 301, it is determined whether the current product has a mapped local product identifier, if so, the process is ended, otherwise, the process proceeds to step 302. In step 302, it is determined whether the current product supports the external product identifier, if so, step 303 is entered, otherwise, step 304 is entered. In step 303, it is determined whether the current product is configured with an external product identifier, if so, the machine sorting is directly performed, for example, identifier mapping is performed, otherwise, the process proceeds to step 304. At step 304, relevant local merchandise identifiers are searched for based on the keywords. In step 305, it is determined whether a matching local merchandise flag is searched, if yes, step 306 is entered, and the current merchandise is associated with the searched local merchandise flag. Otherwise, step 307 is entered, an abnormal event including the unmapped goods is generated, and the user is reminded of the abnormal event.

In one embodiment, a method for searching for relevant local item identifications based on keywords includes determining a finite automata (DFA) algorithm.

In one embodiment, the step of associating the platform metadata with the merchant data to obtain mapping metadata with a mapping relationship receives a manual binding of an unmapped good to a local good identification by a user. FIG. 4 is a diagram illustrating a process for processing unmapped data according to an embodiment of the present application. Referring to fig. 4, in step 401, the system 100 obtains abnormal data of unmapped goods, and in step 402, the system notifies the relevant user to perform manual binding. In step 403, the user logs in the system to pull the abnormal goods and compare the unmapped goods with the local goods one by one. In step 404, it is determined whether there are unmapped goods and local goods with the same name, and if yes, they are manually bound in step 405, and the system records the mapping relationship. If the same name or the same type of commodity does not exist, in step 406, the user creates a system commodity, and jumps to step 405 to operate again until the mapping relations of all the platform commodities are bound.

Fig. 5 is a schematic diagram of a commodity data analysis process according to an embodiment of the present application. Referring to fig. 5, in step 501, commodity list data is obtained, and the commodity list data comes from each commodity system 200 and is heterogeneous data. At step 502, a product mapping relationship is obtained, the mapping relationship being from the results collated in the previous step 204. In step 503, it is determined whether there is a mapping relationship, if there is a mapping relationship, in step 504, the product of the platform 300 is converted into the product of the system 100, and if there is no mapping relationship, the product is directly discarded, and the process is ended. After conversion to a system commodity, various polymerization analyses can be performed. For example, daily sales of system commodities are calculated and stored at step 505. The daily sales of the system commodities is the accumulated value of the daily sales of the commodities. As another example, a single instance of the computing system commodity may be received and stored at step 506. An exemplary single-average revenue calculation formula is as follows:

(commodity sales volume, commodity original price)/order total price) order real collection

In step 507, it is determined whether the system commodity has a cost, if so, the gross profit is calculated and stored, and if not, the process is ended. Exemplary gross profit calculation formula:

(receipt actual sales) ((cost of goods + cost of packaging))

In one embodiment, the calculated gross profit data is presented to the user by the web service 123 of FIG. 1 as aggregated as needed for profit analysis. In one embodiment, the aggregated calculated sales is presented to the user via web service 123 for sales analysis as needed.

FIG. 6 is a system hardware implementation environment for integrating cross-platform merchandise data according to an embodiment of the present application. Integrated system 600 may include internal communication bus 601, Processor (Processor)602, Read Only Memory (ROM)603, Random Access Memory (RAM)604, and communication port 605. When used on a personal computer, the integrated system 600 may also include a hard disk 607. The internal communication bus 601 may enable data communication among the components of the integrated system 600. Processor 602 may make the determination and issue a prompt. In some embodiments, the processor 602 may be comprised of one or more processors. The communication port 605 may enable the integration system 600 to communicate data with the outside. In some embodiments, integration system 600 may send and receive information and data from a network through communication port 605. The integrated system 600 may also include various forms of program storage units and data storage units, such as a hard disk 607, a Read Only Memory (ROM)603 and a Random Access Memory (RAM)604, capable of storing various data files for computer processing and/or communication, as well as possible program instructions for execution by the processor 602. The processor executes these instructions to implement the main parts of the method. The results processed by the processor are communicated to the user device through the communication port and displayed on the user interface.

The method for integrating cross-platform commodity data can be implemented as a computer program, stored in the hard disk 607, and recorded in the processor 602 for execution, so as to implement the method for integrating cross-platform commodity data of the present application.

The present application also provides a computer readable medium having stored thereon computer program code which, when executed by a processor, implements the integrated cross-platform merchandise data as described above.

Having thus described the basic concept, it will be apparent to those skilled in the art that the foregoing disclosure is by way of example only, and is not intended to limit the present application. Various modifications, improvements and adaptations to the present application may occur to those skilled in the art, although not explicitly described herein. Such modifications, improvements and adaptations are proposed in the present application and thus fall within the spirit and scope of the exemplary embodiments of the present application.

Also, this application uses specific language to describe embodiments of the application. Reference throughout this specification to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the present application is included in at least one embodiment of the present application. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, some features, structures, or characteristics of one or more embodiments of the present application may be combined as appropriate.

Aspects of the present application may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. The processor may be one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), digital signal processing devices (DAPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, or a combination thereof. Furthermore, aspects of the present application may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disk, floppy disk, magnetic strips … …), optical disks (e.g., Compact Disk (CD), Digital Versatile Disk (DVD) … …), smart cards, and flash memory devices (e.g., card, stick, key drive … …).

The computer readable medium may comprise a propagated data signal with the computer program code embodied therein, for example, on a baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, and the like, or any suitable combination. The computer readable medium can be any computer readable medium that can communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or device. Program code on a computer readable medium may be propagated over any suitable medium, including radio, electrical cable, fiber optic cable, radio frequency signals, or the like, or any combination of the preceding.

Similarly, it should be noted that in the preceding description of embodiments of the application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not intended to require more features than are expressly recited in the claims. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.

Numerals describing the number of components, attributes, etc. are used in some embodiments, it being understood that such numerals used in the description of the embodiments are modified in some instances by the use of the modifier "about", "approximately" or "substantially". Unless otherwise indicated, "about", "approximately" or "substantially" indicates that the number allows a variation of ± 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that may vary depending upon the desired properties of the individual embodiments. In some embodiments, the numerical parameter should take into account the specified significant digits and employ a general digit preserving approach. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the range are approximations, in the specific examples, such numerical values are set forth as precisely as possible within the scope of the application.

Although the present application has been described with reference to the present specific embodiments, it will be recognized by those skilled in the art that the foregoing embodiments are merely illustrative of the present application and that various changes and substitutions of equivalents may be made without departing from the spirit of the application, and therefore, it is intended that all changes and modifications to the above-described embodiments that come within the spirit of the application fall within the scope of the claims of the application.

Claims

1. A method of integrating cross-platform commodity data, comprising the steps of:

pulling commodity list data from a plurality of commodity platforms and storing the commodity list data into a stream processing platform;

monitoring commodity list data pulled by the stream processing platform, and storing the commodity list data into an original database;

performing data cleaning on the commodity list data to obtain structured platform metadata, and storing the structured platform metadata in a structured database; and

and associating the platform metadata with the merchant data to obtain mapping metadata with a mapping relation, and storing the mapping metadata in a heterogeneous result database.

2. The method of claim 1, wherein the step of data cleansing the inventory data comprises:

removing special characters in the commodity list data;

removing repeated data in the commodity list data;

carrying out consistency check on the commodity inventory data;

filtering invalid data in the commodity list data according to business requirements; and

and analyzing a special format in the commodity list data to obtain the structured platform metadata.

3. The method of claim 1, wherein associating the platform metadata with merchant data, the step of obtaining mapping metadata with a mapping relationship comprises:

a. judging whether the current commodity has the mapped local commodity identification, if so, ending, otherwise, entering the step b;

b. judging whether the current commodity supports an external commodity identifier, if so, entering the step c, otherwise, entering the step d;

c. judging whether the current commodity is configured with an external commodity identifier, if so, entering a step e, and otherwise, entering a step d;

d. and searching related local commodity identifications according to the keywords, and associating the current commodity with the searched local commodity identifications.

4. The method of claim 3, wherein associating the platform metadata with merchant data, the step of obtaining mapping metadata with a mapping relationship further comprises:

when the relevant local commodity identification is not searched in the step d, generating an abnormal event comprising unmapped commodities;

and reminding a user of the abnormal event.

5. The method of claim 4, wherein associating the platform metadata with merchant data, the step of obtaining mapping metadata with a mapping relationship further comprises:

and receiving manual binding of the unmapped commodities and local commodity identifications by the user.

6. A method, according to claim 4, wherein the method of searching for relevant local merchandise identifiers based on keywords comprises determining a finite automaton algorithm.

7. The method of claim 2, wherein the step of checking the inventory data for consistency comprises: and checking whether the commodity list data meet the requirements or not according to the reasonable value range and the mutual relation of each variable.

8. The method of claim 1, further comprising performing an aggregate analysis of the inventory data using mapping metadata in the heterogeneous results database.

9. A method of integrating cross-platform commodity data, comprising:

a memory for storing instructions executable by the processor; and

a processor for executing the instructions to implement the method of any one of claims 1-8.

10. A computer-readable medium having stored thereon computer program code which, when executed by a processor, implements the method of any of claims 1-8.