CN114898349A - Target commodity identification method and device, equipment, medium and product thereof - Google Patents

Abstract

The application discloses a target commodity identification method and a device, equipment, a medium and a product thereof, wherein the method comprises the following steps: acquiring a commodity title and a commodity picture in commodity information of a target commodity; extracting deep semantic information of the commodity picture and the commodity title; fusing the deep semantic information of the commodity title to the deep semantic information of the commodity picture so as to highlight the image characteristics of the target commodity in the deep semantic information of the commodity picture according to the deep semantic information of the commodity title and obtain image-text fusion characteristic information; and inputting the image-text fusion characteristic information into a target detection model trained to be convergent in advance, and identifying the target commodity. The method and the device can accurately identify the target commodity in the commodity picture.

Description

Target commodity identification method and its device, equipment, medium and product

technical field

The present application relates to the field of e-commerce information technology, and in particular, to a target commodity identification method and its corresponding device, computer equipment, computer-readable storage medium, and computer program product.

Background technique

In order to set off the effect of the products on the shelves, sellers and users on the e-commerce platform usually match the products on the shelves with other matching items. The product pictures of the products on the shelves include not only the products on the shelves but also other matching items. For example, the item on the shelf is clothes, and other matching items may be trousers, hats, shoes, etc.; the item on the shelf is a shelf, and other matching items may be household appliances, books, ornaments, etc. Therefore, it is not possible to determine the corresponding product on the shelf simply through the product image.

In the e-commerce platform, it is often necessary to identify the corresponding image of the target product from the product image containing the target product, which is used to achieve other downstream tasks, such as achieving product similarity matching, product image display, etc. If the image of the corresponding target product cannot be quickly obtained from the product image, the efficiency of implementing related downstream tasks will be affected, and the related functions may not be implemented, resulting in decreased user experience.

Traditional solutions often use multi-target recognition methods to identify each item from the product image, including the listed product and its matching items, and then classify and predict according to each item to determine which object belongs to the product. Image. This method requires two-stage processing. The two stages need to be implemented with different models, and different models need to perform image preprocessing and other operations on the image. The process is cumbersome, the efficiency is relatively low, and the more troublesome is The two-stage related models generally need to be trained with corresponding data sets respectively, and the training cost is high.

In view of this, the applicant attempts to explore other ways to quickly identify the target product from the product images containing the target product.

SUMMARY OF THE INVENTION

The primary purpose of the present application is to solve at least one of the above problems and provide a target commodity identification method and its corresponding apparatus, computer equipment, computer-readable storage medium, and computer program product.

In order to meet the various purposes of the application, the application adopts the following technical solutions:

A target commodity identification method provided in accordance with one of the purposes of this application includes the following steps:

Obtain the product title and product image in the product information of the target product;

Extract the deep semantic information of the product image and product title;

Integrating the deep semantic information of the product title with the deep semantic information of the product image, so as to highlight the image features of the target product in the deep semantic information of the product image according to the deep semantic information of the product title, Obtain image-text fusion feature information;

The image-text fusion feature information is input into the target detection model pre-trained to convergence, and the target product is identified.

In a further embodiment, the step of extracting the deep semantic information of the product picture and the product title includes the following steps:

Preprocessing the product picture, and inputting the preprocessed product picture into an image feature extraction model trained in advance to converge to obtain corresponding deep semantic information, which is used to characterize the image features of the product picture;

The product title is preprocessed, and the preprocessed product title is input into a text feature extraction model that has been pre-trained to convergence to obtain corresponding deep semantic information, which is used to characterize the text feature of the product title.

In a further embodiment, the step of preprocessing the product title includes the following steps:

filter invalid characters in the title of the product;

Perform word segmentation on the filtered product title to obtain keywords therein, where the keywords include product words and/or brand words of the target product, and complete the preprocessing of the product title.

In a further embodiment, the deep semantic information of the product title is fused to the deep semantic information of the product image, so as to highlight the deep semantic information of the target product in the product image according to the deep semantic information of the product title. Image features in the information, the steps of obtaining image-text fusion feature information include the following steps:

A multi-modal feature interaction fusion module is used to fuse the deep semantic information of the product title and the deep semantic information of the product image to obtain preliminary fusion feature information, in which the features of the image of the target product are significantly represented in the preliminary fusion feature information. ;

combining the preliminary fusion feature information with the deep semantic information of the commodity image to obtain image-text fusion feature information;

In a preferred embodiment, a multi-modal feature interaction fusion module is used to fuse the deep semantic information of the product title and the deep semantic information of the product image, and the step of obtaining preliminary fusion feature information includes the following steps:

Construct a query vector with the deep semantic information of the commodity picture, construct a key vector and a value vector with the deep semantic information of the commodity title, and input the attention layer;

Interacting and normalizing the query vector and the key vector by the attention layer to obtain a weight matrix;

Preliminary fusion feature information is obtained by matching the value vector with the weight matrix by the attention layer.

In a further embodiment, the image-text fusion feature information is input into a pre-trained target detection model to converge, and the step of identifying the target product includes the following steps:

Use a pre-trained target detection model to converge to detect the target product in the product image according to the image-text fusion feature information, and obtain a corresponding detection area;

A rectangular frame with the smallest area surrounding the detection area is obtained, and the target commodity is selected from the frame as the recognition result.

In an extended embodiment, the image-text fusion feature information is input into a pre-trained target detection model that has converged, and after the step of identifying the target product, the following steps are further included:

Cut out the image of the target product from the product picture according to the rectangular frame of the selected target product, and store the unique identification code associated with the target product in the product database;

In response to the product recommendation request, search the product database according to the unique identification code of the target product to obtain the image of the target product, and match the recommended products similar to it;

Respond to the product recommendation request, and push the recommended product.

A target commodity identification device provided in accordance with one of the purposes of this application includes: a graphic and text acquisition module, a semantic extraction module, a feature fusion module and a target identification module, wherein the graphic and text acquisition module is used for acquiring the commodities of the target commodity. The product title and product image in the information; the semantic extraction module is used to extract the deep semantic information of the product image and the product title; the feature fusion module is used to fuse the deep semantic information of the product title into the product image. deep semantic information, to highlight the image features of the target product in the deep semantic information of the product picture according to the deep semantic information of the product title, and obtain image-text fusion feature information; the target recognition module is used to The image-text fusion feature information is input into the pre-trained to convergent target detection model, and the target product is identified.

In a further embodiment, the semantic extraction module includes: an image feature extraction sub-module, configured to preprocess the commodity picture, input the preprocessed commodity picture into the image feature extraction model that has been pre-trained to convergence, and obtain the corresponding The deep semantic information is used to characterize the image features of the product picture; the text feature extraction sub-module is used to preprocess the product title, and input the preprocessed product title into the text feature extraction model that has been pre-trained to convergence, Corresponding deep semantic information is obtained, which is used to characterize the textual features of the product title.

In a further embodiment, the image feature extraction sub-module includes: a character filtering unit for filtering invalid characters in the commodity title; a text segmentation unit for performing word segmentation on the filtered commodity title, and obtaining the Keywords, where the keywords include product words and/or brand words of the target commodity, to complete the preprocessing of the commodity title.

In a further embodiment, the feature fusion module includes: a semantic fusion sub-module, configured to use a multimodal feature interaction fusion module to fuse the deep semantic information of the product title and the deep semantic information of the product image to obtain a preliminary Fusion feature information, in the preliminary fusion feature information, features that significantly characterize the image of the target product; an information combination sub-module for combining the preliminary fusion feature information with the deep semantic information of the product image to obtain graphic and textual information fusion feature information;

In a preferred embodiment, the semantic fusion sub-module includes: a vector input unit for constructing a query vector with the deep semantic information of the product image, and constructing a key vector and a value vector with the deep semantic information of the product title. , input attention layer; weight extraction unit, used for interacting and normalizing the query vector and the key vector by the attention layer to obtain a weight matrix; feature generation unit, used by the attention layer The layer matches the value vector to the weight matrix to obtain preliminary fusion feature information.

In a further embodiment, the target identification module includes: a target detection unit, configured to detect the target commodity in the commodity picture according to the image-text fusion feature information by using a pre-trained target detection model to converge, and obtain the corresponding target commodity. A detection area; a frame selection identification unit, used to obtain a rectangular frame with a minimum area surrounding the detection area, and select a target commodity by its frame as a recognition result.

In an extended embodiment, after the target identification module, it further includes: an interception storage module, configured to intercept the image of the target commodity from the commodity picture according to the rectangular frame for selecting the target commodity, and associate it with the target commodity. The unique identification code of the commodity is stored in the commodity database; the response request module is used to respond to the commodity recommendation request, retrieve the commodity database according to the unique identification code of the target commodity to obtain the image of the target commodity, and match the recommended commodities similar to it; the response request module is used for Respond to the product recommendation request, and push the recommended product.

The technical solution of the present application has many advantages, including but not limited to the following aspects:

First, the present application uses the deep semantic information of the product title to highlight the image features of the target product in the deep semantic information of the product image, and provides key information indicating the target product in the product image, so that a single object can be identified based on the key information. Identify, quickly and accurately identify the target product from the product image;

Secondly, multi-modal feature fusion is used to fuse the deep semantic information of the product title and the deep semantic information of the product image, and then the target product can be identified according to the fusion features, which can simplify the model architecture and training steps used to realize the recognition, that is, The related models can be trained together by using a single data set, the training cost is low, and the training process is relatively simple. Moreover, it can be understood that the target commodity recognition realized by this application can be recognized in a single stage, and its execution efficiency is relatively high.

In addition, the target commodity identification function realized by the present application can be applied to various downstream tasks related to the e-commerce platform, such as commodity similarity matching, commodity classification, commodity labeling, etc., and it can be understood that since the target commodity can be extracted in a targeted manner , which helps provide exactly the images needed for downstream tasks.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a schematic flowchart of a typical embodiment of the target commodity identification method of the present application;

2 is a schematic diagram of a multimodal feature interaction module in an embodiment of the present application;

3 is a schematic diagram of the implementation architecture of the target commodity identification model adopted in the embodiment of the application;

4 is a schematic flowchart of extracting deep semantic information in an embodiment of the present application;

5 is a schematic flowchart of a preprocessing process of a commodity title in the embodiment of the application;

6 is a schematic flowchart of obtaining feature information of image-text fusion in an embodiment of the application;

7 is a schematic flowchart of obtaining preliminary image-text fusion feature information in an embodiment of the application;

8 is a schematic flowchart of identifying a target product from a product image in an embodiment of the application;

FIG. 9 is a schematic flowchart of commodity recommendation in the embodiment of the application;

10 is a schematic block diagram of the target commodity identification device of the application;

FIG. 11 is a schematic structural diagram of a computer device used in this application.

specific implementation

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, but not to be construed as a limitation on the present application.

It will be understood by those skilled in the art that the singular forms "a", "an", "the" and "the" as used herein can include the plural forms as well, unless expressly stated otherwise. It should be further understood that the word "comprising" used in the specification of this application refers to the presence of stated features, integers, steps, operations, elements and/or components, but does not preclude the presence or addition of one or more other features, Integers, steps, operations, elements, components and/or groups thereof. It will be understood that when we refer to an element as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Furthermore, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combination of one or more of the associated listed items.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It should also be understood that terms, such as those defined in a general dictionary, should be understood to have meanings consistent with their meanings in the context of the prior art and, unless specifically defined as herein, should not be interpreted in idealistic or overly formal meaning to explain.

Those skilled in the art can understand that the "client", "terminal" and "terminal device" used herein include both a wireless signal receiver device that only has a wireless signal receiver without transmission capability, and a wireless signal receiver device. A device with receive and transmit hardware that has receive and transmit hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices such as personal computers, tablet computers, which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display; PCS (Personal Communications Service, Personal Communications System) ), which can combine voice, data processing, fax and/or data communication capabilities; PDA (Personal Digital Assistant), which can include radio frequency receivers, pagers, Internet/Intranet access, web browsers, notepads , calendar and/or GPS (Global Positioning System) receivers; conventional laptop and/or palmtop computers or other devices having and/or conventional laptop and/or palmtop radio frequency receivers computer or other device. As used herein, "client", "terminal", "terminal device" may be portable, transportable, mounted in a vehicle (air, marine and/or land), or adapted and/or configured to be locally operate, and/or in distributed form, operate at any other location on Earth and/or space. The "client", "terminal" and "terminal device" used here can also be a communication terminal, an Internet terminal, and a music/video playing terminal, such as a PDA, MID (Mobile Internet Device) and/or A mobile phone with music/video playback function, or a smart TV, set-top box, etc.

The hardware referred to by names such as "server", "client" and "service node" in this application is essentially an electronic device with the equivalent capability of a personal computer, which is a central processing unit (including an arithmetic unit and a controller). ), memory, input device and output device and other necessary components disclosed by the Von Neumann principle, the computer program is stored in its memory, and the central processing unit transfers the program stored in the external memory into the memory to run, and executes the program. The instructions in the interface interact with input and output devices to complete specific functions.

It should be pointed out that the concept of "server" referred to in this application can also be extended to the case of server clusters in the same way. According to the principles of network deployment understood by those skilled in the art, the servers should be logically divided. In physical space, these servers can be independent from each other but can be called through interfaces, or can be integrated into a physical server. A computer or a group of computers. Those skilled in the art should understand this modification, but should not limit the implementation of the network deployment manner of the present application.

Unless explicitly specified, one or more technical features of the present application can be deployed on the server and remotely invoked by the client to obtain the online service interface provided by the server to implement access, or can be directly deployed and run on the client to implement access.

The neural network model cited or possibly cited in this application, unless specified in plain text, can either be deployed on a remote server and invoked remotely on the client, or deployed on a client with competent device capabilities to directly invoke, in some embodiments , when it runs on the client, its corresponding intelligence can be obtained through transfer learning, so as to reduce the requirements on the client hardware running resources and avoid excessively occupying the client hardware running resources.

All kinds of data involved in this application, unless specified in plain text, can be stored in a server remotely or in a local terminal device, as long as it is suitable for being called by the technical solution of this application.

Those skilled in the art should know that: although the various methods of the present application are described based on the same concept to show commonality with each other, unless otherwise specified, these methods can be independently executed. Similarly, for the various embodiments disclosed in this application, they are all proposed based on the same inventive concept. Therefore, the concepts expressed in the same way, and the concepts that are appropriately transformed for convenience even though the concept expressions are different, should be regarded as equivalent. understand.

In the various embodiments to be disclosed in this application, unless the mutually exclusive relationship between each other is clearly indicated, the related technical features involved in the various embodiments can be cross-combined to flexibly construct new embodiments, as long as the combination does not deviate from the present invention. The creative spirit of the application can meet the needs in the prior art or solve a certain aspect of the deficiencies in the prior art. Variations on this will be known to those skilled in the art.

A target commodity identification method of the present application can be programmed as a computer program product and implemented by being deployed in a server. The interface opened after the computer program product runs, and the method is executed by man-machine interaction with the process of the computer program product through a graphical user interface.

This application adopts the target recognition model implemented in this application to realize the function of target commodity recognition. The target recognition model is an integrated model, which includes a neural network model for extracting deep semantic information corresponding to commodity pictures, and a neural network model for extracting commodity titles corresponding to A neural network model of deep semantic information, a neural network model for identifying target commodities in commodity images.

Referring to FIG. 1, the target commodity identification method of the present application, in its typical embodiment, includes the following steps:

A target commodity identification method provided in accordance with one of the purposes of this application includes the following steps:

Step S1100, obtaining the product title and product picture in the product information of the target product;

In the application scenario of the e-commerce platform, each commodity can be processed as a relatively independent single information unit, and the merchant users of the online store of the e-commerce platform are responsible for publishing, maintaining and updating, and can be provided to consumer users for Browse and place orders, etc. The online store can be an independent site, which independently maintains a commodity database of commodities in its own online store, and can identify the target commodity in the commodity picture by installing the computer program product realized by the present application. Each product has its corresponding product information for describing the product, and the product information usually includes a product title and a product image.

The target commodity is an on-shelf commodity, and the merchant user of the online store of the e-commerce platform is responsible for the release and sale of the target commodity on the shelf.

The product picture is usually used to display the target product, and the picture contains the target product, or also includes other matching items that match the target product in order to bring out the effect of the target product. One can be a model wearing a skirt and matching shoes, clothes, jewelry, and coats to show the effect of the skirt; when the target product is a rack, one of its product images can be the rack and matching with the rack Books, home appliances, and ornaments to show the effect of the shelf. That is, the above-mentioned product image may contain other content than the target product, and in these content, other products than the current target product may also appear.

The commodity title is stored in association with the target commodity and starts from commodity description information provided in the form of text. In terms of use, the product title generally expresses any specific information that accurately describes the name, brand, material, function, use, selling point, etc. of the target product in a concise language;

In one embodiment, the corresponding commodity information can be obtained from the commodity database of the online store according to the unique identification code of the target commodity, and the unique identification code is set by the software engineer to distinguish each target commodity of the e-commerce platform. The unique identifier of the target product is convenient for storing and calling the product information of the target product.

When a user on the online store needs to list a target product, enter the product information corresponding to the target product on the corresponding product release page of the e-commerce platform, and then submit it to the backend server to associate the corresponding product information with the target product. The corresponding unique identification code is stored in the commodity database.

Step S1200, extracting the deep semantic information of the product picture and the product title;

A variety of image feature extraction models that have been pretrained to convergence can be used to extract image features from the product pictures, and deep semantic information corresponding to the visual features representing the target product and other corresponding supporting items in the product picture can be extracted. The image feature extraction model generally includes a CNN-based neural network model suitable for extracting deep semantic features of images, such as Resnet, EfficientNet, etc., which can be flexibly selected by those skilled in the art.

A variety of pre-trained to convergent text feature extraction models can be used to extract text features from the product titles, and to extract deep semantic information corresponding to the text features in the product titles that characterize the target product. The text feature extraction model generally includes an RNN-based neural network model suitable for deep semantic feature extraction of text, such as Bert, LSTM, Electra, etc., which can be flexibly selected by those skilled in the art.

Step S1300: Integrate the deep semantic information of the product title into the deep semantic information of the product image, so as to highlight the target product in the deep semantic information of the product image according to the deep semantic information of the product title. Image features to obtain image-text fusion feature information;

In an embodiment, in the attention layer 200 shown in FIG. 2, the feature interaction corresponding to the self-attention mechanism is performed on the deep semantic information of the product title and the deep semantic information of the product picture, so that the product The deep semantic information of the title and the deep semantic information of the product image realize deep interaction at the feature level, so as to realize the deep fusion of the product image and the product title at the deep semantic level, and obtain the preliminary image-text fusion feature information output by the attention layer, which is understandable , due to the implementation of deep fusion, the deep semantic information of the product title is fused into the deep semantic information of the product image, and the product title has an indicative effect on the target product in terms of text semantics. Therefore, the preliminary image-text fusion feature information has been referenced The text semantics of the product title significantly characterizes the feature of the image of the target product. Subsequent embodiments will further disclose the feature interaction corresponding to this self-attention mechanism, which is not shown here for the time being. Further, please refer to FIG. 3, combining the preliminary fusion feature information and the deep semantic information 300 of the commodity image to obtain the image-text fusion feature information, the combination can be in the form of matrix addition, which is not difficult to understand, because the preliminary fusion In the feature information, the feature of the image of the target product is significantly represented, so that the image-text fusion feature can be used to identify the target product.

Step S1400: Input the image-text fusion feature information into a target detection model that has been pre-trained to convergence, and identify the target product.

Detecting the target commodity in the commodity picture to identify the target commodity may be implemented by using a target detection model pre-trained to a convergent state. The target detection model is generally implemented by a deep learning-based model, such as the RCNN series, the Yolo series, and the SSD (Single Shot MultiBox Detector, one-step multi-box detector) series. The RCNN series is a representative algorithm based on region detection, YOLO is a representative algorithm based on region extraction, and SSD is an improved algorithm based on the first two series.

The RCNN series usually includes different specific models such as R-CNN, SPPNet, FastR-CNN, FasterR-CNN, and the Yolo series also has multiple versions that can be used. Such target detection models are suitable for identifying target image areas from a given picture, so that corresponding target images can be obtained according to the target image areas.

In one embodiment, Yolo-v5 can be used as the target detection model, connected to the classifier, and fine-tuned and trained by using a sufficient number of training samples. The training samples are commodity pictures, including the target commodity and its supporting For other matching items, each training sample provides corresponding training labels for each target product in it, so as to supervise the training of the model and enable it to acquire the ability to accurately identify the area corresponding to the target product from the given product image.

Therefore, the image-text fusion feature information is input into the target detection model that has been pre-trained to convergence, and the model is based on the image-text fusion feature information. Since the image-text fusion feature information has already fused the semantic information corresponding to the product title, therefore, Under the action of the semantic information, the image area corresponding to the target commodity can be detected from the commodity picture, and the coordinate information of the target commodity in the commodity picture can be output. Further, according to the coordinate information output by the target detection model, the image of the target product corresponding to the coordinate information of the target product is correspondingly cut out from the product image, thereby realizing the identification of the target product from the product image.

According to the typical embodiments disclosed in this application, it can be seen that this application has many advantages, including at least:

First, the present application uses the deep semantic information of the product title to highlight the image features of the target product in the deep semantic information of the product image, and provides key information indicating the target product in the product image, so that a single object can be identified based on the key information. Identify, quickly and accurately identify the target product from the product image;

Secondly, multi-modal feature fusion is used to fuse the deep semantic information of the product title and the deep semantic information of the product image, and then the target product can be identified according to the fusion features, which can simplify the model architecture and training steps used to realize the recognition, that is, The related models can be trained together by using a single data set, the training cost is low, and the training process is relatively simple. Moreover, it can be understood that the target commodity recognition realized by this application can be recognized in a single stage, and its execution efficiency is relatively high.

In addition, the target commodity identification function realized by the present application can be applied to various downstream tasks related to the e-commerce platform, such as commodity similarity matching, commodity classification, commodity labeling, etc., and it can be understood that since the target commodity can be extracted in a targeted manner , which helps provide exactly the images needed for downstream tasks.

Referring to FIG. 4, in a further embodiment, step S1200, the step of extracting the deep semantic information of the product picture and the product title, includes the following steps:

Step S1210, preprocessing the product image, and inputting the preprocessed product image into an image feature extraction model trained in advance to converge to obtain corresponding deep semantic information, which is used to characterize the image features of the product image;

In order to facilitate the follow-up model to extract the image features corresponding to the product images, the product images are preprocessed, and the length and width of the product images are enlarged in the same proportion. The ratio can be flexibly modified by those skilled in the art according to prior knowledge or experimental data. set up.

In an embodiment, the image feature extraction model is Resnet50, and the preprocessed product images are input into the Resnet50 that has been pre-trained to converge, and the Resnet50 stem block and 4 residual blocks (bottleneck blocks) step by step. Extract the image features corresponding to the product pictures, in which the shallow stage (stage) extracts basic features such as details and edges corresponding to the target product and other matching items in the product picture, and then extracts deep semantic features and advanced logic features in the deep stage. , and finally obtain the deep semantic information output by the final stage, that is, the Res5 stage, as shown by 300 in Figure 3.

Step S1220: Preprocess the product title, input the preprocessed product title into a text feature extraction model that has been pre-trained to convergence, and obtain corresponding deep semantic information for characterizing the text feature of the product title.

Generally speaking, the text format in the product title is complicated and may contain line breaks, redundant punctuation marks, redundant blank characters, etc. These characters do not have much impact on the semantics of the product title itself, but will interfere with the accuracy of subsequent semantic extraction. Therefore, in order to improve the accuracy of the deep semantic information extracted by the subsequent model, the product title can be subjected to format preprocessing. For example, the preprocessing may include: replacing line breaks with space symbols; replacing two or more blank characters Replace strings to keep only one whitespace character; replace 2 or more punctuation marks to keep only one, etc. The format preprocessing method is adopted as needed, and those skilled in the art can flexibly implement it according to actual business conditions.

In one embodiment, the text feature extraction model is Bert, and the preprocessed product title is input into the Bert that has been pre-trained to converge, and the text features corresponding to the text representing the product category of the target product in the product title are extracted, such as a product. The title is "Handmade Beaded Corsage Suit Elegant Ladies Skirt", in which the text representing the commodity category of the target commodity is "Ladies Skirt", and corresponding deep semantic information is obtained.

In this embodiment, through the image feature extraction model and the text feature extraction model pre-trained to the convergence, intelligent, fast and accurate corresponding extraction of deep semantic information corresponding to the product picture and the product title is realized.

Referring to FIG. 5, in the further embodiment, step S1220, the step of preprocessing the product title, includes the following steps:

Step S1221, filtering invalid characters in the product title;

Generally speaking, the product title usually contains the text of the product category of the target product, as well as the effect, function, texture, material and other modification text of the target product. The text is an invalid character. Therefore, the invalid character in the product title can be filtered, so that the subsequent model can extract the corresponding text features. In one embodiment, the text corresponding to the modified text can be pre-collected manually or artificially and aggregated into an invalid character dictionary, and then the text corresponding to the product title and the text corresponding to the modified text in the invalid character dictionary can be accurately calculated. Matching and/or fuzzy matching, and then determine invalid characters in the product title according to the matching result, and delete them to filter the product title.

Step S1222: Perform word segmentation on the filtered product title to obtain keywords therein, where the keywords include product words and/or brand words of the target product, and complete the preprocessing of the product title.

In one embodiment, the tokenizers BasicTokenizer and WordpieceTokenizer are used to segment the filtered product titles, first obtain a list of tokens with relatively coarse scores through the BasicTokenizer, and then perform WordpieceTokenizer on each token to obtain the keywords in it. , to complete the preprocessing of the product title.

In this embodiment, by filtering invalid characters in the product title, when the subsequent model extracts corresponding text features, less text needs to be processed and less interference, so that the execution efficiency and accuracy of the model can be improved.

Referring to FIG. 6, in a further embodiment, in step S1300, the deep semantic information of the product title is fused to the deep semantic information of the product image, so as to highlight the target product according to the deep semantic information of the product title In the step of obtaining image-text fusion feature information from image features in the deep semantic information of the product image, the following steps are included:

Step S1310, using a multi-modal feature interaction fusion module to fuse the deep semantic information of the product title and the deep semantic information of the product image to obtain preliminary fusion feature information, in which the preliminary fusion feature information significantly characterizes the target commodity. characteristics of the image;

The multi-modal feature interaction fusion module is shown in Figure 2. Specifically, in the attention layer 200, the same two convolution layers are used to extract the corresponding two identical feature information for the deep semantic information of the product title. , and use two other identical convolution layers to extract the corresponding two identical feature information from the deep semantic information of the product image. Further, the feature interaction corresponding to the self-attention mechanism is performed on one of the feature information corresponding to the extracted deep semantic information of the product image and two identical feature information corresponding to the extracted deep semantic information of the product title, Make the deep semantic information of the product title and the deep semantic information of the product image realize deep interaction at the feature level, so as to realize the deep fusion of the product image and the product title at the deep semantic level, and obtain the preliminary fusion image and text output by the attention layer. Fusion of feature information, it can be understood that due to the implementation of deep fusion, the deep semantic information of the product title is fused into the deep semantic information of the product image, and the product title has an indicative effect on the target product in terms of text semantics. In the fusion feature information, the feature of the image of the target product has been significantly characterized with reference to the textual semantics of the product title. Subsequent embodiments will further disclose the feature interaction corresponding to the self-attention mechanism, which is not shown here for the time being.

In an optional embodiment, please refer to FIG. 2, further, a layer of convolution layer can be used to extract the feature information 201 corresponding to the preliminary image-text fusion feature information, and compare it with the extracted deep semantics of the product image. Another feature information 202 corresponding to the information is subjected to matrix dot multiplication. It is not difficult to understand that after the matrix dot multiplication, the features of the image representing the target product in the product picture in the preliminary image-text fusion feature information are further characterized to obtain features. Explicit preliminary image-text fusion feature information. The matrix dot product is two matrices of the same dimension, and the feature data in the two are multiplied bit by bit, that is, the feature data of the first row and the first column in the matrix corresponding to the feature information 201 and the matrix corresponding to the feature information 202 The feature data of the first row and the first column in the matrix are multiplied, and the feature data of the first row and the second column of the matrix corresponding to the feature information 201 is the same as the feature data of the first row and the second column of the matrix corresponding to the feature information 202. Multiplication, the feature data of the second row and the first column in the matrix corresponding to the feature information 201 is multiplied by the feature data of the second row and the first column in the matrix corresponding to the feature information 202 , and so on.

Step S1320, combining the preliminary fusion feature information with the deep semantic information of the commodity image to obtain image-text fusion feature information;

Combining the preliminary fusion feature information and the deep semantic information of the product image as shown in 300 in Figure 3, the graphic and text fusion feature information is obtained. The combination can be in the form of matrix addition, which is not difficult to understand. Since the preliminary fusion features In the information, the features of the image of the target product are significantly characterized, so that the image-text fusion feature can be used to identify the image of the target product.

In this embodiment, through the feature interaction corresponding to the self-attention mechanism, the feature of the image corresponding to the target product in the deep semantic information of the product image is displayed, so that the image of the target product in the product image is represented in the image-text fusion feature information. The features are highlighted and become salient features, which helps to improve the recognition accuracy of the target product in the product image by the subsequent model.

Referring to FIG. 7, in a preferred embodiment, in step S1310, a multi-modal feature interactive fusion module is used to fuse the deep semantic information of the product title and the deep semantic information of the product image, and in the step of obtaining preliminary fusion feature information , including the following steps:

Step S1311, constructing a query vector with the deep semantic information of the product image, constructing a key vector and a value vector with the deep semantic information of the product title, and inputting to the attention layer;

Referring to FIG. 2, in the attention layer (Attention) 200, the deep semantic information of the product image and the deep semantic information of the product title are used as the input of the attention layer, and the corresponding convolution layer, namely the weight matrix W ^Q is used to extract The corresponding query vector ( ^Query ) is obtained from the deep ^semantic information of the product image, and the corresponding key vector (Key) and value vector ( Value). The weight matrices W ^Q , W ^K , and W ^V are all learnable weights.

Step S1312, interact and normalize the query vector and the key vector by the attention layer to obtain a weight matrix;

Continuing to refer to FIG. 2, in the attention layer 200, a matrix multiplication operation is performed on the transposed matrix corresponding to the query vector and the key vector to obtain a deep semantic information that realizes the product image and the product title. The product matrix of the feature interaction of the deep semantic information, the product matrix is the scale of HW*T, after using the Softmax function to activate and output, the obtained weight matrix is the deep semantic information of the product image and the product title. The semantic information after the deep interaction is essentially the result of highlighting the salient features in the deep semantic information of the product picture, that is, the feature of the image of the target product in the product picture, according to the deep semantic information of the product title.

Step S1313, matching the value vector to the weight matrix by the attention layer to obtain initial features;

Continuing to refer to FIG. 2, in the attention layer 200, the weight matrix whose scale is HW*T is activated and output by the Softmax function, and the transposed matrix corresponding to the value vector, that is, the scale is T* The text features of C _i are then multiplied by a matrix to obtain a product matrix with a scale of HW*C _i , which is the feature interaction corresponding to the self-attention mechanism between the deep semantic information of the product image and the deep semantic information of the product title. The obtained preliminary fusion feature information.

In this embodiment, the preliminary fusion feature information is based on the weight matrix obtained after the interaction between the deep semantic information of the product image and the deep semantic information of the product title, multiplied by the deep semantic information of the product title and correspondingly matched. The value vector of the corresponding weight W ^V is obtained, and the deep semantic information of the product image is deeply fused with the deep semantic information of the product title, so that the features of the image of the target product are significantly represented in the preliminary fusion feature information.

Referring to FIG. 8 , in a further embodiment, in step S1400 , the step of inputting the image-text fusion feature information into a target detection model that has been pre-trained to convergence, and identifying the target commodity, includes the following steps:

Step S1410, using a pre-trained target detection model to converge to detect the target commodity in the commodity picture according to the image-text fusion feature information, and obtain a corresponding detection area;

In one embodiment, the target detection model is MaskRCNN, the input value of the image-text fusion feature information is pre-trained into the converged MaskRCNN, and the detection area corresponding to the image of the target product is detected from the product image.

Step S1420: Obtain a rectangular frame with a minimum area surrounding the detection area, and select a target commodity from the frame as a recognition result.

Obtain the rectangular frame with the smallest area surrounding the detection area, so that the rectangular frame completely contains the image of the target product in the detection area in a rectangle, and the area corresponding to the area containing the image of the non-target product is the smallest, and the minimum area is obtained. The rectangular frame of the area and its corresponding position information in the product picture, usually the position information is the coordinates of the four vertices corresponding to the rectangular frame. Further, the image of the target product in the product picture is selected as the recognition result according to the rectangular frame.

In this embodiment, a rectangular frame with a minimum area surrounding the detection area is obtained, and a target commodity is selected by the frame as the recognition result, so as to improve the accuracy of the recognition result.

Referring to FIG. 9 , in an extended embodiment, step S1400 inputs the image-text fusion feature information into a pre-trained target detection model that has converged, and after the step of recognizing the target product, it further includes the following steps:

Step S1500, according to the rectangular frame of the selected target commodity, cut out the image of the target commodity from the commodity picture, and store the unique identification code associated with the target commodity in the commodity database;

The image of the target product is cut out from the product image according to the position information in the product picture corresponding to the rectangular box of the target product, and the image of the target product is stored with the unique identification code of the target product with the product image. database for subsequent calls.

Step S1600: In response to the commodity recommendation request, search the commodity database to obtain the image of the target commodity according to the unique identification code of the target commodity, and match the recommended commodity similar to it;

It can be understood that some e-commerce pages in the e-commerce platform need to load recommended products, which triggers the generation of a product recommendation request and pushes it to the server of the e-commerce platform. The server receives the request and responds to it, and retrieves the product database according to the unique identification code of the target product. Obtain the image of the target product, and perform image similarity matching with the image of the target product in the product database corresponding to the product belonging to the same product category as the target product, and match the product whose similarity exceeds the threshold as the recommended product. Persons skilled in the art set according to business needs. The commodity category is generally set by the e-commerce platform. The merchant user of the online store usually needs to select the commodity category corresponding to the published commodity when publishing the commodity. Therefore, in the e-commerce platform. Each product has its corresponding product category.

Step S1700: Respond to the commodity recommendation request, and push the recommended commodity.

Further, in response to the product recommendation request, the recommended product is pushed to a corresponding e-commerce page, and the e-commerce page receives the recommended product and loads and displays it.

In this embodiment, since the images of the products in the e-commerce platform are all cut out images of the corresponding target products, the accuracy of the product recommendation based on this can be guaranteed, and the accurate recommendation can be achieved.

Please refer to FIG. 10 , a target commodity identification device provided for one of the purposes of the present application is a functional embodiment of the target commodity identification method of the present application. The device includes: a graphic and text acquisition module 1100, a semantic extraction module 1200, The feature fusion module 1300 and the target recognition module 1400, wherein, the graphic acquisition module 1100 is used to acquire the commodity title and commodity picture in the commodity information of the target commodity; the semantic extraction module 1200 is used to extract the commodity picture and commodity title. deep semantic information; the feature fusion module 1300 is used to fuse the deep semantic information of the product title into the deep semantic information of the product picture, so as to highlight the target product in the product according to the deep semantic information of the product title The image features in the deep semantic information of the product images are used to obtain image-text fusion feature information; the target recognition module 1400 is used to input the image-text fusion feature information into a pre-trained target detection model that has converged to identify the target product. .

In a further embodiment, the semantic extraction module 1200 includes: an image feature extraction sub-module, configured to preprocess the product image, input the preprocessed product image into the image feature extraction model that has been pre-trained to convergence, and obtain Corresponding deep semantic information is used to characterize the image features of the product picture; a text feature extraction sub-module is used to preprocess the product title, and input the preprocessed product title into the text feature extraction model that has been pre-trained to convergence , to obtain the corresponding deep semantic information, which is used to characterize the text features of the product title.

In a further embodiment, the image feature extraction sub-module includes: a character filtering unit for filtering invalid characters in the commodity title; a text segmentation unit for performing word segmentation on the filtered commodity title, and obtaining the Keywords, where the keywords include product words and/or brand words of the target commodity, to complete the preprocessing of the commodity title.

In a further embodiment, the feature fusion module 1300 includes: a semantic fusion sub-module, configured to use a multi-modal feature interaction fusion module to fuse the deep semantic information of the product title and the deep semantic information of the product image to obtain Preliminary fusion feature information, in which the features of the image of the target commodity are significantly represented in the preliminary fusion characteristic information; an information combination sub-module is used to combine the preliminary fusion characteristic information with the deep semantic information of the commodity image to obtain a graph text fusion feature information;

In a preferred embodiment, the semantic fusion sub-module includes: a vector input unit for constructing a query vector with the deep semantic information of the product image, and constructing a key vector and a value vector with the deep semantic information of the product title. , input attention layer; weight extraction unit, used for interacting and normalizing the query vector and the key vector by the attention layer to obtain a weight matrix; feature generation unit, used by the attention layer The layer matches the value vector to the weight matrix to obtain preliminary fusion feature information.

In a further embodiment, the target identification module 1400 includes: a target detection unit, configured to detect the target commodity in the commodity picture according to the image-text fusion feature information by using a pre-trained target detection model to converge, and obtain the corresponding target commodity. the detection area; the frame selection recognition unit is used to obtain a rectangular frame with the smallest area surrounding the detection area, and select the target product by its frame as the recognition result.

In an extended embodiment, after the target identification module 1400, it further includes: an interception storage module, configured to intercept the image of the target commodity from the commodity picture according to the rectangular frame for selecting the target commodity, and associate it with the image of the target commodity. The unique identification code of the target commodity is stored in the commodity database; the response request module is used to respond to the commodity recommendation request, retrieve the commodity database according to the unique identification code of the target commodity to obtain the image of the target commodity, and match the recommended commodity similar to it; In response to the product recommendation request, push the recommended product.

To solve the above technical problems, the embodiments of the present application also provide computer equipment. As shown in Figure 11, a schematic diagram of the internal structure of the computer equipment. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected by a system bus. Wherein, the computer-readable storage medium of the computer device stores an operating system, a database and computer-readable instructions, and the database may store a sequence of control information. When the computer-readable instructions are executed by the processor, the processor can be made to implement a Target product identification method. The processor of the computer device is used to provide computing and control capabilities and support the operation of the entire computer device. Computer-readable instructions may be stored in the memory of the computer device, and when executed by the processor, the computer-readable instructions may cause the processor to execute the target commodity identification method of the present application. The network interface of the computer equipment is used for communication with the terminal connection. Those skilled in the art can understand that the structure shown in FIG. 11 is only a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer equipment to which the solution of the present application is applied. Include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In this embodiment, the processor is used to execute the specific functions of each module and its sub-modules in FIG. 10 , and the memory stores program codes and various types of data required to execute the above-mentioned modules or sub-modules. The network interface is used for data transmission between user terminals or servers. The memory in this embodiment stores program codes and data required to execute all modules/sub-modules in the target commodity identification device of the present application, and the server can call the server's program codes and data to execute the functions of all sub-modules.

The present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors can execute the target commodity identification method of any embodiment of the present application. step.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above-mentioned embodiments of the present application can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the program is executed, it may include the flow of the embodiments of the above-mentioned methods. The aforementioned storage medium may be a computer-readable storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM).

Those skilled in the art can understand that various operations, methods, steps, measures, and solutions in the process discussed in this application may be alternated, modified, combined or deleted. Further, other steps, measures, and solutions in the various operations, methods, and processes that have been discussed in this application may also be alternated, modified, rearranged, decomposed, combined, or deleted. Further, steps, measures and solutions in the prior art with various operations, methods, and processes disclosed in this application may also be alternated, modified, rearranged, decomposed, combined or deleted.

The above are only part of the embodiments of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (10)

1. a target commodity identification method, is characterized in that, comprises the steps: Obtain the product title and product image in the product information of the target product; Extract the deep semantic information of the product image and product title; Integrating the deep semantic information of the product title with the deep semantic information of the product image, so as to highlight the image features of the target product in the deep semantic information of the product image according to the deep semantic information of the product title, Obtain image-text fusion feature information; The image-text fusion feature information is input into the target detection model pre-trained to convergence, and the target product is identified. 2. The target commodity identification method according to claim 1, wherein the step of extracting the deep semantic information of the commodity picture and the commodity title comprises the following steps: Preprocessing the product picture, and inputting the preprocessed product picture into an image feature extraction model trained in advance to converge to obtain corresponding deep semantic information, which is used to characterize the image features of the product picture; The product title is preprocessed, and the preprocessed product title is input into a text feature extraction model that has been pre-trained to convergence to obtain corresponding deep semantic information, which is used to characterize the text feature of the product title. 3. target commodity identification method according to claim 2, is characterized in that, in the step of preprocessing described commodity title, comprises the following steps: filter invalid characters in the title of the product; Perform word segmentation on the filtered product title to obtain keywords therein, where the keywords include product words and/or brand words of the target product, and complete the preprocessing of the product title. 4 . The target commodity identification method according to claim 1 , wherein the deep semantic information of the commodity title is fused into the deep semantic information of the commodity picture, so as to be highlighted according to the deep semantic information of the commodity title. 5 . The image features of the target product in the deep semantic information of the product picture, and the step of obtaining image-text fusion feature information includes the following steps: A multimodal feature interaction fusion module is used to fuse the deep semantic information of the product title and the deep semantic information of the product image to obtain preliminary fusion feature information, in which the features of the image of the target product are significantly represented in the preliminary fusion feature information. ; combining the preliminary fusion feature information with the deep semantic information of the commodity image to obtain image-text fusion feature information; 5 . The target commodity identification method according to claim 3 , wherein a multimodal feature interactive fusion module is used to fuse the deep semantic information of the commodity title and the deep semantic information of the commodity picture to obtain preliminary fusion feature information. 6 . The steps include the following steps: Construct a query vector with the deep semantic information of the commodity picture, construct a key vector and a value vector with the deep semantic information of the commodity title, and input the attention layer; Interacting and normalizing the query vector and the key vector by the attention layer to obtain a weight matrix; Preliminary fusion feature information is obtained by matching the value vector with the weight matrix by the attention layer. 6. The target commodity identification method according to claim 1, wherein the image-text fusion feature information is input into a target detection model trained in advance to converge, and the step of recognizing the target commodity comprises the following steps : Use a pre-trained target detection model to converge to detect the target product in the product image according to the image-text fusion feature information, and obtain a corresponding detection area; A rectangular frame with the smallest area surrounding the detection area is obtained, and the target commodity is selected from the frame as the recognition result. 7. The target product identification method according to claim 1, wherein the image-text fusion feature information is input into a target detection model trained in advance to converge, and after the step of recognizing the target product, the method further comprises the following steps: step: Cut out the image of the target product from the product picture according to the rectangular frame of the selected target product, and store the unique identification code associated with the target product in the product database; In response to the product recommendation request, search the product database according to the unique identification code of the target product to obtain the image of the target product, and match the recommended products similar to it; Respond to the product recommendation request, and push the recommended product. 8. A target commodity identification device, characterized in that, comprising: The graphic and text acquisition module is used to acquire the product title and product picture in the product information of the target product; A semantic extraction module, used for extracting the deep semantic information of the commodity picture and commodity title; A feature fusion module is used to fuse the deep semantic information of the product title to the deep semantic information of the product image, so as to highlight the deep semantic information of the target product in the product image according to the deep semantic information of the product title Image features in the information to obtain image-text fusion feature information; The target recognition module is used for inputting the image-text fusion feature information into the target detection model pre-trained to convergence, and identifying the target product. 9. A computer device comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the computer program according to any one of claims 1 to 7 steps of the method. 10. A computer-readable storage medium, characterized in that it stores a computer program implemented by the method according to any one of claims 1 to 7 in the form of computer-readable instructions, and the computer program is called by a computer At runtime, the steps included in the corresponding method are executed.

Images