CN117474630A - Artificial intelligence platform based on E-commerce application - Google Patents

Artificial intelligence platform based on E-commerce application Download PDF

Info

Publication number
CN117474630A
CN117474630A CN202311495986.XA CN202311495986A CN117474630A CN 117474630 A CN117474630 A CN 117474630A CN 202311495986 A CN202311495986 A CN 202311495986A CN 117474630 A CN117474630 A CN 117474630A
Authority
CN
China
Prior art keywords
data
platform
model
user
algorithm
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202311495986.XA
Other languages
Chinese (zh)
Inventor
季恩卉
房鹏展
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Focus Technology Co Ltd
Original Assignee
Focus Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Focus Technology Co Ltd filed Critical Focus Technology Co Ltd
Priority to CN202311495986.XA priority Critical patent/CN117474630A/en
Publication of CN117474630A publication Critical patent/CN117474630A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9535Search customisation based on user profiles and personalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90Details of database functions independent of the retrieved data types
    • G06F16/95Retrieval from the web
    • G06F16/953Querying, e.g. by the use of web search engines
    • G06F16/9536Search customisation based on social or collaborative filtering
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q30/00Commerce
    • G06Q30/06Buying, selling or leasing transactions
    • G06Q30/0601Electronic shopping [e-shopping]
    • G06Q30/0631Item recommendations
    • YGENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02DCLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00Energy efficient computing, e.g. low power processors, power management or thermal management

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Business, Economics & Management (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Accounting & Taxation (AREA)
  • Finance (AREA)
  • Physics & Mathematics (AREA)
  • Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Strategic Management (AREA)
  • Marketing (AREA)
  • Development Economics (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

An artificial intelligence platform based on e-commerce application comprises the following modules: (1) AI platform base; (2) AI development platform; (3) AI service platform; (4) AI application platform. And bridging is carried out between the AI development platform and the AI service platform as well as between the AI development platform and the application through a network. The AI platform base includes underlying hardware, data storage, an algorithm framework, and a resource management module. The basic hardware module comprises hardware supported by a platform, the data storage module comprises data storage, the algorithm framework module represents an algorithm framework supported by a development platform, and the resource management module comprises a kube container arrangement engine and a yarn resource management system. The AI development platform comprises a data center, an algorithm center, a calculation force center, a training center and a model center. The AI service platform comprises a CV middle platform, an NLP middle platform, a recommendation middle platform and a knowledge graph. The AI application platform comprises a foreign trade robot, an intelligent sales service module and an intelligent recommendation service module.

Description

Artificial intelligence platform based on E-commerce application
Technical Field
The invention relates to the technical field of machine learning and computer service systems and the field of cross-border electronic commerce, in particular to an artificial intelligent platform based on electronic commerce application.
Background
Deep Learning (DL) is a new research direction in the field of Machine Learning (ML), which was introduced to Machine Learning to bring it closer to the original goal-artificial intelligence (AI, artificial Intelligence).
CN202111361216.7 discloses a multi-user storage interfacing method of an AI platform and the AI platform, wherein the multi-user storage interfacing method of the AI platform is used for the AI platform, and the AI platform comprises an identification storage block; the multi-user storage docking method comprises the following steps: when a switching instruction of switching the user storage system by the AI platform is obtained, judging whether the user storage system to be switched is available or not by using the identification storage block; if the user storage system to be switched is judged to be available, backing up the data information of the current user storage system to a database of the AI platform; synchronizing data information of a user storage system to be switched to a database of an AI platform; and when the data information is synchronized, feeding back a synchronization result to a service layer of the AI platform. The technical scheme of the invention can solve the problems that the AI platform is incompatible with different user storage systems in the prior art, so that the user storage systems which are in butt joint need to be determined in advance, and the working and using efficiency of the AI platform are reduced.
Application number: 202010988313.8 discloses a massive file retrieval method, device and equipment based on an AI training platform, wherein the method comprises the following steps: the AI training platform acquires a retrieval task issued by a user; the AI training platform generates a retrieval thread flow according to the retrieval task, and controls the service logic of the retrieval process according to the retrieval thread flow; the AI training platform sequentially encodes files in the database by taking folders as units to generate ordered queue folders, extracts search keywords from the search task, and then performs keyword search on each ordered queue folder in a binary search and depth-first traversal combined mode. The invention provides a service logic in the searching process by utilizing the searching thread flow control, which prevents the AI training platform from occupying the CPU of the server resource for a long time, and simultaneously improves the searching efficiency by combining the depth-first traversal and the binary search mode, avoids the defect of long searching time after the file is searched by using the depth-first traversal singly, and shortens the training time of the AI training platform.
Deep learning is the inherent regularity and presentation hierarchy of learning sample data, and the information obtained during such learning is helpful in interpreting data such as text, images and sounds. Its final goal is to have the machine have analytical learning capabilities like a person, and to recognize text, image, and sound data. Deep learning is a complex machine learning algorithm that achieves far greater results in terms of speech and image recognition than prior art.
Deep learning has achieved many results in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech, recommendation and personalization techniques, and other related fields. The deep learning makes the machine imitate the activities of human beings such as audio-visual and thinking, solves a plurality of complex pattern recognition problems, and makes the related technology of artificial intelligence greatly advanced.
Along with the continuous development of informatization technology and the continuous breakthrough of artificial intelligence technology, the cross-border electronic commerce field also has a great breakthrough in business, such as a plurality of fields of foreign trade robots, intelligent marketing, intelligent operation, intelligent recommendation and the like. However, algorithm engineers often deploy the development algorithm on a bare metal server, the efficiency is low, and a personal technology cocoon house exists, so that the business effect is difficult to reach an ideal state, and therefore a development platform with a complete algorithm and a development platform easy to use are necessary for the algorithm engineers.
Disclosure of Invention
The invention aims to solve the problems of normalizing the development flow of algorithm engineers in the field of cross-border electronic commerce and improving the development efficiency, and provides an artificial intelligent platform based on electronic commerce application. And can replace the development mode of the traditional bare metal server, such as using pyrarm, vscore, vim programming, etc.
In order to solve the problems, the invention provides an artificial intelligence platform based on e-commerce application, which comprises the following modules: (1) AI platform base; (2) AI development platform; (3) AI service platform; (4) AI application platform. And bridging is carried out between the AI development platform and the AI service platform as well as between the AI development platform and the application through a network.
The AI platform base includes underlying hardware, data storage, an algorithm framework, and a resource management module. The modules comprise the following components in detail:
(1) The basic hardware of the AI platform base supports CPU, GPU, FPGA, NPU and other hardware facilities;
(2) The data storage mode of the AI platform base supports fast storage OSS comprising FFS and Ceph storage, vector storage vearch, big data storage mode HDFS, hive, hudi and the like;
(3) The algorithm framework of the AI platform base supports pytorch, tensorflow, sklearn, keras, paddlepaddle, caffe, autoML, sparkML and other common algorithm libraries;
(4) The resource management system of the AI platform base includes a kubespher container orchestration engine and a yarn resource management system.
The AI development platform comprises five modules, namely a data center, an algorithm center, a calculation center, a training center and a model center. The modules comprise the following components in detail:
(1) The data center module of the AI development platform comprises data processing functional modules such as data management, data interaction, data labeling and the like, and simultaneously supports complete data types of sample data, characteristic data and vector data;
(2) The algorithm center module of the AI development platform comprises functional modules such as algorithm management, algorithm evaluation and the like, and comprises common algorithms such as CV algorithm library, NLP algorithm library, recommended algorithm library, voice algorithm library, reinforcement learning algorithm library and the like;
(3) The power center module of the AI development platform comprises management modules such as user management, resource management, task management and the like, and simultaneously manages an A100 cluster, a V100 cluster, a 4090 cluster and the like;
(4) The training center module of the AI development platform comprises four functional modules of data preparation, model training, model verification and model iteration;
(5) The model center module of the AI development platform comprises two modules of model management and reasoning optimization. .
The AI service platform comprises a CV middle platform, an NLP middle platform, a recommendation middle platform, a knowledge graph, an AI portal and the like.
The AI application platform comprises two large modules, namely a platform intelligent module and a service intelligent module, wherein each module comprises the following components in detail:
(1) The platform intelligence comprises a machine learning algorithm, a deep learning algorithm, CV tasks, NLP tasks, recommended tasks and the like;
(2) The business intelligence comprises business modules such as foreign trade robots, intelligent sales, intelligent marketing, intelligent operation, intelligent recommendation, intelligent operation and maintenance.
The artificial intelligent platform based on the e-commerce application is a high-performance, easily-extensible, agile and efficient artificial intelligent development platform and resource management platform oriented to cross-border e-commerce industry-level artificial intelligent development and training scenes, provides unified management of large-scale heterogeneous computing clusters, convenient and easy-to-use AI development environments, large-scale AI distributed training, full life cycle management of training operation, version management and tracing of data sets, cluster resource statistics and report forms and other capabilities, helps users to achieve accurate resource management and scheduling, agile data integration and acceleration, and flow AI scene and service integration, effectively breaks through elements such as AI development environments, computing power and data, manages full-period AI workflows, and improves AI development efficiency of users.
The beneficial effects are that: based on an artificial intelligent platform applied by an e-commerce, the system supports rich machine learning algorithms, one-stop machine learning experience, mainstream machine learning frames and visual modeling modes, such as the functional characteristics of PAI. The algorithm of the artificial intelligent platform based on the e-commerce application is deposited by the cross-border e-commerce business level large-scale service, so that not only basic clustering and regression algorithms are supported, but also complex algorithms such as analysis and feature processing are supported. The system has one-stop machine learning experience, and an artificial intelligent platform based on the e-commerce application supports the whole process from data uploading, data preprocessing, feature engineering, model training, model evaluation and model release. Meanwhile, a mainstream deep learning framework is supported, and an artificial intelligence platform based on the e-commerce application supports a TensorFlow, caffe mainstream machine learning framework, an MXNet mainstream machine learning framework and the like. The modeling method of visualization is possessed, and the artificial intelligence platform based on the e-commerce application encapsulates a classical machine learning algorithm, so that the modeling method has the following advantages: supporting to build machine learning experiments in a dragging mode. The built-in AutoML is supported to carry out parameter tuning, and model parameter automatic exploration, model effect automatic evaluation, model automatic downward conduction and model automatic optimization are realized. The system has the function of one-key model deployment service, and an artificial intelligent platform based on the e-commerce application supports one-key release of a training model generated by PAI-Studio, DSW and Autolearning as a Restful API interface, so that seamless connection from the model to the service is realized.
Drawings
Fig. 1 is an exemplary architecture diagram of an AI platform of the present invention.
Fig. 2 is a functional flow diagram of the present invention.
Detailed Description
The core idea of the invention is that the following architecture is set from bottom to top according to fig. 1:
(1) The base of the AI platform is an infrastructure and a bottom layer system of the AI platform, and comprises a basic hardware module, a data storage module, an algorithm frame module and a resource management module;
(2) The AI development platform is applied to algorithm development and model development of algorithm engineers, and comprises a data center module, an algorithm center module, a calculation center module, a training center module and a model center module;
(3) The AI service platform is used for packing and uploading service functions for other people to call after an algorithm engineer finishes model development, and comprises a CV middle platform, an NLP middle platform, a recommended middle platform, a knowledge graph and an AI portal;
(4) The AI application platform is based on the service abstraction of the AI service platform, and comprises the following cross-border electric service: intelligent sales, intelligent marketing, intelligent operations, intelligent recommendations, intelligent operations and maintenance.
The invention is further described below with reference to the drawings and exemplary embodiments:
1. the AI platform base comprises basic hardware, data storage, an algorithm framework, a resource management module and a GPU pooling module. The modules comprise the following components in detail:
(1) The basic hardware of the AI platform base supports CPU, GPU, FPGA, NPU and other hardware facilities;
(2) The data storage mode of the AI platform base supports fast storage OSS comprising FFS and Ceph storage, vector storage vearch, big data storage mode HDFS, hive, hudi and the like;
(3) The algorithm framework of the AI platform base supports pytorch, tensorflow, sklearn, keras, paddlepaddle, caffe, autoML, sparkML and other common algorithm libraries;
(4) The resource management system of the AI platform base includes a kubespher container orchestration engine and a yarn resource management system.
(5) The AI platform base comprises a self-grinding GPU pooling module, and the GPU pooling function can comprehensively stage all GPU hardware resources, and provides a flexible and efficient GPU resource management mechanism, so that a plurality of tasks can share and compete with limited GPU resources, and the utilization rate of computing resources and the performance of the whole system are improved. Comprising the following steps: (A) The GPU pooling function is responsible for managing all GPU hardware resources on the platform. It tracks the status, availability and load conditions of each GPU and maintains a pool of available GPUs; (B) When a user submits a task or job, the GPU pooling function dynamically allocates an appropriate amount of GPU resources to the task according to the task's requirements and the current availability of GPU resources. Thus, resource waste and contention can be avoided, and each task can be ensured to obtain enough computing resources; (C) The GPU pooling function maximizes the utilization of GPU resources through flexible resource scheduling and task queuing. When one task is completed, the pooling function will re-bring the released GPU resources into the available pool and assign other tasks as needed to ensure full utilization of all GPU resources. (D) The GPU pooling functionality may support management of task priorities. The method can be used for preferentially distributing GPU resources to the high-priority tasks according to the priority setting of the tasks, and guaranteeing the execution efficiency and response time of the key tasks.
GPU pooling details are as follows: (1) data preparation: input data is copied from the host memory into the GPU device memory. May be implemented using CUDA memory allocation and transfer functions (cudaMalloc and cudamecpy). (2) Writing a kernel function: kernel functions are written to implement the pooling operation. The kernel function should include the following: the thread and block index of the CUDA is used to determine the location of each thread and block on the input data. The pooling results within each region are calculated. A max pooling or average pooling operation may be used. The pooled results are stored into output data. (3) Parallelization: the parallel mechanism of CUDA is used to process data for multiple regions. (4) Shared memory: shared memory is used in the kernel function to improve data access efficiency. The shared memory of the CUDA is used to store local copies of the input data to reduce access to global memory. (5) And (3) data transmission: the output data is copied from the GPU device memory back to the host memory. Using the memory transfer function of CUDA (cudamecpy).
The kernel function is a similar function. Providing a function of the machine learning algorithm. It accepts two inputs and calculates their degree of similarity. With gaussian (sigma) or SVM kernel functions, sigmoid kernel functions may also be used, which are suitable for the classification problem.
2. The AI development platform comprises five modules, namely a data center, an algorithm center, a calculation center, a training center and a model center. The modules comprise the following components in detail:
(1) The data center module of the AI development platform comprises data processing functional modules such as data management, data interaction, data labeling and the like, and simultaneously supports complete data types of sample data, characteristic data and vector data;
(2) The algorithm center module of the AI development platform comprises functional modules such as algorithm management, algorithm evaluation and the like, and comprises common algorithms such as CV algorithm library, NLP algorithm library, recommended algorithm library, voice algorithm library, reinforcement learning algorithm library and the like;
(3) The power center module of the AI development platform comprises management modules such as user management, resource management, task management and the like, and simultaneously manages an A100 cluster, a V100 cluster, a 4090 cluster and the like;
(4) The training center module of the AI development platform comprises four functional modules of data preparation, model training, model verification and model iteration;
(5) The model center module of the AI development platform comprises two modules of model management and reasoning optimization.
3. The AI service platform comprises a CV middle platform, an NLP middle platform, a recommendation middle platform, a knowledge graph, an AI portal and the like.
4. The AI application platform comprises two large modules, namely a platform intelligent module and a service intelligent module, wherein each module comprises the following components in detail:
(1) The platform intelligence comprises a machine learning algorithm, a deep learning algorithm, CV tasks, NLP tasks, recommended tasks and the like;
(2) The business intelligence comprises business modules such as foreign trade robots, intelligent sales, intelligent marketing, intelligent operation, intelligent recommendation, intelligent operation and maintenance.
(3) The application platform comprises a business workflow, and the details are as follows:
(A) The foreign trade robot intelligent customer service comprises the following steps: user access: the intelligent customer service workflow begins when a user makes contact with the system, and can go through various channels such as web chat windows, mobile applications, telephones, etc. The user's question, demand or intent will be passed as input to the intelligent customer service system. And (5) intention recognition: upon receiving user input, the intelligent customer service system may identify user intent using Natural Language Processing (NLP) techniques. Through text classification or sequence labeling and other techniques, the system can understand the problem types of users, such as inquiring information, submitting complaints, seeking technical support and the like. Knowledge base query: once the system recognizes the user's intent, it may retrieve relevant information via a search or query based on a pre-built knowledge base or database. These knowledge bases may include common problem solutions (FAQ), product documents, operating guidelines, and the like. The system will attempt to find an answer or solution from the knowledge base that matches the user's question. Automatic reply: if the system is able to find the appropriate answer from the knowledge base, it will generate an automatic reply and return it to the user. These automatic replies may be predefined templates, dynamically generated text, or generated results based on machine learning models. The goal of the automatic reply is to provide a clear, accurate, timely response to meet the needs of the user. Manual intervention: in some cases, the system may not be able to accurately understand the user's intent or find an appropriate answer. At this time, the intelligent customer service workflow can forward the questions to the human customer service operator for processing. The mechanism for transferring to the manual customer service can be through chatting or telephone and the like, so that the user can interact with the real customer service personnel. Continuous learning and optimization: intelligent customer service workflow is a continuously improved process. The system can continuously learn and optimize through user feedback, manual intervention data, monitoring indexes and other information. The foreign trade robot intelligent customer service also comprises the steps of updating a knowledge base, improving an intention recognition model, optimizing a reply generation algorithm and the like so as to provide better customer service experience and meet user requirements.
(B) Foreign trade intelligent recommendation system: user behavior collection: the intelligent recommendation workflow begins with the collection and recording of user behavior. This includes the user's browsing history on the platform, purchasing records, click behavior, ratings and comments, etc. Through the collection of these behavioral data, the system can learn about the user's preferences, interests, and behavioral patterns. Data cleaning and pretreatment: the collected user behavior data needs to be cleaned and preprocessed to remove noise and unnecessary information and converted into a format usable by the machine learning algorithm. This may include steps of data cleaning, feature extraction, data conversion, etc. Characteristic engineering: after data preprocessing, feature engineering is required to extract useful features to describe users and items. This may include user features (e.g., age, gender, geographic location, etc.) and item features (e.g., category, tags, keywords, etc.). The goal of feature engineering is to transform raw data into a representation of features that the algorithm can understand and process.
Model training: on the prepared features, training of the recommendation model may be performed using a machine learning algorithm or a deep learning model. Common recommendation algorithms include collaborative filtering, content-based recommendation, matrix factorization, and the like.
The model implementation details are as follows:
(1) the collaborative filtering recommendation algorithm is realized: a) Data preparation: the user-item scoring data is converted to a tensor or dataset format of pyrerch. B) Model definition: a PyTorch-based model is defined, which may be User-based collaborative filtering (User-Based Collaborative Filtering) or Item-based collaborative filtering (Item-Based Collaborative Filtering). C) Model training: the model is trained using training data, and parameter updates and optimizers using the PyTorch optimizer and loss function. D) Model evaluation: the trained model is evaluated using test data, and an evaluation index such as Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) may be used to evaluate model performance.
(2) The content-based recommendation algorithm implements: a) Data preparation: the feature data of the user and the item is converted into a tensor or dataset format of PyTorch. B) Model definition: a pyrerch-based model is defined and content recommendation algorithms implemented using neural networks and other machine learning models. D) Model training: training the model using training data allows for parameter updating and optimization using a PyTorch optimizer and loss function. D) Model evaluation: the trained model is evaluated using test data, and model performance may be evaluated using an evaluation index such as accuracy, recall, or F1-score.
(3) Matrix decomposition recommendation algorithm implementation: a) Data preparation: the user-item scoring data is converted to a tensor or dataset format of pyrerch. B) Model definition: a model based on pyrerch is defined and a matrix decomposition method such as SVD (singular value decomposition) or MF (matrix decomposition) is used to implement the recommendation algorithm. C) Model training: training the model using training data allows for parameter updating and optimization using a PyTorch optimizer and loss function. D) Model evaluation: the trained model is evaluated using test data, and an evaluation index such as Root Mean Square Error (RMSE) or Mean Absolute Error (MAE) may be used to evaluate model performance.
Through training the model, the system can learn the interests and preferences of the user and predict the user's preference for unknown items. Recommendation generation: once the trained model is available, the system may utilize the model for recommendation generation based on the user's personal information and current context. The recommendation generation may be user-user collaborative filtering based on user interest similarity, content-content collaborative filtering based on content similarity, or a hybrid recommendation algorithm combining the two. Recommendation feedback and assessment: after providing the recommended content to the user, the intelligent recommendation workflow may collect feedback and evaluation data of the user. These feedback data may be used to evaluate the performance of the recommendation algorithm and as part of the training data to refine and optimize the recommendation model.
Examples
The existing embodiments according to the invention are as follows:
(1) Through the connection with the online business data, the data hive table is obtained and updated in real time, and the data hive table comprises client consultation, order information, product characteristics and the like. These data will serve as the basis for training and evaluating intelligent customer service systems. The method comprises the following steps: (1) service data source on connection line: the system acquires the access right of the data source by connecting with an online service system. These data sources may include customer consultation records, order information, product characteristics, etc. to meet customer needs and provide accurate solutions. (2) The system establishes a real-time data synchronization mechanism with the online service system to ensure the timeliness and accuracy of the data. When online service data changes, the system can update the data hive table in time to reflect the latest service condition. (3) In the data reflow process, the system cleans and preprocesses the acquired data to remove noise, process missing values, normalize the data format, and the like. This helps to improve the data quality of subsequent training and evaluation and provides a reliable input for the deep learning algorithm. (4) The system stores the cleaned and preprocessed data in the hive table for subsequent training and evaluation.
(2) An intention recognition module is arranged, and the intention recognition is to classify the problems raised by the clients by developing an intention recognition model through a deep learning algorithm so as to determine the intention of the problems. Foreign trade intention recognition, which classifies the user's input words using a classification technique, plays an important role in the intelligent question-answering process, including product search, supplier information question-answering, merchandise information question-answering, foreign trade common knowledge question-answering, and other intents, etc. After the intention is identified, the user can interact with the user more accurately, and further the user satisfaction is improved.
Intent recognition is the classification of sentences or queries into corresponding intent categories by classification, which plays an important role in search engines as well as intelligent questions and answers. In short, this is important when used in search engines as well as intelligent questioning and answering. In short, when a user inputs a sentence or a text, the intention recognition can accurately recognize the problem of which domain, and then the problem is distributed to the corresponding domain robot for secondary processing, so that the accuracy of problem matching can be obviously improved under the condition that the problem is classified in a lot.
Examples are as follows: suppose that the rear end of the current customer service robot is connected with an investment advice (device), an investment education (edu) and a FAQ field robot (FAQ), and the corresponding corpus are respectively:
after the user asks questions, the current questions can be processed by corresponding subsequent field robots through the intention recognition module, for example, the investment advice class can further carry out semantic analysis, then an API interface is called to return results, the investment education class can call the API of the encyclopedia class to return results, and the questions of the FAQ class can be directly transferred to the interface of the manual customer service.
The basic methods for intent recognition commonly used at present are as follows:
dictionary-based and template-based rule method
Different domain dictionaries, such as book names, song names, trade names, etc., may be intended for different purposes. The user's intention and the matching degree or overlapping degree of the dictionary are judged, and the simplest rule is to judge the query to domain with high overlapping degree with the dictionary. The focus of this work is that the domain dictionary must be made sufficiently good.
Discriminating the intention of the user based on the machine learning model
The method is mainly characterized in that corpus in the marked field is trained and learned in a machine learning and deep learning mode, and an intention recognition model is obtained. By using the model, when a test set is input, the classification corresponding to the corpus can be rapidly predicted, and the corresponding confidence level is provided. One benefit of using this approach is that the accuracy of the model is continually improved after the corpus is continually enriched. The primary description is to use this approach for intent recognition.
Through training the data set and iterative optimization, the model can improve accuracy and generalization capability, so that the customer needs can be better understood. The method comprises the following steps: (1) data is prepared for training and evaluation of the intent recognition model, the dataset being derived from the hive table in step (1). This includes collecting and sorting a number of tagged customer question data, which is divided into training and test sets. The data should cover various possible intents to ensure the comprehensiveness and robustness of the model. (2) In the development process of the intent recognition model, a deep learning library, pyTorch, is introduced as a basic framework for model development and training. An appropriate deep learning model architecture is selected for the intent recognition task. And adopting a pre-trained BERT model as a basic model, and carrying out fine adjustment according to specific requirements. The BERT model has excellent performance in natural language processing tasks and can learn semantic and contextual information effectively. (3) The intent recognition model is trained using the prepared training data set. By associating the input questions with the corresponding intent tags, the model can learn the association between the questions and the intent. During training, model parameters are updated using a back-propagation algorithm and optimizer (Adam optimizer) to minimize the loss function.
The generated question-answer model utilizes a deep learning algorithm to generate accurate and consistent answers based on the client questions and the business data. By training a large corpus, the model can improve the quality and diversity of answers, so that the system can better cope with various client questioning conditions. The method comprises the following steps: (1) data is prepared for training to generate a question-answer model. This involves collecting and collating large-scale corpus data, including customer questions and corresponding accurate answers. The corpus should cover various possible question and answer scenarios to improve the generalization ability and coverage of the model. The data is obtained by (1) data reflow operation, (2) selecting a deep learning development framework pytorch, stacking transformer block to meet the requirement of generating a language model, training by utilizing the 3D parallel and data parallel frameworks of the platform, and evaluating the trained model by using a test data set. The evaluation index may include accuracy, fluency, diversity, etc. of generating the answer. According to the evaluation result, optimization adjustment of the model, such as adjustment of model architecture, increase of training data amount, adjustment of super parameters, etc., can be performed to further improve quality and diversity of generated answers.
And carrying out effect evaluation on the intention recognition, the named entity recognition and the generated question-answer model according to the online service data. The evaluation index may include accuracy, recall, F1 value, etc., as well as customer satisfaction and the ability to resolve problems.
The above embodiments do not limit the present invention in any way, and through the above description, the related workers can completely make various changes and modifications without departing from the scope of the technical idea of the present invention, and all other improvements and applications made to the above embodiments in equivalent transformation form belong to the protection scope of the present invention, and the technical scope of the present invention is not limited to the content on the description, and must be determined according to the scope of claims.

Claims (8)

1. An artificial intelligence platform based on e-commerce application is characterized by comprising the following modules: (1) AI platform base; (2) AI development platform; (3) AI service platform; (4) AI application platform; the AI development platform is bridged with the AI service platform and the application through a network; the AI platform base comprises basic hardware, data storage, an algorithm frame and a resource management module; the modules comprise the following components in detail:
(1) The basic hardware of the AI platform base supports CPU, GPU, FPGA, NPU hardware facilities;
(2) The data storage mode of the AI platform base supports fast storage OSS comprising FFS and Ceph storage, vector storage vearch, big data storage mode HDFS, hive, hudi and the like;
(3) The algorithm framework of the AI platform base supports pytorch, tensorflow, sklearn, keras, paddlepaddle, caffe, autoML, sparkML and other common algorithm libraries;
(4) The resource management system of the AI platform base comprises a kubespher container scheduling engine and a yarn resource management system;
the AI development platform comprises five modules, namely a data center, an algorithm center, a calculation center, a training center and a model center; the modules comprise the following components in detail:
(1) The data center module of the AI development platform comprises data processing functional modules such as data management, data interaction, data labeling and the like, and simultaneously supports complete data types of sample data, characteristic data and vector data;
(2) The algorithm center module of the AI development platform comprises functional modules such as algorithm management, algorithm evaluation and the like, and comprises common algorithms such as CV algorithm library, NLP algorithm library, recommended algorithm library, voice algorithm library, reinforcement learning algorithm library and the like;
(3) The computing power center module of the AI development platform comprises management modules such as user management, resource management, task management and the like, and simultaneously manages an A100 cluster, a V100 cluster and a 4090 cluster;
(4) The training center module of the AI development platform comprises four functional modules of data preparation, model training, model verification and model iteration;
(5) The model center module of the AI development platform comprises two modules of model management and reasoning optimization. .
2. The e-commerce application based deep learning development platform of claim 1, wherein the AI service platform comprises a CV middle stage, an NLP middle stage, a recommendation middle stage, a knowledge graph, and an AI portal.
3. The deep learning development platform based on the e-commerce application of claim 1, wherein the AI application platform comprises two large modules of platform intelligence and business intelligence, and the modules comprise the following details:
(1) The platform intelligence comprises a machine learning algorithm, a deep learning algorithm, CV tasks, NLP tasks and recommended tasks;
(2) The business intelligence comprises business modules such as foreign trade robots, intelligent sales, intelligent marketing, intelligent operation, intelligent recommendation, intelligent operation and maintenance.
4. The e-commerce application-based deep learning development platform according to claim 1, wherein an intention recognition module is provided, and the intention recognition is to classify questions raised by a customer by developing an intention recognition model using a deep learning algorithm to determine the intention thereof; foreign trade intention recognition classifies the user's input text using a classification technique; wherein the foreign trade intention recognition includes product search, supplier information question-answering, commodity information question-answering, foreign trade common knowledge question-answering, and other intents; the user can interact with the user more accurately after the intention is identified.
5. The e-commerce application-based deep learning development platform of claim 1, wherein the AI platform base comprises a self-lapping GPU pooling module, wherein the GPU pooling is configured to pool all GPU hardware resources; comprising the following steps: (A) The GPU pooling function is responsible for managing all GPU hardware resources on the platform; it tracks the status, availability and load conditions of each GPU and maintains a pool of available GPUs; (B) When a user submits a task or a job, the GPU pooling function dynamically allocates a proper amount of GPU resources to the task according to the requirements of the task and the current availability of the GPU resources; (C) The GPU pooling function is used for scheduling and queuing tasks through flexible resources; when one task is completed, the pooling function can bring the released GPU resources into an available pool again and distribute the released GPU resources to other tasks when needed so as to ensure the full utilization of all GPU resources; (D) GPU pooling supports management of task priorities;
the GPU pooling steps are as follows: (1) data preparation: copying input data from a host memory into a GPU device memory; using the memory allocation and transfer function of CUDA to realize; (2) writing a kernel function: writing a kernel function to realize pooling operation; the kernel function includes the following: determining a position of each thread and block on the input data using the thread and block index of the CUDA; calculating a pooling result in each region; using a max pooling or average pooling operation; storing the pooling result into output data; (3) parallelization: processing data of the plurality of regions using a parallel mechanism of CUDA; (4) shared memory: the shared memory is used in the kernel function to improve the data access efficiency; using the shared memory of the CUDA to store a local copy of the input data to reduce access to global memory; (5) and (3) data transmission: copying the output data from the GPU equipment memory back to the host memory; using the memory transfer function of CUDA (cudamecpy).
6. The deep learning development platform based on e-commerce application of claim 1, wherein the foreign trade robot intelligent customer service comprises the following procedures: user access: the intelligent customer service workflow starts with the contact of the user with the system, including a webpage chat window, a mobile application and a telephone; the user's question, demand or intent will be passed as input to the intelligent customer service system; performing intention recognition: after receiving the user input, the intelligent customer service system recognizes the user intention by using a Natural Language Processing (NLP) technology; through text classification or sequence labeling and other technologies, the system understands the problem types of the user, including inquiring information, submitting complaints and seeking technical support; knowledge base query: once the system recognizes the intention of the user, related information is obtained through searching or inquiring according to a pre-constructed knowledge base or database; the knowledge base comprises common problem solutions (FAQ), product documents and operation guidelines; the system will attempt to find an answer or solution from the knowledge base that matches the user's question; automatic reply: if the system is able to find the appropriate answer from the knowledge base, it will generate an automatic reply and return it to the user; these automatic replies may be predefined templates, dynamically generated text, or machine learning model based generation results; the aim of automatic reply is to provide clear, accurate and timely response so as to meet the demands of users;
continuous learning and optimization: intelligent customer service workflow is a continuously improved process; the system continuously learns and optimizes through user feedback, manual intervention data, monitoring indexes and other information; the foreign trade robot intelligent customer service also comprises the steps of updating a knowledge base, improving an intention recognition model and optimizing a reply generation algorithm.
7. The deep learning development platform based on the e-commerce application of claim 6, wherein a foreign trade intelligent recommendation system is provided: user behavior collection: the intelligent recommendation workflow starts with the collection and recording of user behavior; the method comprises the steps of browsing history, purchase records, clicking behaviors, evaluation and comments of a user on a platform; through the collection of these behavioral data, the system knows the user's preferences, interests and behavioral patterns; data cleaning and pretreatment: the collected user behavior data needs to be cleaned and preprocessed to remove noise and unnecessary information, and the collected user behavior data is converted into a format usable by a machine learning algorithm; the method comprises the steps of data cleaning, feature extraction and data conversion; characteristic engineering: after data preprocessing, feature engineering is required to extract useful features to describe users and projects; including user features and project features; the feature engineering converts the original data into feature representations which can be understood and processed by the algorithm;
model training: training a recommendation model on the prepared features by using a machine learning algorithm or a deep learning model; recommendation algorithms include collaborative filtering, content-based recommendation, and matrix factorization.
8. The e-commerce application based deep learning development platform of claim 7, wherein the model implementation steps are as follows:
(1) the collaborative filtering recommendation algorithm is realized: a) Data preparation: converting the user-item scoring data into a tensor or dataset format of pyrerch; b) Model definition: defining a PyTorch-based model, and collaborative filtering based on users; c) Model training: training the model by using training data, and updating and optimizing parameters by using an optimizer and a loss function of PyTorch; d) Model evaluation: evaluating the trained model using the test data, evaluating model performance using an evaluation index such as Root Mean Square Error (RMSE) or Mean Absolute Error (MAE);
(2) the content-based recommendation algorithm implements: a) Data preparation: converting the characteristic data of the user and the article into a tensor or dataset format of PyTorch; b) Model definition: defining a PyTorch-based model, and implementing a content recommendation algorithm by using a neural network and other machine learning models; d) Model training: training the model by using training data, and updating and optimizing parameters by using an optimizer and a loss function of PyTorch; d) Model evaluation: evaluating the trained model using the test data, evaluating model performance using an evaluation index;
(3) matrix decomposition recommendation algorithm implementation: a) Data preparation: converting the user-item scoring data into a tensor or dataset format of pyrerch; b) Model definition: defining a PyTorch-based model, and implementing a recommendation algorithm by using a matrix decomposition method such as SVD (singular value decomposition) or MF (matrix decomposition); c) Model training: training the model by using training data, and updating and optimizing parameters by using an optimizer and a loss function of PyTorch; d) Model evaluation: evaluating the trained model using the test data, evaluating model performance using an evaluation index such as Root Mean Square Error (RMSE) or Mean Absolute Error (MAE);
through a training model, the system learns interests and preferences of the user and predicts the preference degree of the user for unknown items; recommendation generation: once the trained model is available, the system uses the model to conduct recommendation generation according to the personal information and the current context of the user; recommendation generates a user-user collaborative filtering based on user interest similarity, a content-content collaborative filtering based on content similarity, or a hybrid recommendation algorithm combining the two; recommendation feedback and assessment: after providing recommended content to the user, the intelligent recommendation workflow will collect feedback and evaluation data of the user; these feedback data may be used to evaluate the performance of the recommendation algorithm and as part of the training data to refine and optimize the recommendation model.
CN202311495986.XA 2023-11-10 2023-11-10 Artificial intelligence platform based on E-commerce application Pending CN117474630A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311495986.XA CN117474630A (en) 2023-11-10 2023-11-10 Artificial intelligence platform based on E-commerce application

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311495986.XA CN117474630A (en) 2023-11-10 2023-11-10 Artificial intelligence platform based on E-commerce application

Publications (1)

Publication Number Publication Date
CN117474630A true CN117474630A (en) 2024-01-30

Family

ID=89634565

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311495986.XA Pending CN117474630A (en) 2023-11-10 2023-11-10 Artificial intelligence platform based on E-commerce application

Country Status (1)

Country Link
CN (1) CN117474630A (en)

Similar Documents

Publication Publication Date Title
CN109767255B (en) Method for realizing intelligent operation and accurate marketing through big data modeling
Chen et al. Automatic concept classification of text from electronic meetings
Fernández-García et al. A recommender system for component-based applications using machine learning techniques
CN108701256A (en) The system and method that the external interface of metadata driven for Application Programming Interface generates
US20150095303A1 (en) Knowledge Graph Generator Enabled by Diagonal Search
US20080086436A1 (en) Knowledge pattern search from networked agents
CN109145168A (en) A kind of expert service robot cloud platform
WO2020162943A1 (en) Holistic intelligence and autonomous information system and method thereof
de Carvalho et al. Knowledge management software
Calderón et al. Distributed supervised sentiment analysis of tweets: Integrating machine learning and streaming analytics for big data challenges in communication and audience research
Albreiki et al. Extracting topological features to identify at-risk students using machine learning and graph convolutional network models
Stollberg et al. H-Techsight—A next generation knowledge management platform
Shibolet et al. Resources for comparative analysis of IDN authoring tools
Li et al. A distributed data fabric architecture based on metadate knowledge graph
US20230281212A1 (en) Generating smart automated data movement workflows
CN117474630A (en) Artificial intelligence platform based on E-commerce application
US20180150543A1 (en) Unified multiversioned processing of derived data
Li et al. Enterprise precision marketing effectiveness model based on data mining technology
de Lucena Framework for collaborative knowledge management in organizations
Gil et al. Personalization on E-content retrieval based on semantic web services
Tran et al. Rules for an Ontology-based Approach to Adaptation
Devi et al. SoloDB for social media’s big data using deep natural language with AI applications and Industry 5.0
Prakash et al. App Review Prediction Using Machine Learning
US20220335307A1 (en) Knowledge graph management based on multi-source data
Tostrup et al. Massive patent data mining

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination