CN114612917A

CN114612917A - Order payment processing method and device, equipment, medium and product thereof

Info

Publication number: CN114612917A
Application number: CN202210217223.8A
Authority: CN
Inventors: 李保俊
Original assignee: Guangzhou Huaduo Network Technology Co Ltd
Current assignee: Guangzhou Huaduo Network Technology Co Ltd
Priority date: 2022-03-07
Filing date: 2022-03-07
Publication date: 2022-06-10

Abstract

The application discloses an order payment processing method and a device, equipment, medium and product thereof, wherein the method comprises the following steps: acquiring an image to be identified submitted by audience users in a live broadcast room; adopting a bill judging model to carry out classification judgment on the image to be identified so as to determine whether the image to be identified is a payment bill image; performing character recognition on the payment bill image by adopting a text detection recognition model to obtain a plurality of items of character information in the payment bill image; classifying the text information by adopting a text classification model so as to identify payment information in a classification way from the text information, wherein the payment information comprises payment amount and a remark order number; and responding to a payment confirmation instruction submitted by the anchor user, starting a payment cancellation event, and canceling the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information. The method and the device can identify and process the payment bill images uploaded by audience users, and finally realize the business closed loop of the e-commerce order.

Description

Order payment processing method and device, equipment, medium and product thereof

Technical Field

The present application relates to the field of e-commerce information technology, and in particular, to an order payment processing method, and a corresponding apparatus, computer device, computer-readable storage medium, and computer program product.

Background

On one hand, with the continuous change of online media forms, the service scene of the e-commerce platform is continuously changed, and the background service logic is also changed from early page shopping to live-broadcast-room shopping; on the other hand, in a cross-border e-commerce platform operated by multiple independent sites, different independent sites are located at different physical spatial positions, and different business logics need to be correspondingly realized by adapting to laws, humanities, infrastructure conditions and the like of different regions, so that the business logics are more and more complex. Therefore, in the existing shopping platform in advanced areas, the logic of full-automatic business from ordering to paying to logistics on line is not necessarily applicable to areas with insufficient infrastructure development.

In a typical practical application scenario, for example, in south-east asian countries, due to the underdeveloped infrastructure, after a user places an order in an e-commerce platform, the payment link of the user cannot directly call a bank payment interface to realize payment, so that the user needs to enter an application program of a bank or go to an online for additional payment, and finally communicates with a seller to confirm the payment fact, so that the user can complete an online shopping process. This phenomenon is caused by the market environment and legal environment of the concerned area, and the lack of adaptive matching technical conditions to systemize the service required in the situation. Therefore, related business logic is realized in the e-commerce platform, the execution efficiency of the e-commerce order is improved, the requirements of corresponding regions are met, and the e-commerce platform has practical significance.

Disclosure of Invention

A primary object of the present application is to solve at least one of the above problems and provide an order payment processing method and a corresponding apparatus, computer device, computer readable storage medium, and computer program product.

In order to meet various purposes of the application, the following technical scheme is adopted in the application:

an order payment processing method adapted to one of the purposes of the present application is provided, which comprises the following steps:

acquiring an image to be identified submitted by audience users in a live broadcast room;

adopting a bill judgment model trained to a convergence state in advance to carry out classification judgment on the image to be recognized so as to determine whether the image to be recognized is a payment bill image;

carrying out character recognition on the payment bill image by adopting a text detection recognition model trained to a convergence state in advance so as to obtain a plurality of items of character information in the payment bill image;

classifying the text information by adopting a text classification model trained to a convergence state in advance so as to identify payment information in a classification manner from the text information, wherein the payment information comprises payment amount and a remark order number, and pushing the payment information to a terminal device of a main broadcasting user in a live broadcasting room for display;

and responding to a payment confirmation instruction submitted by the anchor user, starting a payment cancellation event, and canceling the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information.

In an extended embodiment, the training process of the bill decision model includes the following steps:

calling a training sample in a first training data set, wherein the training sample comprises a positive sample and a negative sample, the positive sample is a payment bill image, and the negative sample is a non-payment bill image;

extracting deep semantic information from the training sample by adopting an image feature extraction network in a bill judgment model to obtain image feature information;

classifying and mapping the image characteristic information by adopting a classifier in a bill judgment model to obtain a corresponding classification label;

and calculating the loss value of the classification label according to the fact that the training sample belongs to a positive sample or a negative sample, correcting the weight parameter of the bill judgment model according to the loss value when the loss value is not converged, continuing iterative training, and terminating the training when the loss value reaches a convergence state.

In a further embodiment, the method for recognizing the characters of the payment bill image by using the text detection recognition model trained to the convergence state in advance to obtain a plurality of items of character information comprises the following steps:

performing text region identification on the payment bill image by the text detection and identification model to identify one or more text regions in the payment bill image;

cutting out text area images corresponding to the text areas from the payment bill image by the text detection and recognition model;

and performing character recognition on each text region image by the text detection recognition model to obtain corresponding character information.

In a further embodiment, the text classification model trained to a convergence state in advance is used to classify the text information, and the method includes the following steps:

performing word vector coding on all the items of character information by the text classification model to obtain corresponding word vector sequences;

extracting deep semantic information of the word vector sequence by the text classification model to obtain corresponding text characteristic information;

and carrying out classification mapping on the text characteristic information by the text classification model to obtain classification labels corresponding to the text information, wherein the classification labels comprise classification labels corresponding to payment amount and remark order numbers.

In a further embodiment, the step of initiating a payment cancellation event in response to a payment confirmation instruction submitted by the anchor user comprises the steps of:

receiving a payment confirmation instruction submitted by the anchor user, and acquiring a remark order number in payment information corresponding to the payment confirmation instruction;

calling an order corresponding to the remark order number, and offsetting the amount to be paid in the order by the payment amount contained in the payment information;

and judging whether the difference of the amount to be paid of the order after offset is less than or equal to 0, if so, triggering the order delivery business process, otherwise, sending a notification message to the anchor user and the audience user.

In an expanded embodiment, determining whether the difference of the reimbursed amount of the order is less than or equal to 0, and if yes, the method includes the following steps:

pushing a notification message representing that the user corresponding to the order completes payment to the live broadcast room, wherein the notification message comprises a link of an order placing business process of a commodity corresponding to the order;

responding to an access instruction of any user triggering the link in the live broadcast room, pushing a ordering page of the commodity corresponding to the order to the user, and sending a corresponding notification message to the live broadcast user.

An order payment processing apparatus adapted to one of the objects of the present application includes: the system comprises an image submitting module, an image classifying module, a text recognizing module, a text classifying module and a payment offset module. The image submitting module is used for acquiring an image to be identified submitted by a viewer user in a live broadcast room; the image classification module is used for classifying and judging the image to be recognized by adopting a bill judgment model which is trained to a convergence state in advance so as to determine whether the image to be recognized is a payment bill image; the text recognition module is used for carrying out character recognition on the payment bill image by adopting a text detection recognition model which is trained to be in a convergence state in advance so as to obtain a plurality of items of character information in the payment bill image; the text classification module is used for classifying the text information by adopting a text classification model which is trained to be in a convergence state in advance so as to classify and identify payment information from the text information, wherein the payment information comprises payment amount and a remark order number, and the payment information is pushed to terminal equipment of a main broadcasting user in a live broadcast room for display; and the payment cancellation module is used for responding to a payment confirmation instruction submitted by the anchor user, starting a payment cancellation event and canceling the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information.

In an extended embodiment, the order payment processing apparatus of the present application includes a training unit for executing a training process of the ticket decision model, and the training unit includes: the system comprises a sample calling module, a data acquisition module and a data processing module, wherein the sample calling module is used for calling a training sample in a first training data set, the training sample comprises a positive sample and a negative sample, the positive sample is a payment bill image, and the negative sample is a non-payment bill image; the semantic extraction module is configured to extract deep semantic information from the training samples by adopting an image feature extraction network in a bill judgment model to obtain image feature information; the classification mapping module is configured to perform classification mapping on the image characteristic information by adopting a classifier in a bill judgment model to obtain a corresponding classification label; and the iterative decision module is configured to calculate a loss value of the classification label according to whether the training sample belongs to a positive sample or a negative sample, modify the weight parameter of the bill judgment model according to the loss value when the loss value is not converged, continue iterative training and terminate the training when the loss value reaches a convergence state.

In a further embodiment, the text recognition module includes: a region identification sub-module configured to perform text region identification on the payment receipt image by the text detection identification model to identify one or more text regions in the payment receipt image; a region cropping sub-module configured to crop text region images corresponding to the plurality of text regions from the payment receipt image by the text detection recognition model; and the character recognition sub-module is configured to perform character recognition on each text region image by the text detection recognition model to obtain corresponding character information.

In a further embodiment, the text classification module includes: the vector coding sub-module is configured to perform word vector coding on all items of text information by the text classification model to obtain corresponding word vector sequences; the feature extraction sub-module is configured to extract deep semantic information of the word vector sequence by the text classification model to obtain corresponding text feature information; and the label judgment submodule is configured to perform classification mapping on the text characteristic information through the text classification model so as to obtain classification labels corresponding to the text information, wherein the classification labels comprise classification labels corresponding to the payment amount and the remark order number.

In a further embodiment, the payment cancellation module comprises: the instruction response submodule is used for receiving a payment confirmation instruction submitted by the anchor user and acquiring a remark order number in the payment information corresponding to the payment confirmation instruction; the order calling sub-module is used for calling the order corresponding to the remark order number and canceling the amount to be paid in the order by the payment amount contained in the payment information; and the flow control submodule is used for judging whether the difference of the cancelled to-be-paid money of the order is less than or equal to 0, if so, triggering an order delivery service process, and otherwise, sending a notification message to the anchor user and the audience user.

In an expanded embodiment, the flow direction control submodule determines whether a difference between the reimbursed amounts of the orders to be paid is less than or equal to 0, and if the difference is less than or equal to 0, the flow direction control submodule further performs the following functions: pushing a notification message representing that the user corresponding to the order completes payment to the live broadcast room, wherein the notification message comprises a link of an order placing business process of a commodity corresponding to the order; responding to an access instruction of any user triggering the link in the live broadcast room, pushing a ordering page of the commodity corresponding to the order to the user, and sending a corresponding notification message to the live broadcast user.

A computer device adapted for one of the purposes of the present application comprises a central processor and a memory, the central processor being adapted to invoke execution of a computer program stored in the memory to perform the steps of the order payment processing method described herein.

A computer-readable storage medium is provided, which stores in the form of computer-readable instructions a computer program implemented according to the order payment processing method, which when invoked by a computer performs the steps comprised by the method.

A computer program product, provided to adapt to another object of the present application, comprises computer programs/instructions which, when executed by a processor, implement the steps of the method described in any of the embodiments of the present application.

Compared with the prior art, the application has the following advantages: the method comprises the steps of firstly distinguishing the payment bill image of an image to be identified submitted by audience users in a live broadcast room, carrying out character identification on the payment bill image after the payment bill image is determined, obtaining a plurality of character information, then identifying the payment information according to preset business classification, wherein the payment information comprises a remark order number and a payment amount, and finally carrying out payment cancellation on the e-commerce order corresponding to the remark order number according to the payment amount to realize business closed loop of an ordering process, wherein the whole process is implemented based on an artificial intelligence technology, the functions of distinguishing the payment bill image, identifying the characters, respectively identifying the payment information and the like are respectively realized by means of each neural network model in the middle, the identification processing efficiency and the accuracy of the payment bill image can be ensured, thereby effectively avoiding manual checking of the payment bill image and excessive manual operation of an order, and improving the ordering automation efficiency of the e-commerce order, thereby perfecting the service closed loop of the e-commerce order.

Drawings

The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:

FIG. 1 is a schematic flow chart diagram of an exemplary embodiment of an order payment processing method of the present application;

FIG. 2 is a schematic flow chart of a training process of a bill decision model of the present application;

FIG. 3 is a schematic flow chart illustrating a text recognition process performed by the text detection recognition model in the embodiment of the present application;

FIG. 4 is a schematic flow chart illustrating classification and recognition of text information by a text classification model according to an embodiment of the present application;

FIG. 5 is a schematic flow chart of a process of an offset order in an embodiment of the present application;

FIG. 6 is a functional block diagram of an order payment processing arrangement of the present application;

fig. 7 is a schematic structural diagram of a computer device used in the present application.

Detailed Description

Reference will now be made in detail to the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.

As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.

It will be understood by those within the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

As will be appreciated by those skilled in the art, "client," "terminal," and "terminal device" as used herein include both devices that are wireless signal receivers, which are devices having only wireless signal receivers without transmit capability, and devices that are receive and transmit hardware, which have receive and transmit hardware capable of two-way communication over a two-way communication link. Such a device may include: cellular or other communication devices such as personal computers, tablets, etc. having a single line display or a multi-line display or cellular or other communication devices without a multi-line display; PCS (Personal Communications Service), which may combine voice, data processing, facsimile and/or data communication capabilities; a PDA (Personal Digital Assistant), which may include a radio frequency receiver, a pager, internet/intranet access, a web browser, a notepad, a calendar and/or a GPS (Global Positioning System) receiver; a conventional laptop and/or palmtop computer or other device having and/or including a radio frequency receiver. As used herein, a "client," "terminal device" can be portable, transportable, installed in a vehicle (aeronautical, maritime, and/or land-based), or situated and/or configured to operate locally and/or in a distributed fashion at any other location(s) on earth and/or in space. The "client", "terminal Device" used herein may also be a communication terminal, a Internet access terminal, and a music/video playing terminal, and may be, for example, a PDA, an MID (Mobile Internet Device), and/or a Mobile phone with music/video playing function, and may also be a smart television, a set-top box, and other devices.

The hardware referred to by the names "server", "client", "service node", etc. is essentially an electronic device with the performance of a personal computer, and is a hardware device having necessary components disclosed by the von neumann principle such as a central processing unit (including an arithmetic unit and a controller), a memory, an input device, an output device, etc., a computer program is stored in the memory, and the central processing unit calls a program stored in an external memory into the internal memory to run, executes instructions in the program, and interacts with the input and output devices, thereby completing a specific function.

It should be noted that the concept of "server" as referred to in this application can be extended to the case of a server cluster. According to the network deployment principle understood by those skilled in the art, the servers should be logically divided, and in physical space, the servers may be independent from each other but can be called through an interface, or may be integrated into one physical computer or a set of computer clusters. Those skilled in the art will appreciate this variation and should not be so limited as to restrict the implementation of the network deployment of the present application.

One or more technical features of the present application, unless expressly specified otherwise, may be deployed to a server to implement access by a client remotely invoking an online service interface provided by a fetch server, or may be deployed directly and run on a client to implement access.

Unless specified in clear text, the neural network model referred to or possibly referred to in the application can be deployed in a remote server and used for remote call at a client, and can also be deployed in a client with qualified equipment capability for direct call.

Various data referred to in the present application may be stored in a server remotely or in a local terminal device unless specified in the clear text, as long as the data is suitable for being called by the technical solution of the present application.

The person skilled in the art will know this: although the various methods of the present application are described based on the same concept so as to be common to each other, they may be independently performed unless otherwise specified. In the same way, for each embodiment disclosed in the present application, it is proposed based on the same inventive concept, and therefore, concepts of the same expression and concepts of which expressions are different but are appropriately changed only for convenience should be equally understood.

Unless expressly stated otherwise, the technical features of the embodiments disclosed in the present application may be cross-linked to form a new embodiment, so long as the combination does not depart from the spirit of the present application and can satisfy the requirements of the prior art or solve the disadvantages of the prior art. Those skilled in the art will appreciate variations therefrom.

The order payment processing method can be programmed into a computer program product and is deployed in an e-commerce platform cluster server to run. Referring to fig. 1, the order payment processing method of the present application, in an exemplary embodiment thereof, includes the following steps:

step S1100, acquiring an image to be identified submitted by a viewer user in a live broadcast room:

in an exemplary application scenario of the application, a live webcast is started on the basis of an online shop of an independent site of a cross-border e-commerce platform, a main broadcasting user promotes commodities to audience users in a live webcast room, the audience users perform commodity operation ordering in the online shop of the independent site, the independent site creates an order for the commodities at the background for the ordering event, generates a corresponding order number and an amount to be paid, and sends a payment notification message to the corresponding audience users, wherein the payment notification message comprises the order number, the amount to be paid, payment account information and the like, so that the audience users are instructed to finish payment of the order through other payment channels, and closed loop of ordering business is realized.

The viewer user can perform corresponding remittance based on the payment notification message in other application programs suitable for performing online payment of the mobile terminal of the viewer user so as to obtain a screenshot of a payment bill information interface returned after payment. The audience user can also handle the remittance service corresponding to the payment notification message by using an automatic teller machine of a bank or a manual remittance mode, and finally obtain a corresponding paper payment bill, and the paper payment bill can obtain a corresponding electronic image by using a scanning or shooting mode. Generally, when the audience user transacts the payment service, the order number is generally required to be provided as the remarked order number, and therefore, the payment information included in the obtained payment bill generally includes the remarked order number, the payment amount, the payment time, and the like, wherein the remarked order number and the payment amount have relatively more critical roles in realizing the closed loop of the order service.

As described above, the audience user can convert the payment bill obtained by completing payment outside the live broadcast room into image information by means of screenshot, shooting, scanning and the like, and the image information can be further submitted to a background system of the live broadcast room by the audience user, so that the application can recognize the payment information accordingly.

In order to realize the recognition of the payment information, the audience user is allowed to submit an image file in a graphical user interface of a live broadcast room of the audience user through a preset access entrance, and the image file is recognized by a server of the live broadcast room, namely a cluster server of an E-commerce platform as an image to be recognized so as to judge whether the image is a payment bill image containing the payment information or not.

Therefore, in an embodiment, an image publishing control is provided in an input field of a public screen running speech area of a live broadcast room, and after an audience user touches the image publishing control, the audience user can further determine an image to be recognized by any one of screenshot, shooting, searching local pictures and the like, then determine and send the image to be recognized to the public screen running speech area, and accordingly submit the image to be recognized to a server of an e-commerce platform through a background. This embodiment helps stimulate the traffic flow in the live room by allowing the audience users to publish their payment information to the public screen water announcement area to enable the public publication of the user payment information in the live room.

Step S1200, adopting a bill judgment model which is trained to a convergence state in advance to perform classification judgment on the image to be recognized so as to determine whether the image to be recognized is a payment bill image:

taking the above example of issuing the image to be recognized from the public screen flow area as an example, the image to be recognized uploaded to the e-commerce platform server by a user due to an error or due to the need of sending different types of pictures is not necessarily a payment bill image containing the payment information, in order to improve the efficiency of background payment information extraction, the application provides a bill judgment model for performing image type recognition on the image to be recognized so as to quickly determine whether the image to be recognized belongs to the payment bill image, and when the image to be recognized is not the payment bill image, the image to be recognized is ignored so as to be submitted to other business logic for processing, for example, when the image to be recognized is an expression image, the image to be recognized is directly pushed to the public screen flow area; when the image to be identified is the payment bill image, the image to be identified is submitted to the subsequent steps of the application for further payment information identification processing. Therefore, the payment bill images can be processed in a centralized manner in the subsequent steps of the method and the device, and the centralized processing efficiency of the subsequent steps is improved.

The bill judgment model is implemented by a convolution neural network model, for example, a basic model of a Resnet series version is used as an image feature extraction network for extracting deep semantic information from an image to be recognized, corresponding image feature information is obtained, and then the image feature information is classified and judged by a classifier to determine whether the image is the payment bill image. It can be easily understood that the bill judgment model is trained to a convergence state in advance, so that the bill judgment model learns the capability of representing and learning the image to be recognized and judging whether the image belongs to the bill image for payment according to the capability so as to be put into use. The following embodiments of the present application will further disclose the training process of the bill decision model, which is not shown here.

Step 1300, carrying out character recognition on the payment bill image by adopting a text detection recognition model which is trained to be in a convergence state in advance so as to obtain a plurality of items of character information:

in order to realize the character recognition of the payment bill image, the text recognition method is implemented by adopting a text detection recognition model, and the text detection recognition model consists of a line text detection network and a line text recognition network, wherein the line text detection network is used for detecting line text boxes in the payment bill image, and the line text recognition network is used for detecting character information of local images corresponding to the line text boxes.

In one embodiment, the line text detection network is used to detect the payment receipt image, and the detection method is text detection based on candidate boxes, specifically, pixel features of the payment receipt image are extracted, and by using the pixel features, a plurality of default boxes (also called anchor boxes) are set to generate a plurality of corresponding candidate text boxes, and a series of adjustment and screening are performed on the candidate text boxes. And further, calling an NMS (non-maximum suppression) algorithm to obtain a final text bounding box, namely the line text box and the coordinate position of the line text box corresponding to the payment bill image. The default box can be flexibly set by those skilled in the art based on a priori knowledge or experimental experience, and the line text detection network can be R2CNN, TextBox + +, SegLink, RFCN, CTPN, EAST, etc., and can be preferably called by those skilled in the art as required. Therefore, the local image is intercepted according to the coordinate position of the line text frame corresponding to the payment bill image, and a corresponding line text image and the line text image coordinates are obtained, wherein the line text image comprises one or more texts.

Further, a line text recognition network is called to recognize each line text image, the recognition is text line recognition, specifically, visual features of the line text image are extracted from a convolution backbone network to obtain a character feature sequence, the character feature sequence is input to a feature acquisition coding device aggregation part or the whole character feature sequence, and then the character feature sequence is input to a decoder to be transcribed to obtain corresponding text, namely, the line text data, the line text recognition network can be CNN + RNN + CTC, Attention-based CNN + RNN, and the like, and the line text recognition network can be called by technicians in the field according to needs in the field.

It is understood that the text detection and recognition model is implemented by using a convolutional neural network, and is trained to a convergence state in advance so as to enable the text detection and recognition model to learn the capability of obtaining one or more items of text information from the payment bill image, and the training process can be flexibly implemented by a person skilled in the art according to various training ideas without affecting the inventive embodiment of the application.

Each item of text information identified from the payment bill image is different according to the different information items included by the corresponding payment bill, but as the payment information required for completing order closed loop, the text information mainly comprises the remark order number and the payment amount, which are usually indispensable, therefore, the text information contains the indispensable key information, and can flexibly contain other richer information.

Step S1400, classifying the text information by adopting a text classification model trained to a convergence state in advance so as to classify and identify payment information from the text information, wherein the payment information comprises payment amount and a remark order number, and pushing the payment information to a terminal device of a user of a main broadcast of a live broadcast room for display:

in order to realize the utilization of various items of character information identified from the payment bill image, a text classification model is further adopted to extract deep semantic information of a word vector sequence formed by various items of character codes, and the deep semantic information is classified and mapped to a preset classification space, so that various items of character information are correspondingly corresponding to various preset classification labels of the classification space, wherein the classification labels include but are not limited to indication of payment amount, remark order number, payment time and the like in the payment information.

Similarly, the text classification model is suitable for being implemented by a convolutional neural network model, is trained to a convergence state in advance, and is used for obtaining the capability of performing classification mapping according to the coding results corresponding to the multiple items of character information so as to determine the classification labels corresponding to the various items of character information. The training process of the text classification model can be flexibly determined by a person skilled in the art according to a specifically adopted basic model, and the creative embodiment of the application is not influenced. In an embodiment actually measured by the applicant, LSTM is used as a basic model of the text classification model, so that a good test effect is obtained, correct classification of the text information can be realized, and the requirements of e-commerce platform application are met.

After the payment amount and the remark order number in the payment information are determined, generating a unicast message, wherein the unicast message comprises the payment information, particularly the payment amount and the remark order number, calling the summary information for generating an order corresponding to the remark order number to be contained in the unicast message, pushing the unicast message to the terminal equipment of the anchor user in the live broadcast room, displaying the unicast message in a graphical user interface of the anchor user after the unicast message is analyzed by the terminal equipment, and simultaneously providing a corresponding confirmation key so that the anchor user can verify and confirm the unicast message; if necessary, other correction keys can be provided, so that the anchor user temporarily archives the corresponding transaction for subsequent processing when the anchor user fails to confirm, or the anchor user directly executes error correction processing to correct the payment amount or the key payment information such as the memo order number. In any case, once the anchor user confirms that the payment information is correct, the confirmation button can be operated to trigger the corresponding payment confirmation instruction.

Step S1500, responding to a payment confirmation instruction submitted by the anchor user, starting a payment cancellation event, and canceling the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information:

and after the anchor user triggers the payment information, triggering a corresponding payment confirmation instruction to be sent to a cluster server of the E-commerce platform, and starting a payment cancellation event by the server in response to the payment confirmation instruction so as to implement a cancellation process of the to-be-paid amount of the order corresponding to the specified remark order number in the payment information.

The reimbursement process is mainly used for reimbursing the to-be-paid amount of the order corresponding to the remark order number in the payment information according to the payment amount in the payment information, namely obtaining the difference after reimbursement after subtracting the payment amount in the payment information from the to-be-paid amount. Generally speaking, the payment amount is equal to the amount to be paid, so that effective transaction can be achieved only when the difference is zero, thereby completing the payment business process of the order and realizing business closed loop. In some special cases, the payment amount in the payment information is allowed to be larger than the amount to be paid, so that the offset difference is larger than zero, for example, in the case that the audience user needs to pay additional tax based on the amount to be paid; in another special case, the payment amount in the payment information may even be allowed to be smaller than the amount to be paid, so that the difference after cancellation is smaller than zero, which is suitable for the case of discounting the commodity. In either case, selective presence is allowed because there is a link for the anchor user to confirm. However, if the convenience of the order delivery link is automatically driven in order to realize the automatic service closed loop, it can be generally set that when the payment amount in the payment information is greater than or equal to the amount to be paid, that is, the difference after the cancellation is less than or equal to zero, the triggering condition of the automatic service closed loop can be satisfied and the subsequent service link can be entered.

It is understood that after the cancellation process is completed, a payment service link of the order is theoretically completed, in the process, manual verification of the relevant image to be identified by the anchor user is avoided to the greatest extent, and payment condition confirmation only needs to be performed by means of payment information provided by the server. Because this application uses artificial intelligence technique in a large number, its intelligent degree is higher, and image discrimination efficiency, characters discernment rate of accuracy, words information classification efficiency to and operating efficiency all have good expression, therefore, can promote the execution efficiency of electricity merchant platform order greatly.

According to the typical embodiment and the variant embodiment of the application, the application firstly judges the payment bill image of the image to be identified submitted by the audience user in the live broadcast room, after the payment bill image is determined, character identification is carried out on the payment bill image to obtain a plurality of character information, then the payment information is identified according to the preset service classification, the payment information comprises the remark order number and the payment amount, finally the payment cancellation is carried out on the e-commerce order corresponding to the remark order number according to the payment amount to realize the service closed loop of the order process, the whole process is implemented based on the artificial intelligence technology, the functions of judging the payment bill image, identifying the character, respectively identifying the payment information and the like are respectively realized by means of each neural network model in the middle, the identification processing efficiency and the accuracy of the payment bill image can be ensured, thereby effectively avoiding the manual check of the payment bill image and the excessive manual operation of the order, the ordering automation efficiency of the e-commerce order is improved, and therefore the service closed loop of the e-commerce order is perfected.

Referring to fig. 2, in an expanded embodiment, the training process of the bill decision model includes the following steps:

step S2100, calling a training sample in a first training data set, where the training sample includes a positive sample and a negative sample, the positive sample is a payment bill image, and the negative sample is a non-payment bill image:

preparing a training data set, storing images serving as training samples in the training data set, and respectively marking the training samples in the training data set as positive samples and negative samples according to whether the images are payment bill images, so as to call each training sample in the training data set one by one to carry out iterative training on the bill judgment model of the application. And in each iterative training process, calling one training sample in the training data set to input the bill decision model.

Step S2200, extracting deep semantic information from the training sample by adopting an image feature extraction network in a bill judgment model to obtain image feature information:

and the image feature extraction network in the bill judgment model is responsible for representing and learning the training samples so as to extract deep semantic information of the training samples and obtain corresponding image feature information, and the image feature extraction network can adopt Resnet, CNN and the like as basic networks.

Step S2300, classifying and mapping the image characteristic information by adopting a classifier in a bill judgment model to obtain a corresponding classification label:

after the image feature information is obtained, the image feature information is mapped to a classifier through a full connection layer, and the classifier is preferably a binary classifier in this embodiment, so that the mapping result is a binarization result, and whether the image feature information is mapped to a classification label corresponding to a payment bill image is obtained.

Step S2400, calculating a loss value of the classification label according to whether the training sample belongs to a positive sample or a negative sample, correcting a weight parameter of a bill judgment model according to the loss value when the loss value is not converged, continuing iterative training, and terminating the training when the loss value reaches a convergence state:

as described above, the training data set is labeled with sample attributes corresponding to each training sample, i.e. positive samples or negative samples, and this is used as a supervision label, and a cross entropy function is applied to calculate a loss value of the binarization result obtained by the two classifiers, and then whether the bill decision model has been trained to a convergence state is determined accordingly according to whether the loss value reaches a preset value. When the ticket decision model has reached a converged state, the model training may be terminated; when the bill judgment model does not reach the convergence state, gradient updating needs to be carried out on the model by taking the loss value as a basis, weight parameters in each link of the model are corrected through back propagation, so that the loss function of the model further approaches to convergence, then the process goes to step S2100 to call the next training sample to continue to carry out iterative training on the model, and so on until the model is trained to the convergence state.

This embodiment adapts to this application needs, adopt the training data set to implement the training to bill judgement model in advance, so that the model can be put into this application after being trained to the convergence and use, because the model adopts two classifiers to carry out classification judgement, comparatively speaking, it is more simple high-efficient, can be trained to the convergence state more fast, when it is put into the inference phase, can judge fast high-efficiently whether treat the discernment image belongs to payment bill image, thereby avoid follow-up step of carrying out character recognition to produce a large amount of invalid calculations, the operating efficiency of this application technical scheme has been promoted.

Referring to fig. 3, in a further embodiment, the step S1300 of performing text recognition on the payment ticket image by using a text detection recognition model trained to a convergence state in advance to obtain a plurality of items of text information includes the following steps:

step S1310, performing text region identification on the payment receipt image by the text detection and identification model to identify one or more text regions in the payment receipt image:

as mentioned above, the text detection and recognition model includes a line text detection network, and in a preferred embodiment, the line text detection model is a textfusetnet model, which is responsible for text region detection of the payment receipt image. The payment bill image is input into a textFuseNet model which is pre-trained to be convergent after being preprocessed, the features of the global level are extracted through a semantic segmentation branch, the features of the character level and the word level are respectively extracted through a detection branch and a mask branch, after the features of the three levels of the character level, the word level and the global level are obtained, the features of the three levels are further fused by calling a multi-path feature fusion system structure, a more representative feature representation is generated, a text area in the image to be recognized is detected, a multi-point line text box surrounding the text area is obtained, the multi-point line text box is converted into a four-point line text box, four points of the four-point line text box are four vertexes corresponding to rectangles, and therefore, the rectangular line text box corresponding to the four-point line text box is further constructed to represent the corresponding text area.

Step S1320, cutting out text region images corresponding to the text regions from the payment receipt image by the text detection and recognition model:

and selecting a corresponding text area from the image to be recognized according to the rectangular line text box, taking a rectangular long line text image as a text area image, and obtaining the position of the text area image in the payment bill image represented by coordinates as the coordinates of the text area image.

In the process, the characters, the words and the global features are fused and converted into the detection result, so that the text detection with high accuracy can be realized, and the robustness and the reliability of the text detection are improved.

Step S1330, performing character recognition on each text region image by the text detection recognition model to obtain corresponding character information:

as mentioned above, the text detection and recognition model further includes a line text recognition network, which is responsible for performing text recognition on each text region image.

Before performing the character recognition, the text region image may be subjected to image preprocessing. The image preprocessing is to generate a new text region image by transforming an original text region image through data enhancement so as to expand a data source for an input model, and the data enhancement mode can include operations of horizontal or vertical flipping, multi-angle rotation, scaling, clipping, translation, interpolation, gaussian noise, contrast transformation, Fancy PCA and the like, and can be flexibly selected by a person skilled in the art as required.

The line text recognition network can preferably select a CRNN + CTC model, the preprocessed text region image is input into a pre-trained to converged CRNN + CTC model, a CNN is used in a convolution layer of the CRNN + CTC model to recognize and extract an image characteristic sequence from the text region image, the image characteristic sequence is input into a circulation layer of the CRNN + CTC model to predict the image characteristic sequence by using the RNN, and the corresponding label (true value) distribution is obtained. Further, the image feature sequence is subjected to full connection through a full connection layer in a transcription layer of the CRNN + CTC model, and the CTC is used for carrying out operations such as de-duplication integration and the like on label distribution corresponding to the image feature sequence, so that the image features in the image feature sequence are correspondingly mapped to a preset classification space, and corresponding text information is obtained according to the determined labels corresponding to the image feature sequence. Therefore, each text area can obtain the corresponding text information.

The method has the advantages that the CRNN + CTC model is called to identify the text region image to obtain the text, namely the text information, in the process, the robust features are extracted, the character segmentation with high difficulty in the traditional algorithm is avoided through sequence identification, and meanwhile, the sequence dependency is embedded in the sequence identification, so that the sequence with any length can be processed, and the robustness of text identification is greatly improved.

In the embodiment, the text information contained in the text area in the payment bill image is identified step by step through the text detection and identification model, so that character data which is easier to understand by a computer is obtained, the category of the character information is convenient to further identify subsequently, the identification accuracy of the payment information contained in the payment bill image is improved, and the robustness of the operation of the model can be enhanced because the model can select a high-quality basic model, so that the robust operation of the online service constructed on the basis of the model can be ensured.

Referring to fig. 4, in a further embodiment, the step S1400 of classifying the text information by using a text classification model trained to a convergence state in advance includes the following steps:

step S1410, the text classification model carries out word vector coding on all the items of character information to obtain corresponding word vector sequences:

in order to realize the type recognition of each item of text information recognized from the payment bill information, the text classification model is used for representing and learning the coding information of each item of text information, and the text classification model can be suitable for being trained to a convergence state by using basic models such as LSTM, Bert and the like which are suitable for deep semantic information extraction of texts.

Before representing and learning each item of text information, converting each item of text information into feature sequence information according to a preset dictionary, and then carrying out word vector coding on the feature sequence information by adopting a full-connection linear layer to obtain a word vector sequence. In one embodiment, the linear layer output dimension is set to 512 dimensions.

Step S1420, extracting deep semantic information of the word vector sequence by the text classification model to obtain corresponding text feature information:

and inputting the word vector sequence into the LSTM for representing learning by taking a text classification model adopting the LSTM as an example, and correspondingly setting the number of the hidden state features of the LSTM to be 512 dimensions, and acquiring text feature information corresponding to the word vector sequence after the LSTM performs memory learning.

Step S1430, the text classification model performs classification mapping on the text characteristic information to obtain classification labels corresponding to each text information, wherein the classification labels include classification labels corresponding to payment amount and remark order number:

in the text classification model, the text feature information obtained by LSTM is input to a classifier via a fully-connected linear layer for classification, the classification space of the classifier is set to multiple classifications corresponding to specific types of the payment information, that is, the fully-connected linear layer is set to corresponding multiple outputs, and different classifications correspond to different specific payment information, for example: amount name, payment amount, remark order number, payment time, other information, etc. Therefore, after the text characteristic information is classified and mapped by the text classification model, the classification labels corresponding to all the text information can be obtained, so that the payment information, particularly the payment amount and the remark order number, can be used for executing subsequent business links.

In this embodiment, the text classification model classifies each item of serialized text information, and distinguishes information categories corresponding to each item of text information, so that each item of specific payment information can be determined for completing a business closed loop of an e-commerce order.

Referring to fig. 5, in a further embodiment, the step S1500 of starting a payment cancellation event in response to a payment confirmation instruction submitted by the anchor user includes the following steps:

step S1510, receiving a payment confirmation instruction submitted by the anchor user, and acquiring a remark order number in the payment information corresponding to the payment confirmation instruction:

after the anchor user triggers the payment confirmation instruction and sends the payment confirmation instruction to the server of the e-commerce platform, the server can determine the remark order number in the corresponding payment information corresponding to the payment confirmation instruction so as to hit the order corresponding to the remark order number.

Step S1520, invoking the order corresponding to the remark order number, and reimbursing the amount to be paid in the order by the payment amount contained in the payment information:

according to the remark order number, the server calls a corresponding order from an order queue, the order is described by information such as the characteristic information of the audience user, the characteristic information of the commodity, the to-be-paid amount of the commodity and the like which are issued, so that the to-be-paid amount in the order can be offset by using the payment amount in the payment information identified from the payment bill image, namely the payment amount in the payment information is subtracted from the to-be-paid amount, offset is realized, the offset is obtained, and the offset is used for representing whether the order completes payment.

Step S1530, determining whether the difference between the cancelled amounts to be paid of the orders is less than or equal to 0, if yes, triggering an order shipping service process, otherwise, sending a notification message to the anchor user and the audience user:

in order to control the flow direction of the order, further comparing whether the difference is less than or equal to 0, if the judgment is not satisfied, according to the default service logic of the embodiment, it is determined that the order does not complete the full payment, and at this time, corresponding notification messages can be simultaneously sent to the anchor user and the audience user, so that the two parties coordinate with each other to complete the order payment process. If the judgment is true, the order is successful in completing payment, so that the business process corresponding to order delivery can be triggered, and the order is submitted to the service interface corresponding to the logistics link so as to start the delivery process.

In an embodiment expanded on this basis, when determining whether the difference between the cancelled amounts of money to be paid of the order is less than or equal to 0, if the difference is determined to be true, the following steps may be further performed:

step S1540, pushing a notification message indicating that the user corresponding to the order completes payment to the live broadcast, where the notification message includes a link of an order placing service process of a commodity corresponding to the order:

now that it is confirmed that the order has been paid in full, a notification message can be pushed to the live broadcast room where the anchor user and the audience user are located, to indicate the fact that the order has been paid in full, so that all users in the live broadcast room can receive the notification message. Typically, this notification message may be pushed directly into the public screen watershed of the live room, or other notification area presented in the graphical user interface of the live room. In order to facilitate the touch access of the user, the notification message contains a link of the order placing business process of the commodity corresponding to the order, and when the notification message is displayed on the graphical user interface, the text or image information displayed on the interface is associated with the link, so that the link is conveniently touched and accessed through the text or image information.

Step S1550, responding to the access instruction of any user triggering the link in the live broadcast room, pushing the order-placing page of the commodity corresponding to the order to the user, and sending a corresponding notification message to the live broadcast user:

after the display of the notification message on the graphical user interface of the live broadcast room is completed, when any user touches the notification message on the graphical user interface of the live broadcast room, the user jumps to enter a ordering page pointed by the link through the link associated with the notification message, and the ordering page displays the shopping business process of the commodity pointed by the order, so that the user can conveniently and quickly order the shopping page of the commodity, further and quickly recommend the commodity, and the execution efficiency of the user in the live broadcast room for ordering the commodity is improved.

In the embodiment, the functions of the method and the device are expanded by providing richer subsequent steps after the order is paid, so that the order delivery business process and the recommendation event of the commodity corresponding to the order can be seamlessly connected, thereby being beneficial to further improving the order execution efficiency of an e-commerce platform and realizing complete business closed loop.

Referring to fig. 6, an order payment processing apparatus adapted to one of the purposes of the present application is a functional implementation of the order payment processing method of the present application, and the apparatus includes: an image submission module 1100, an image classification module 1200, a text recognition module 1300, a text classification module 1400, and a payment cancellation module 1500. The image submission module 1100 is configured to obtain an image to be identified submitted by a viewer user in a live broadcast room; the image classification module 1200 is configured to perform classification judgment on the image to be recognized by using a bill judgment model trained to a convergence state in advance, so as to determine whether the image to be recognized is a payment bill image; the text recognition module 1300 is configured to perform text recognition on the payment ticket image by using a text detection recognition model trained to a convergence state in advance to obtain multiple items of text information therein; the text classification module 1400 is configured to classify each item of text information by using a text classification model trained to a convergence state in advance, so as to classify and identify payment information from the text information, where the payment information includes a payment amount and a remark order number, and push the payment information to a terminal device of a user anchor in a live broadcast room for display; the payment reimbursement module 1500 is configured to respond to a payment confirmation instruction submitted by the anchor user, start a payment reimbursement event, and reimburse the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information.

In a further embodiment, the text recognition module 1300 includes: a region identification sub-module configured to perform text region identification on the payment receipt image by the text detection identification module to identify one or more text regions in the payment receipt image; a region cropping sub-module configured to crop text region images corresponding to the plurality of text regions from the payment receipt image by the text detection recognition model; and the character recognition sub-module is configured to perform character recognition on each text region image by the text detection and recognition model to obtain corresponding character information.

In a further embodiment, the text classification module 1400 includes: the vector coding submodule is configured to perform word vector coding on all the items of character information by the text classification model to obtain corresponding word vector sequences; the feature extraction submodule is configured to extract deep semantic information of the word vector sequence by the text classification model to obtain corresponding text feature information; and the label judgment submodule is configured to perform classification mapping on the text characteristic information through the text classification model so as to obtain classification labels corresponding to the text information, wherein the classification labels comprise classification labels corresponding to the payment amount and the remark order number.

In a further embodiment, the payment cancellation module 1500 includes: the instruction response submodule is used for receiving a payment confirmation instruction submitted by the anchor user and acquiring a remark order number in the payment information corresponding to the payment confirmation instruction; the order calling sub-module is used for calling the order corresponding to the remark order number and canceling the amount to be paid in the order by the payment amount contained in the payment information; and the flow control submodule is used for judging whether the difference of the cancelled to-be-paid money of the order is less than or equal to 0, if so, triggering an order delivery service process, and otherwise, sending a notification message to the anchor user and the audience user.

In order to solve the technical problem, an embodiment of the present application further provides a computer device. As shown in fig. 7, the internal structure of the computer device is schematic. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected by a system bus. The computer readable storage medium of the computer device stores an operating system, a database and computer readable instructions, the database can store control information sequences, and the computer readable instructions can make the processor realize an order payment processing method when being executed by the processor. The processor of the computer device is used for providing calculation and control capability and supporting the operation of the whole computer device. The memory of the computer device may have stored therein computer readable instructions that, when executed by the processor, may cause the processor to perform the order payment processing method of the present application. The network interface of the computer device is used for connecting and communicating with the terminal. Those skilled in the art will appreciate that the architecture shown in fig. 7 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.

In this embodiment, the processor is used to execute the specific functions of each unit and its sub-unit in fig. 6, and the memory stores the program code and various data required for executing the above units or sub-units. The network interface is used for data transmission to and from a user terminal or a server. The memory in the present embodiment stores program codes and data necessary for executing all units/subunits in the order payment processing apparatus of the present application, and the server can call the program codes and data of the server to execute the functions of all subunits.

The present application also provides a storage medium having stored thereon computer-readable instructions which, when executed by one or more processors, cause the one or more processors to perform the steps of the order payment processing method of any of the embodiments of the present application.

The present application also provides a computer program product comprising computer programs/instructions which, when executed by one or more processors, implement the steps of the method as described in any of the embodiments of the present application.

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments of the present application can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when the computer program is executed, the processes of the embodiments of the methods can be included. The storage medium may be a computer-readable storage medium such as a magnetic disk, an optical disk, a Read-Only Memory (ROM), or a Random Access Memory (RAM).

In conclusion, the method and the device can ensure that the payment bill images uploaded by audience users are identified in the E-commerce live broadcast scene, and ensure the identification efficiency and accuracy, so that manual checking of the payment bill images and excessive manual operation of orders are effectively avoided, the ordering automation efficiency of E-commerce orders is improved, and finally the business closed loop of E-commerce orders is realized.

Those of skill in the art will appreciate that the various operations, methods, steps in the processes, acts, or solutions discussed in this application can be interchanged, modified, combined, or eliminated. Further, other steps, measures, or schemes in various operations, methods, or flows that have been discussed in this application can be alternated, altered, rearranged, broken down, combined, or deleted. Further, the steps, measures, and schemes in the various operations, methods, and flows disclosed in the present application in the prior art can also be alternated, modified, rearranged, decomposed, combined, or deleted.

The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.

Claims

1. An order payment processing method, comprising the steps of:

classifying each item of the character information by adopting a text classification model trained to a convergence state in advance so as to classify and identify payment information from the character information, wherein the payment information comprises payment amount and a remark order number, and pushing the payment information to a terminal device of a main broadcasting user in a live broadcasting room for display;

2. The order payment processing method of claim 1, wherein the training process of the bill decision model comprises the following steps:

3. The order payment processing method of claim 1, wherein the text detection recognition model pre-trained to a convergence state is adopted to perform text recognition on the payment ticket image to obtain a plurality of text messages therein, comprising the following steps:

4. The order payment processing method of claim 1, wherein the text classification model trained to a convergence state in advance is adopted to classify the text information, and the method comprises the following steps:

5. The order payment processing method of any one of claims 1 to 4, wherein initiating a payment cancellation event in response to a payment confirmation instruction submitted by the anchor user comprises the steps of:

6. The order payment processing method of claim 5, wherein the step of determining whether the difference of the amount to be paid of the order after being offset is less than or equal to 0, if yes, comprises the steps of:

7. An order payment processing apparatus, comprising:

the image submitting module is used for acquiring an image to be identified submitted by a viewer user in a live broadcast room;

the image classification module is used for carrying out classification judgment on the image to be recognized by adopting a bill judgment model which is trained to be in a convergence state in advance so as to determine whether the image to be recognized is a payment bill image;

the text recognition module is used for carrying out character recognition on the payment bill image by adopting a text detection recognition model which is trained to a convergence state in advance so as to obtain a plurality of items of character information;

the text classification module is used for classifying the text information by adopting a text classification model which is trained to be in a convergence state in advance so as to classify and identify payment information from the text information, wherein the payment information comprises payment amount and a remark order number, and the payment information is pushed to terminal equipment of a main broadcasting user in a live broadcast room to be displayed;

and the payment cancellation module is used for responding to a payment confirmation instruction submitted by the anchor user, starting a payment cancellation event and canceling the to-be-paid amount of the order corresponding to the remark order number by the payment amount in the payment information.

8. A computer device comprising a central processor and a memory, characterized in that the central processor is adapted to invoke execution of a computer program stored in the memory to perform the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that it stores a computer program implemented according to the method of any one of claims 1 to 6 in the form of computer-readable instructions, which, when invoked by a computer, performs the steps comprised by the corresponding method.

10. A computer program product comprising computer program/instructions, characterized in that the computer program/instructions, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 6.