KR102038111B1 - Payment computing device - Google Patents

Abstract

Disclosed is a method for making payment for a product according to one embodiment of the present disclosure. Operations stored in a computer program for processing the method comprise: an operation displaying a first screen to trigger payment including a first object for triggering an operation for product recognition; displaying a second screen for displaying a product image and a product recognition result based on the product image when a selection input for the first object is recognized; and displaying a third screen for displaying payment product information based on the product recognition result.

Description

Payment Computing Device {PAYMENT COMPUTING DEVICE}

The present invention relates to a payment method using a computing device, and more particularly to a payment method using a computing device.

In a time when many customers are in the store, there is a problem that the unnecessary waiting time of customers is long to make a calculation. Accordingly, there is a demand for a method that can speed up the calculation to reduce the waiting time.

In addition, due to the recent increase in labor costs and rising store rents, the demand for ways to reduce the operating costs of stores is increasing. Therefore, there is a demand for a method for reducing labor costs, which accounts for a large portion of the store operating costs.

Republic of Korea Patent No. 10-1817562 discloses a coffee beverage order processing device using an unmanned coffee beverage manufacturing machine and its operation method.

The present disclosure has been made in response to the above-described background, and an object thereof is to provide a product payment method.

Disclosed is a computer program stored in a computer readable storage medium according to one embodiment of the present disclosure for realizing the above-described problem. The computer program, when executed on one or more processors of a computing device, performs the following operations for product payment, the operations comprising: a first object for triggering an operation for product recognition. Displaying a first screen; Displaying a second screen for displaying a product image and a product recognition result based on the product image when a selection input to the first object is recognized; And displaying a third screen for displaying payment product information based on the product recognition result.

In an alternative embodiment of computer program operations for performing the following operations for product payment, the first screen comprises a first layer and a second layer in a form separated from each other, the first screen of the first screen The layer may include information for triggering a payment, and the second layer of the first screen may include the first object.

In an alternate embodiment of computer program operations for performing the following operations for product payment, the first area of the first screen includes information for triggering payment, and the second area of the first screen is It includes a first object, and the first region and the second region may be included on one layer.

In an alternate embodiment of computer program operations for performing the following operations for product payment, a selection input for at least one second object of at least one second object for selecting a payment product included in the first screen Recognizing; And generating the payment product information based on the selection input to the at least one second object.

In an alternative embodiment of the computer program operations to perform the following operations for product payment, the product recognition result is obtained by calculating the product image using a product recognition model including a pre-learned network function, A recognition result for each of one or more products included in the product image, and the product recognition result may be generated by the computing device or an external device.

In an alternative embodiment of the computer program operations to perform the following operations for product payment, the product recognition model performs an operation with the product image as input and is applied to each of one or more products included in the product image. Outputting a scoring value for each of the one or more classification classes, and outputting the product recognition result for the one or more products based on the classification class having a maximum scoring value of the scoring values for each of the one or more products. It may be a model.

In an alternative embodiment of computer program operations for performing the following operations for product payment, the second screen comprises a third object for triggering an operation for displaying payment product information, and the product recognition The operation of displaying a third screen for displaying payment product information based on a result may include determining that the product recognition result is an accurate result when the selection input to the third object is recognized, and display the third screen. It may include an operation to.

In an alternative embodiment of the computer program operations for performing the following operations for product payment, if there is an error in the product recognition result displayed on the second screen, receiving the correction information regarding the product recognition result. Based on the above, the method may further include generating additional training data for further training the product recognition model.

In an alternative embodiment of computer program operations for performing the following operations for product payment, the operation of displaying a third screen for displaying payment product information based on the product recognition result is one included in the product image. Receiving, for each of the above products, information converted into a product tag based on the product recognition result; And comparing the received product tag with a product tag that is matched to each of one or more products in a memory, and displaying the payment product information about the received product tag.

In an alternative embodiment of computer program operations to perform the following operations for product payment, the method comprises: sensing the weight of a product; And when the weight of more than a predetermined value is detected for more than a predetermined time, displaying the second screen.

Disclosed is a computer program stored in a computer readable storage medium according to one embodiment of the present disclosure for realizing the above-described problem. The computer program, when executed on one or more processors of a server, performs the following operations for product payment, the operations comprising: receiving a product image from a payment computing device; Generating a product recognition result by performing recognition on each of one or more products included in the product image; Identifying a product tag for the product recognition result based on one or more product tags stored in correspondence with each of the one or more products; And transmitting the identified product tag to the payment computing device.

A method for product payment performed in a computing device according to an embodiment of the present disclosure for realizing the above-described problem includes a first object for triggering an operation for product recognition. Displaying a first screen; If a selection input for the first object is recognized, displaying a second screen for displaying a product image and a product recognition result based on the product image; And displaying a third screen for displaying payment product information based on the product recognition result.

A computing device for product payment, according to an embodiment of the present disclosure for realizing the above-mentioned object, comprising: a processor including one or more cores; And memory; Wherein the processor displays a first screen for triggering a payment, the first screen including a first object for triggering an action for product recognition, and when a selection input for the first object is recognized, the product A second screen for displaying an image and a product recognition result based on the product image may be displayed, and a third screen for displaying payment product information based on the product recognition result may be displayed.

The present disclosure can provide a payment computing device.

Various aspects are now described with reference to the drawings, wherein like reference numerals are used to refer to like components throughout. In the following examples, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will be apparent that such aspect (s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects.
1 exemplarily illustrates a payment computing device and a payment server according to an embodiment of the present disclosure.
2 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.
3 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.
4 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.
5 exemplarily illustrates a payment method according to an embodiment of the present disclosure.
6 exemplarily illustrates a payment method according to an embodiment of the present disclosure.
7 illustrates an example flowchart of a payment method according to an embodiment of the present disclosure.
8 shows a general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments and / or aspects are now disclosed with reference to the drawings. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. However, it will also be appreciated by one of ordinary skill in the art that this aspect (s) may be practiced without these specific details. The following description and the annexed drawings set forth in detail certain illustrative aspects of the one or more aspects. However, these aspects are exemplary and some of the various methods in the principles of the various aspects may be used and the descriptions described are intended to include all such aspects and their equivalents.

Moreover, various aspects and features will be presented by a system that may include one or more computing devices, terminals, servers, devices, components and / or modules, and the like. Various systems may include additional computing devices, terminals, servers, devices, components and / or modules, and / or the devices, terminals, servers discussed in connection with the drawings and / or drawings. It should also be understood and appreciated that it may not include all of the devices, components, modules, and the like.

As used herein, the terms "computer program", "component", "module", and the like may be used interchangeably. For example, a component may be, but is not limited to being, a process running on a processor, a processor, an object, a thread of execution, a program, and / or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components can reside within a processor and / or thread of execution. One component can be localized within one computer. One component may be distributed between two or more computers.

In addition, these components can execute from various computer readable storage media having various data structures stored thereon. The components may, for example, via a network such as the Internet and other systems via signals with one or more data packets (e.g., data and / or signals from one component interacting with other components in a local system, distributed system). Data transmitted) may be communicated via local and / or remote processes.

As used herein, “an embodiment”, “an example”, “aspect”, “an example”, etc., may not be construed that any aspect or design described is better or advantageous than other aspects or designs. . The terms 'component', 'module', 'system', 'interface', etc. as used below generally refer to a computer-related entity, for example, a combination of hardware, hardware and software. Or software.

In addition, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. In other words, unless specified otherwise or unambiguously in context, "X uses A or B" is intended to mean one of the natural implicit substitutions. That is, X uses A; X uses B; Or where X uses both A and B, "X uses A or B" may apply in either of these cases.

In addition, the terms "comprises" and / or "comprising" mean that such features and / or components are present, but exclude the presence or addition of one or more other features, components, and / or groups thereof. It should be understood that it does not. Also, unless otherwise specified or in the context of indicating a singular form, the singular in the specification and claims should generally be interpreted as meaning "one or more."

Computer-readable media herein can include any kind of media on which programs and data are stored so that they can be read by a computer system. Computer readable media in the present disclosure may include computer readable storage media and computer readable transmission media. According to one aspect of the present invention, a computer readable storage medium includes: ROM (Read Only Memory), RAM (Random Access Memory), CD (Compact Disk) -ROM, DVD (Digital Video Disk) -ROM, Magnetic Tape, Floppy Disk, optical data storage, and the like. In addition, the computer readable transmission medium may include any transmission type of medium implemented in the form of a carrier wave (eg, transmission over the Internet). In addition, such computer readable media may be distributed over network coupled systems to store computer readable codes and / or instructions in a distributed fashion.

Before describing the details for carrying out the present invention, it should be noted that configurations that are not directly related to the technical gist of the present invention are omitted within the scope of not distracting the technical gist of the present invention. In addition, the terms or words used in the present specification and claims are intended to comply with the technical spirit of the present invention based on the principle that the inventor can define the concept of appropriate terms in order to best explain the invention. It should be interpreted as a concept.

1 exemplarily illustrates a payment computing device and a payment server according to an embodiment of the present disclosure.

The computing device 100 of the present disclosure may include other components for performing the computing environment of the computing device. The computing device 100 of the present disclosure may be a computing device (eg, a POS device of a store) operating in a store. Hereinafter, the computing device 100 may be referred to as a payment computing device 100.

The computing device 100 may include a network unit 110, a processor 120, a memory 130, and a display unit 140. Although not shown in FIG. 1, the computing device 100 may further include a camera unit.

The network unit 110 may transmit / receive data for performing a product payment method according to an embodiment of the present disclosure to the server 200, another computing device, and the like. The network unit 110 may transmit / receive data necessary for an embodiment of the present disclosure, such as a product image, a product recognition result, a product tag, and the like, to the server 200 or another computing device. The network unit 110 may enable communication with a plurality of computing devices or servers to distribute processing for performing product payment in each of the plurality of computing devices.

The processor 120 may include one or more cores, and may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. and a processor for payment of a product such as a unit). The processor 120 may read a computer program stored in the memory 130 to perform a product payment method according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 120 may perform calculation for product payment.

At least one of the CPU, the GPGPU, and the TPU of the processor 120 may process a product image operation, a product tag, or the like for product payment. For example, the CPU and GPGPU can work together to compute product images for product payments, compute product tags, and so on. In addition, in an embodiment of the present disclosure, a processor of a plurality of computing devices may be used together to process operations for product images for product payment, payment product information calculation based on product tags, and the like. In addition, the computer program executed in the computing device according to an embodiment of the present disclosure may be a CPU, a GPGPU, or a TPU executable program.

The memory 130 may store a computer program for performing a product payment method according to an embodiment of the present disclosure, and the stored computer program may be read and driven by the processor 120.

The memory 130 according to embodiments of the present disclosure may store a program for the operation of the processor 120, and input / output data (eg, a product image, a product recognition result, a product tag, etc.). It can also be stored temporarily or permanently. The memory 130 may store product tag and payment product information for each of one or more products. The memory 130 may store data regarding a display and sound. The memory 130 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM (Random Access Memory, RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic At least one type of storage medium may be included.

A camera (not shown) may generate a product image by photographing a product in order to perform a product payment according to an embodiment of the present disclosure. In one embodiment of the present disclosure, computing device 100 may include one or more cameras.

The server 200 of the present disclosure may include other components for performing the server environment of the server. The server 200 of the present disclosure may be a server managing one or more computing devices 100 (for example, a head office server operating a plurality of stores).

The server 200 may include a network unit 210, a processor 220, and a memory 230.

The network unit 210 may transmit / receive data for performing a product payment method according to an embodiment of the present disclosure to the server 200, another computing device, and the like. The network unit 210 may transmit and receive data necessary for an embodiment of the present disclosure, such as a product image, a product recognition result, a product tag, and the like to the computing device 100, another server, or another computing device.

The processor 220 may include one or more cores, and may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a server. And a processor for product payment. The processor 220 may read a computer program stored in the memory 230 to perform a product payment method according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor 220 may perform a calculation for product payment.

At least one of the CPU, the GPGPU, and the TPU of the processor 220 may process a product image operation, a product tag, or the like for product payment. For example, the CPU and GPGPU can work together to compute product images for product payments, compute product tags, and so on. In addition, in an embodiment of the present disclosure, a processor of a plurality of servers may be used together to process operations on product images for product payment, product tag identification based on product images, and the like. In addition, the computer program executed in the server according to an embodiment of the present disclosure may be a CPU, GPGPU or TPU executable program.

Throughout this specification, neural networks, network functions, neural networks may be used in the same sense. In this specification, the network function may include one or more neural networks, in which case the output of the network function may be an ensemble of the outputs of the one or more neural networks. A deep neural network (DNN) may refer to a neural network including a plurality of hidden layers in addition to an input layer and an output layer. Deep neural networks can be used to identify potential structures in your data. In other words, you can identify potential structures of photos, text, video, voice, and music (e.g., what objects are in the picture, what the content and emotions of the text are, and what the content and emotions of the voice are). . Deep neural networks include convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), and deep belief networks (DBNs). , Q network, U network, Siamese network and the like.

In this specification, the model may include a network function. The model may include one or more network functions, in which case the output of the model may be an ensemble of outputs of one or more network functions.

The memory 230 may store a computer program for performing a product payment method according to an embodiment of the present disclosure, and the stored computer program may be read and driven by the processor 220.

The memory 230 according to embodiments of the present disclosure may store a program for the operation of the processor 220 and store input / output data (eg, a product image, a product recognition result, a product tag, etc.). It can also be stored temporarily or permanently. The memory 230 may store a product tag or a product identification model for each of one or more products. The memory 230 may store data regarding a display and a sound. The memory 230 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM (Random Access Memory, RAM), Static Random Access Memory (SRAM), Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Programmable Read-Only Memory (PROM), Magnetic Memory, Magnetic At least one type of storage medium may be included.

Hereinafter, a payment method using the computing device 100 and the server 200 according to an embodiment of the present disclosure will be described.

The computing device 100 may display a first screen 300 for triggering a payment, including a first object 310 for triggering an operation for product recognition. Hereinafter, the first screen 300 will be described with reference to FIG. 2. 2 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.

The computing device 100 may display the first screen 300 on the display 140 by default in order to perform a payment. If the computing device 100 does not recognize an input for an object displayed on a screen for a predetermined time or more while performing a payment, or does not receive data from the server 200 or another computing device 100, One screen 300 may be displayed. Alternatively, for example, when driving a program for payment of the computing device 100, the first screen 300 may be displayed as a start screen.

The first screen 300 may be a screen for triggering a payment. When recognizing an input for one object of two or more objects included in the first screen 300, an operation for payment may be started. When the selection input to the first object 310 for triggering an operation for product recognition included in the first screen 300 is recognized, the first payment method of the method of recognizing the product based on the product image is triggered. Can be. Alternatively, when a selection input for two or more second objects 320 for selecting a payment product included in the first screen 300 is recognized, a second payment method based on the recognized payment product may be triggered. The above-described first payment method and second payment method will be described in detail later.

The first screen 300 may include information for triggering payment and the first object 310. The information for triggering the payment may include information regarding product selection and payment for payment. The product selection information for payment may be two or more second objects 320 for selecting a payment product. Each of the two or more second objects 320 included in the first screen 300 may be an object for distinguishing one product from another product and recognizing an input. For example, if a user in a store wants to purchase A and B products, the store clerk enters for the second object A and the second object B, and the computing device 100 is one of the two or more second objects 320. The payment may be performed by recognizing inputs to the second object A and the second object B. FIG. In FIG. 2, only one second object is displayed, but other objects for recognizing information about a product may also be the second object. For example, although only the second object 320 for 'milk plus' is shown in FIG. 2, the object for 'coffee bun', 'mochi mochi cheese stick', etc. may also be the second object 320. The information for triggering the payment may include payment means information (e.g., points, cash, cards, etc.), payment amount information, information for inputting a payment amount (e.g., numbers, erase, Enter, etc. of FIG. 2). And the like may further be included. Specific description of the information for triggering the above-described payment is only an example and the present disclosure is not limited thereto.

The first object 310 may be an object for triggering product recognition. Although the first object is shown in the shape of a camera in FIG. 2, this is an exemplary shape and may be displayed on the first screen 300 in any other shape.

The first object 310 is included in the first screen 300 and may also be displayed on at least one of the second screen 400 and the third screen 500.

A layer according to an embodiment of the present disclosure may refer to a layer used to overlap and display a plurality of images on a screen displayed on the computing device 100.

In an embodiment of the present disclosure, the first screen 300 includes a first layer and a second layer separated from each other, and the first layer of the first screen 300 includes information for triggering a payment. And the second layer of the first screen may include the first object 310. The first layer and the second layer may be layers of different sizes. The first layer may be located on a higher layer than the second layer and may be a layer having a smaller size. Although the first layer is illustrated as an opaque layer in FIG. 2, the first layer may be displayed on the first screen 300 as a semi-transparent layer. The second layer may include a first object for triggering an operation for product recognition.

The first layer may be moved to another position of the first screen 300 based on the recognition of the input of the display unit 140 of the computing device 100. When the input to the first layer is recognized for more than a predetermined time and the drag input to another position of the first screen is recognized, the input may be moved from the first position to the second position and displayed.

In one embodiment of the present disclosure, the first area of the first screen includes information for triggering a payment, the second area of the first screen includes the first object, and the first area and The second area may be included on one layer.

One layer may be divided into a plurality of regions. The layer of the first screen displayed on the computing device 100 according to an embodiment of the present disclosure may be divided into a first area and a second area. The first area and the second area may be areas of the first screen 300 for displaying different information that do not overlap. The first area and the second area may be fixed according to a setting of a program running on the computing device 100.

A method of manually performing a payment according to an embodiment of the present disclosure will be described. In detail, when a selection input for two or more second objects 320 for selecting a payment product included in the first screen 300 is recognized, a second payment method based on the recognized payment product will be described.

The computing device 100 may recognize a selection input for at least one second object 320 among two or more second objects 320 for selecting a payment product included in the first screen. Product information and product value of each of the two or more second objects 320 may be stored in a memory of the computing device 100.

The computing device 100 may generate the payment product information based on the selection input for the at least one second object. The generated payment product information will be described with reference to FIG. 4. 4 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.

When the computing device 100 recognizes a selection input for at least one second object 320 of the two or more second objects 320, the computing device 100 identifies the stored product information and the product value with respect to the recognized second object, The payment product information 510 may be generated. The payment product information 510 may include information about each of two or more products that a user intends to purchase at a store. The generated payment product information 510 may be displayed on the third screen 500. The payment product information 510 may be generated based on two or more second objects 320 in which an input is recognized. For example, based on the egg tart second object 320 and the coffee burn second object 320 entered as a store clerk identifies the egg tart and the coffee burn brought by the user for payment, the computing device 100 May recognize selection inputs for the egg tart second object 320 and the coffee burn second object 320, and for each second object 320, product information and product value stored in the memory 130 may be obtained. The calculation may display the payment product information 510 of the third screen 500. Specific description of the above-described second payment method is only an example and the present disclosure is not limited thereto.

According to a second payment method according to an embodiment of the present disclosure, a store clerk recognizes a product brought by a customer to make a payment with the naked eye, and based on the recognized product, a second object included in the first screen 300 ( By inputting 320, the computing device 100 may be a method of performing a payment by recognizing a product to be paid based on the second object 320 on which the input is recognized. If the customer wants to pay a small amount (e.g., one or two), the clerk checks the product with the naked eye without using the first payment method to reduce the consumption of computing power. can do. Another payment method may be performed based on whether the store clerk inputs the first object 310.

A method of performing payment based on a product image according to an embodiment of the present disclosure will be described. Specifically, when a selection input for the first object 310 for triggering an operation for product recognition included in the first screen 300 is recognized, a first payment of a method of recognizing a product based on a product image The method is explained.

When the selection input to the first object 310 is recognized, the computing device 100 may display a second screen 400 for displaying a product image and a product recognition result based on the product image. Hereinafter, the second screen will be described with reference to FIG. 3. 3 illustrates an example UI configuration for payment according to an embodiment of the present disclosure.

When the selection input to the first object 310 is recognized, the computing device 100 connects the camera unit (not shown) included in the computing device 100, or the computing device 100 and the network unit 110. This allows the camera to send and receive data to capture images of the product. For example, computing device 100 may allow a customer to photograph one or more products placed on a tray of a cashier for payment. The computing device 100 may obtain the product image 410 by using the camera unit or the camera.

The computing device 100 may display the obtained product image 410 on the second screen 400 of the display 140.

The computing device 100 may display the product recognition results 413 and 450 of the product 411 included in the product image 410 on the second screen 400.

The product recognition result may be generated at the computing device 100 or the server 200. The product recognition result includes a recognition result for each of one or more products included in the product image, obtained by calculating the product image using a product recognition model including a pre-learned network function, and the product The recognition result may be generated by the computing device or an external device. The product recognition model performs an operation by inputting the product image, outputs a scoring value for each of one or more classification classes, for each of one or more products included in the product image, and outputs each of the one or more products. It may be a model that outputs the product recognition results for the one or more products based on the classification class having the maximum scoring value of the scoring values for.

The computing device 100 may transmit the product image to the server 200 through the network unit 110 to obtain a product recognition result based on one or more products included in the product image.

The server 200 may receive a product image from the payment computing device 100. The server 200 may generate a product recognition result by performing recognition on each of one or more products included in the product image.

The processor 220 of the server 200 may output a scoring value for each of one or more classification classes by calculating a product image using a product recognition model including one or more network functions.

The server 200 may obtain a training data set. The training data set may be labeled with learning product information on each of the entire learning product image and the entire learning product image. The learning product information may include information for identifying the product object with other product objects. The learning product information may be a trade name for a product object. The learning product information may be a product name for a product object and a product value matching the product name. For example, the learning product information (in this example, coffee bun) may be labeled on a product (in this example, coffee bun product) included in the learning product image. The server 200 may obtain product information by calculating each of the training data included in the labeled training data set using a product recognition model. The server 200 may calculate an error by comparing the product information obtained by calculating using the product recognition model and the learning product information that is a label of the learning data. The server 200 may reverse propagate the error calculated in the direction of the input layer in the output layer of the product recognition model, which is the reverse direction. The server 200 may propagate the error in the direction of the input layer from the output layer to update the connection weight of each node of each layer according to the back propagation. The connection weight of each node to be updated may be determined according to a learning rate. The description of the learning of the aforementioned product recognition model is merely an example and the present disclosure is not limited thereto.

The server 200 may output a scoring value for each of one or more classification classes by calculating a product image input to an input node of the product recognition model through the product recognition model. The classification class may be for identifying product information. For example, the classification class may include product information for all the confections and bakeries stored in the memory 230 of the server 200 (in this example, the head office server managing a plurality of stores). For example, the classification class is arranged by the computing device 100 of the product information (in this example, cheese bun, coffee bun, pretzel) stored in the memory 230 of the server 200 (store head office in this example). Only the product information sold at the store (in this example, cheese buns, pretzel, which are the breads produced at the store) (or in this example, all three products were produced at the store, but the customer bought all the cheese buns. Coffee burns, pretzel) containing only the remaining products. The description of the above-described classification class is only an example and the present disclosure is not limited thereto.

The server 200 may output a product recognition result based on the scoring value for each of the one or more classification classes. The product recognition result may include information about each of the products and a location on the product image for each of one or more products included in the product image. The server 200 may output a product recognition result based on a classification class that is equal to or greater than a predetermined threshold value among scoring values for each of the one or more classification classes and has a maximum scoring value. For example, when a product image includes an image about a coffee bun product, a scoring value of the coffee bun classification class among two or more classification classes is equal to or greater than a predetermined threshold value and may be a scoring value having a maximum scoring value.

The server 200 may transmit the product recognition result obtained through the product recognition model to the computing device 100 through the network unit 210.

The computing device 100 may display the received product recognition results 413 and 450 on the second screen 400.

According to an embodiment of the present disclosure, the computing device 100 displays a product recognition result 413 received from the server 200 on a product image 410 including one or more products 411, so that each product You can check the product recognition result for. The computing device 100 may identify the location and product information of the product in the product image included in the product recognition result received from the server 200, and display the product information in a portion corresponding to the location of the product. For example, referring to FIG. 3, the computing device 100 displays a product recognition result for each product 411 included in the product image 410, and displays the product recognition result of the product included in the product recognition result. The location may be identified and the product information may be displayed in the part corresponding to the location of the identified product. For example, the computing device 100 confirms where the whole wheat ham sandwich product included in the product recognition result received from the server 200 is located on the product image, and receives the product recognition result received from the server 200. By checking the 'whole wheat ham sandwich' product information of the whole wheat ham sandwich product included in the product, and displaying the product information 'whole wheat ham sandwich' on the upper portion of the whole wheat ham sandwich product, the product on the second screen 400 The recognition result 413 may be displayed. The computing device 100 may display the product recognition result 413 opaquely on the product image 410 of the second screen 400.

According to an embodiment of the present disclosure, the computing device 100 lists the product information on one side of the second screen 400 based on one or more product information included in the product identification result received from the server 200. As a result, the product recognition result 450 may be displayed. For example, when the product included in the product image 410 is a whole wheat ham sandwich, breakfast n fresh milk, egg tart and coffee bun, the computing device 100 may allow the store clerk to check product information about each product. The product recognition result 450 may be displayed by displaying the product information 'whole wheat ham sandwich', 'morning & fresh milk', 'eg tart' and 'coffee bun' on one side of the second screen 400. The store clerk can check at a glance which of the products the customer intends to pay through the product recognition result 450 displayed on the second screen 400.

The detailed description of the method of displaying the above-described product recognition result on the second screen is only an example and the present disclosure is not limited thereto.

When there is an error in the product recognition results 413 and 450 displayed on the second screen 400, the computing device 100 may generate a product recognition model based on receiving correction information regarding the product recognition result. Additional training data can be generated for further training.

If there is an error in the product recognition results 413 and 450, product information regarding at least some of the one or more products included in the product image 410 may be displayed incorrectly. For example, for the egg tart product, the product information called coffee burn is displayed on the second screen 400, or the egg tart product information is displayed on the tray on which the product is not placed, rather than the egg tart product. In this case, there may be an error in the product recognition result. The detailed description of the above-described product recognition result error is only an example and the present disclosure is not limited thereto.

The correction information regarding the product recognition result may include a product position and product information of the product in the product image. For example, when a store clerk has an error in a product recognition result displayed on the second screen 400, correction information regarding the product recognition result may be input to the computing device 100, and the computing device 100 may be input. May receive the egg tart, which is the location and product information of the egg tart, from the product image.

The computing device 100 may generate additional training data for further learning the product recognition model based on the correction information about the product recognition result. The computing device 100 generates a bounding box of the product based on the position of the product in the product image included in the correction information, and adds the labeling of the product information to the bounding box based on the product information included in the correction information. Training data can be generated.

The computing device 100 may transmit the generated additional training data to the server 200 to perform additional training on the product recognition model.

The server 200 may receive additional learning data from the computing device 100.

According to another exemplary embodiment of the present example, the server 200 receives correction information about a product recognition result from the computing device 100, so that the processor 220 of the server 200 may apply correction information about the product recognition result. Further training data for further training the product recognition model may be generated based on this.

The processor 220 of the server 200 may retrain the product recognition model based on the additional learning data received from the computing device 100.

The additional learning data may be learning data using a product image or a portion corresponding to a bounding box for a product included in the product image as input of the training data, and product information as a label of the product image. The server 200 may generate additional learning data sets including additional learning data and existing learning data. The server 200 may retrain the image identification model based on the retraining data. The server 200 may retrain the product recognition model whenever the additional training data is generated, or may retrain the product recognition model whenever the additional training data is greater than or equal to a predetermined number. As the product recognition model is used in the store, the accuracy of the product recognition model can be improved by retraining the product recognition model with additional learning data generated based on the result of the store clerk directly correcting the error.

According to an embodiment of the present disclosure, the second screen may be triggered based on the weight identification of the product.

The computing device 100 may further include a load cell (not shown) or may transmit / receive data with the load cell through the network unit 110.

The computing device 100 may detect the weight of the product. The computing device 100 may detect the weight of the product using the load cell. The computing device 100 may display the second screen when a weight equal to or greater than a predetermined value for the product is detected for more than a predetermined time. In other words, if the number of products is significantly smaller (for example, one product), the cashier of the store makes payments directly, so that the consumption of computing power can be reduced, and if the number of products is large, the payment time is shortened. Product recognition using the product recognition model is started so that the second screen 400 can be displayed without a separate user input. For example, if a user puts more than 800g of product on the tray for more than 5 seconds, and there is no change in weight for 5 seconds, the computing device 100 recognizes that the user puts all the products on the tray for the product to be paid. By displaying the second screen 400 and taking a product image of the product to transmit the product image to the server 200. The detailed description of the method of triggering the second screen based on the weight identification of the above-described product is merely an example and the present disclosure is not limited thereto.

The second screen 400 may include a third object 440 for triggering an operation for displaying payment product information. The third object 440 may be a display object for recognizing an input when there is no error in the product recognition result. For example, when a store clerk determines that the product recognition results 413 and 450 of the second screen 400 are correct and correct, the third object 440 may be selected and input, and the computing device ( When the selection input for the third object 440 is received, 100 determines that the product recognition results 413 and 450 displayed on the second screen 400 are correct, and displays the third screen 500. Can be.

The computing device 100 may display a third screen for displaying payment product information based on the product recognition result. Hereinafter, the third screen will be described with reference to FIG. 4. 4 illustrates an example UI configuration for payment according to an embodiment of the present disclosure. The payment product information 510 may include information about each of two or more products that a user intends to purchase at a store.

The computing device 100, based on receiving the selection input for the third object 440, requests the server 200 to transmit information indicating that the product recognition result is correct or a product tag based on the product recognition result. Information can be sent.

When the server 200 receives from the computing device 100 information that the product recognition result is accurate or request information for transmitting a product tag based on the product recognition result, the server 200 matches one or more product tags and stores each of the one or more products. Based on the product tag for the product recognition result can be identified.

The memory 230 of the server 200 may store product information about each of two or more products by matching the product tag. For example, a product tag that is a barcode for a coffee bun product may be stored in the memory 230, and a product tag that is a barcode for a chicken salad product may be stored in the 'coffee salad' product information. have. Detailed descriptions of the product information and the product tag stored in the above-described memory are only examples, and the present disclosure is not limited thereto.

 The server 200 identifies a product tag for each of two or more product information included in a product recognition result obtained by using a product image as an input of a product recognition model, based on the product information and the product tag stored in the memory 230. You can convert product information into product tags. For example, when the product information acquired by the server 200 using the product recognition model is 'eg tart' and 'night bread', the server 200 matches each of the 'eg tart' and 'night bread' stored in the memory 230. Based on the first product tag, which is a barcode for the egg tart product, and the second product tag, which is a barcode for the bread product, the server 200 can identify the first product tag and the second product tag with respect to the product information. have. The server 200 may convert the product information into the identified product tag. Specific description of the method for converting the above-described product tag is only an example and the present disclosure is not limited thereto.

The server 200 may transmit the identified product tag to the payment computing device.

The computing device 100 may receive information, which is converted into a product tag, for each of one or more products included in the product image, based on the product recognition result.

The computing device 100 may display the payment product information regarding the received product tag by comparing the received product tag with a product tag that is matched with each of one or more products in the memory 130.

The memory 130 of the computing device 100 may store payment product information that matches a product tag for each of two or more products. The settlement product information may include product information (product name) and product value. For example, product information and product value matched with barcodes for each product may be stored in the memory 130. When the payment is performed using the computing device 100 in a store, a barcode of a product is scanned, and product information and product value matching the barcode are acquired using the stored data of the memory 130, and the obtained Payment can be performed based on product information and product value. Accordingly, the memory 130 of the computing device 100 may store a barcode, product information corresponding to the barcode, and a product value. Based on the payment method according to an embodiment of the present disclosure, the computing device 100 transmits a product image to the server 200, the product matching the product recognition result obtained by the server 200 using the product image The tag may be received to identify a product tag stored in the memory 130 and display payment product information 510 on the third screen 500. For example, when the computing device 100 receives the first product tag and the second product tag from the server 200, the coffee is matched and stored in each of the first product tag and the second product tag in the memory 130. The payment product information 510 regarding the number 2,500 won and the mochi mochi cheese stick 1,800 won may be displayed on the third screen 500. Specific description of the method of displaying the above-described payment product information is only an example and the present disclosure is not limited thereto.

The computing device 100 may perform a payment based on the payment product information. The computing device 100 may perform payment through communication with a point server, a card company server, and the like.

The computing device 100 of the present disclosure may be a POS device used in a store. If only the program for recognizing the product is installed in the computing device 100, it may be possible to identify the product based on the product image without having to install or develop a separate interlocking program. Specifically, when the store owner installs only one program for product recognition without having to purchase a device other than the existing POS device, as the clerk executes the program, the image of the product placed on the checkout counter of the store is transmitted to the head office. The head office can transmit the product recognition result to the store. In this case, since the existing POS device is programmed to read a barcode, when the head office transmits a product recognition result to a store, the product recognition result is converted into a barcode in order to give an effect such as reading a barcode through a scanner. Can be sent to the store. That is, even though the store transmits a product image to the head office and receives a product recognition result, the store does not need to develop a separate interface or program because the result received from the head office server is in the form of a barcode.

In general, people who perform calculations in stores often work part-time, and those who perform calculations often change. In the case of a bakery, there are many kinds of bread that have a similar shape, and it is not easy for a newly adopted person to memorize the kind of bread. Therefore, in the case of using the computing device 100 according to the present disclosure, even a person who does not know the type of bread, a product image photographing the product is sent to the main office server to identify the product through the product recognition model of the main office server. Since the information can be received, it is possible to adapt quickly to the task and save time and money.

5 exemplarily illustrates a payment method according to an embodiment of the present disclosure.

The computing device 100 may display 610 a first screen for triggering payment, including a first object for triggering an operation for product recognition. The first screen includes a first layer and a second layer separated from each other, the first layer of the first screen includes information for triggering a payment, and the second layer of the first screen. May include the first object. The first area of the first screen includes information for triggering a payment, the second area of the first screen includes the first object, and the first area and the second area are on one layer. Can be included.

When the selection input for the first object is recognized, the computing device 100 may display 620 a second screen for displaying a product image and a product recognition result based on the product image. The computing device 100 may transmit the product image to the server to obtain a product recognition result based on one or more products included in the product image. The computing device 100 may display the received product recognition result on the second screen. The second screen may include a third object for triggering an operation for displaying payment product information. The third object may be a display object for recognizing an input when there is no error in the product recognition result.

The computing device 100 may display a third screen 630 for displaying payment product information based on the product recognition result. The computing device 100 may transmit information to the server that the product recognition result is correct or request information about transmitting a product tag based on the product recognition result, based on the selection input for the third object. The computing device 100 may receive information, which is converted into a product tag, for each of one or more products included in the product image, based on the product recognition result. The computing device 100 may display the payment product information regarding the received product tag by comparing the received product tag with a product tag that is matched with each of one or more products in a memory.

6 exemplarily illustrates a payment method according to an embodiment of the present disclosure.

The server 200 may receive 710 a product image from a payment computing device.

The server 200 may generate 720 a product recognition result by performing recognition on each of one or more products included in the product image. The server 200 may calculate a product image by using a product recognition model including one or more network functions, and output a scoring value for each of the one or more classification classes. The server 200 may obtain a training data set. The server 200 may output a product recognition result based on the scoring value for each of the one or more classification classes.

The server 200 may transmit a product recognition result obtained through the product recognition model to the computing device.

The server 200 may receive additional learning data from the computing device. The product recognition model may be retrained based on the additional learning data received from the computing device of the server 200. The additional learning data may be learning data using a product image or a portion corresponding to a bounding box for a product included in the product image as input of the training data, and product information as a label of the product image.

The server 200 may identify 730 a product tag for the product recognition result based on one or more product tags that are matched with and stored in each of the one or more products. When the server 200 receives from the computing device 100 information that the product recognition result is accurate or request information for transmitting a product tag based on the product recognition result, the server 200 matches one or more product tags and stores each of the one or more products. Based on the product tag for the product recognition result can be identified. The memory of the server 200 may store product information about each of two or more products by matching the product tag. The server 200 identifies a product tag for each of two or more product information included in a product recognition result obtained by using a product image as an input of a product recognition model, based on the product information and the product tag stored in the memory. Can be converted to a product tag.

The server 200 may transmit 740 the identified product tag to the payment computing device.

7 illustrates an example flowchart of a payment method according to an embodiment of the present disclosure.

The computing device 100 may display a first screen for triggering a payment, including a first object for triggering an operation for product recognition.

The computing device 100 may recognize 801 a selection input for the first object. When the selection input for the first object is recognized, the computing device 100 may display a second screen for displaying a product image and a product recognition result based on the product image. When the selection input for the first object is recognized, the computing device 100 includes an image of a product in a camera unit included in the computing device 100 or a camera that transmits and receives data through the network unit with the computing device 100. Can be taken. The computing device 100 may display the obtained product image on the second screen of the display unit.

The computing device 100 may transmit the product image 803 to the server 200.

The server 200 may generate 805 a product recognition result based on the product image. The server 200 may generate a product recognition result by performing recognition on each of one or more products included in the product image. The server 200 may output a scoring value for each of one or more classification classes by calculating a product image input to an input node of the product recognition model through the product recognition model. The server 200 may output a product recognition result based on the scoring value for each of the one or more classification classes.

The server 200 may transmit 807 the product recognition result to the computing device 100.

The computing device 100 may display a product recognition result 809. The computing device 100 may display a product recognition result for the product included in the product image on the second screen. The second screen may include a third object for triggering an operation for displaying payment product information.

The computing device 100 may recognize 811 the third object selection input. The computing device 100 transmits, to the server 200, the request information regarding the transmission of the product recognition result or the product tag based on the product recognition result, based on the selection input for the third object. Can be.

The computing device 100 may transmit 813 product recognition result confirmation information to the server 200.

The server 200 may identify 815 a product tag for the product recognition result. The memory of the server 200 may store product information about each of two or more products by matching the product tag. The server 200 identifies a product tag for each of two or more product information included in a product recognition result obtained by using a product image as an input of a product recognition model, based on the product information and the product tag stored in the memory. Can be converted to a product tag.

The server 200 may transmit 817 the identified product tag to the computing device 100.

The computing device 100 may display 819 payment product information based on the received product tag.

8 shows a general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

8 shows a brief general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above generally with respect to computer executable instructions that may be executed on one or more computers, those skilled in the art will appreciate that the present disclosure may be implemented in combination with other program modules and / or as a combination of hardware and software. will be.

Generally, program modules include routines, programs, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In addition, those skilled in the art will appreciate that the methods of the present disclosure may include uniprocessor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which And other computer system configurations, including one or more associated devices, which may operate in conjunction with one or more associated devices.

The described embodiments of the present disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer readable media. Any medium accessible by a computer can be a computer readable medium. Computer readable media includes volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and computer readable transmission media. Computer-readable storage media are volatile and nonvolatile media, temporary and non-transitory media, removable and non-removable implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data. Media. Computer storage media may include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROMs, digital video disks or other optical disk storage devices, magnetic cassettes, magnetic tapes, magnetic disk storage devices or other magnetic storage devices, Or any other medium that can be accessed by a computer and used to store desired information.

Computer-readable transmission media typically embody computer readable instructions, data structures, program modules or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and the like. Includes all information delivery media. The term modulated data signal means a signal that has one or more of its characteristics set or changed to encode information in the signal. By way of example, and not limitation, computer readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above should also be included within the scope of computer readable transmission media.

An example environment 1100 is illustrated that implements various aspects of the present disclosure, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 connects system components, including but not limited to system memory 1106, to processing unit 1104. Processing unit 1104 may be any of a variety of commercial processors. Dual processor and other multiprocessor architectures may also be used as the processing unit 1104.

The system bus 1108 may be any of several types of bus structures that may be further interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 1110 and random access memory (RAM) 1112. The basic input / output system (BIOS) is stored in nonvolatile memory 1110, such as ROM, EPROM, EEPROM, and the like, and the BIOS provides a basic aid for transferring information between components in the computer 1102, such as during startup. Contains routines. RAM 1112 may also include fast RAM, such as static RAM, for caching data.

Computer 1102 also includes an internal hard disk drive (HDD) 1114 (eg, EIDE, SATA) —this internal hard disk drive 1114 may also be configured for external use within a suitable chassis (not shown). Yes, magnetic floppy disk drive (FDD) 1116 (eg, for reading from or writing to removable diskette 1118), and optical disk drive 1120 (eg, CD-ROM Disk 1122 for reading from or writing to or reading from other high capacity optical media such as DVD). The hard disk drive 1114, the magnetic disk drive 1116, and the optical disk drive 1120 are connected to the system bus 1108 by the hard disk drive interface 1124, the magnetic disk drive interface 1126, and the optical drive interface 1128, respectively. ) Can be connected. The interface 1124 for external drive implementation includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer readable media provide nonvolatile storage of data, data structures, computer executable instructions, and the like. In the case of computer 1102, drives and media correspond to storing any data in a suitable digital format. Although the above description of computer readable media refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those skilled in the art will appreciate zip drives, magnetic cassettes, flash memory cards, cartridges, and the like. Other types of media readable by the computer, etc. may also be used in the exemplary operating environment and it will be appreciated that any such media may include computer executable instructions for performing the methods of the present disclosure.

Multiple program modules may be stored in the drive and RAM 1112, including operating system 1130, one or more application programs 1132, other program modules 1134, and program data 1136. All or a portion of the operating system, applications, modules and / or data may also be cached in RAM 1112. It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer 1102 via one or more wired / wireless input devices, such as a keyboard 1138 and a mouse 1140. Other input devices (not shown) may include a microphone, IR remote control, joystick, game pad, stylus pen, touch screen, and the like. These and other input devices are often connected to the processing unit 1104 via an input device interface 1142, which is connected to the system bus 1108, but the parallel port, IEEE 1394 serial port, game port, USB port, IR interface, Etc. can be connected by other interfaces.

A monitor 1144 or other type of display device is also connected to the system bus 1108 via an interface such as a video adapter 1146. In addition to the monitor 1144, the computer generally includes other peripheral output devices (not shown) such as speakers, printers, and the like.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer (s) 1148, via wired and / or wireless communications. Remote computer (s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and typically is associated with computer 1102. Although many or all of the components described above are included, for simplicity, only memory storage 1150 is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) 1152 and / or a larger network, such as a telecommunications network (WAN) 1154. Such LAN and WAN networking environments are commonplace in offices and businesses, facilitating enterprise-wide computer networks such as intranets, all of which may be connected to worldwide computer networks, such as the Internet.

When used in a LAN networking environment, the computer 1102 is connected to the local network 1152 via a wired and / or wireless communication network interface or adapter 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed therein for communicating with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158, connect to a communication computing device on the WAN 1154, or establish communications over the WAN 1154, such as over the Internet. Other means. The modem 1158, which may be an internal or external and wired or wireless device, is connected to the system bus 1108 via the serial port interface 1142. In a networked environment, program modules or portions thereof described with respect to computer 1102 may be stored in remote memory / storage device 1150. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers can be used.

Computer 1102 is associated with any wireless device or entity disposed and operating in wireless communication, such as a printer, scanner, desktop and / or portable computer, portable data assistant, communications satellite, wireless detectable tag. Communicate with any equipment or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technology. Thus, the communication can be a predefined structure as in a conventional network or simply an ad hoc communication between at least two devices.

Wireless Fidelity (Wi-Fi) allows you to connect to the Internet without wires. Wi-Fi is a wireless technology such as a cell phone that allows such a device, for example, a computer, to transmit and receive data indoors and outdoors, ie anywhere within the coverage area of a base station. Wi-Fi networks use a wireless technology called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, high-speed wireless connections. Wi-Fi may be used to connect computers to each other, to the Internet, and to a wired network (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in unlicensed 2.4 and 5 GHz wireless bands, for example, at 11 Mbps (802.11a) or 54 Mbps (802.11b) data rates, or in products that include both bands (dual band). have.

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, instructions, information, signals, bits, symbols, and chips that may be referenced in the above description may include voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. Or particles, or any combination thereof.

One of ordinary skill in the art of the disclosure will appreciate that the various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be implemented in electronic hardware, It will be appreciated that for purposes of the present invention, various forms of program or design code, or combinations thereof, may be implemented. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. One skilled in the art of the present disclosure may implement the described functionality in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein may be embodied in a method, apparatus, or article of manufacture using standard programming and / or engineering techniques. The term "article of manufacture" includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media may include magnetic storage devices (eg, hard disks, floppy disks, magnetic strips, etc.), optical discs (eg, CDs, DVDs, etc.), smart cards, and flash memory. Devices, such as, but not limited to, EEPROM, cards, sticks, key drives, and the like. In addition, various storage media presented herein include one or more devices and / or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of exemplary approaches. Based upon design priorities, it is understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, but are not meant to be limited to the specific order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those of ordinary skill in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Thus, the present disclosure should not be limited to the embodiments presented herein but should be construed in the broadest scope consistent with the principles and novel features presented herein.

Claims (13)

A computer program stored in a computer readable storage medium, the computer program configured to perform the following operations for product payment when executed on one or more processors of a computing device,
Displaying a first screen for triggering payment comprising a first layer comprising a first object for triggering an action for product recognition and a second layer comprising a second object for selecting a payment product ;
Displaying a product image for performing a first payment method and a second screen for displaying a product recognition result based on the product image when the selection input to the first object is recognized;
Generating payment product information based on a second payment method when a selection input to the second object is recognized; And
Displaying a third screen for displaying the product recognition result based on the first payment method or the payment product information based on the second payment method;
Including,
The first layer and the second layer is a layer of a form separated from each other,
Computer program stored on a computer readable storage medium.
delete The method of claim 1,
The first area of the first screen includes information for triggering a payment, the second area of the first screen includes the first object, and the first area and the second area are one layer Included in the award,
Computer program stored on a computer readable storage medium.
The method of claim 1,
When the selection input to the second object is recognized, generating the payment product information based on the second payment method,
Recognizing a selection input for at least one second object of two or more second objects for selecting a payment product included in the first screen; And
Generating the payment product information based on the selection input to the at least one second object;
Including,
Computer program stored on a computer readable storage medium.
The method of claim 1,
The product recognition result includes a recognition result for each of one or more products included in the product image obtained by calculating the product image using a product recognition model including a pre-learned network function, and the product The recognition result is generated by the computing device or an external device,
Computer program stored on a computer readable storage medium.
The method of claim 5,
The product recognition model,
Perform an operation by inputting the product image, output a scoring value for each of one or more classification classes for each of the one or more products included in the product image, and maximize the scoring values for each of the one or more products. A model that outputs the product recognition result for the one or more products based on a classification class having a scoring value,
Computer program stored on a computer readable storage medium.
The method of claim 1,
The second screen includes a third object for triggering an operation for displaying payment product information, and
The operation of displaying a third screen for displaying the product recognition result based on the first payment method or the payment product information based on the second payment method may include:
If the selection input to the third object is recognized, determining that the product recognition result is an accurate result and displaying the third screen;
Including,
Computer program stored on a computer readable storage medium.
The method of claim 1,
If there is an error in the product recognition result displayed on the second screen, generating additional learning data for further learning a product recognition model based on receiving correction information about the product recognition result;
Including more;
Computer program stored on a computer readable storage medium.
The method of claim 1,
The operation of displaying a third screen for displaying the product recognition result based on the first payment method or the payment product information based on the second payment method may include:
Receiving information, which is converted into a product tag, for each of one or more products included in the product image, based on the product recognition result; And
Displaying the payment product information about the received product tag by comparing the received product tag with a product tag that is matched with each of one or more products in a memory;
Including,
Computer program stored on a computer readable storage medium.
The method of claim 1,
Sensing the weight of the product; And
Displaying the second screen when the weight of more than a predetermined value is sensed for more than a predetermined time;
Including more;
Computer program stored on a computer readable storage medium.
delete In the method for product payment performed on a computing device,
Displaying a first screen for triggering a payment comprising a first layer comprising a first object for triggering an action for product recognition and a second layer comprising a second object for selecting a payment product ;
If a selection input for the first object is recognized, displaying a product image for performing a first payment method and a second screen for displaying a product recognition result based on the product image;
Generating payment product information based on a second payment method when a selection input to the second object is recognized; And
Displaying a third screen for displaying the product recognition result based on the first payment method or the payment product information based on the second payment method;
Including,
The first layer and the second layer is a layer of a form separated from each other,
How to pay for the product.
A computing device for product payment,
A processor including one or more cores; And
Memory;
Including,
The processor,
Display a first screen for triggering a payment, comprising a first layer comprising a first object for triggering an action for product recognition and a second layer comprising a second object for selecting a payment product,
When a selection input for the first object is recognized, a second screen for displaying a product image for performing a first payment method and a product recognition result based on the product image is displayed.
When the selection input to the second object is recognized, the payment product information is generated based on the second payment method.
Displaying a third screen for displaying the product recognition result based on the first payment method or the payment product information based on the second payment method, and
The first layer and the second layer is a layer of a form separated from each other,
Computing device for product payment.

Images