KR102124568B1 - System for payment solution service based on object recognition in image with machine learning using artificial intelligent - Google Patents

Abstract

Provided is an image recognition based unmanned payment system using artificial intelligence and big data. The system comprises: a scanner photographing a product and transmitting the photographed product image; a store terminal recognizing an object included in the product image transmitted from the scanner, and outputting a product identification code pre-mapped to the recognized object and stored; a sales terminal recognizing a product name and a product price using the product identification code outputted from the store terminal and performing payment; and an unmanned payment service providing server, wherein the unmanned payment service providing server comprises a receiving part receiving at least one product image photographed from the scanner through the store terminal, a performing part using at least one received product image to perform artificial intelligence learning for recognizing an object which is a product in the product image, and a provision part providing the learning results to the store terminal such that in the store terminal, the object is recognized in the product image.

Description

Image recognition based unmanned payment system using artificial intelligence and big data{SYSTEM FOR PAYMENT SOLUTION SERVICE BASED ON OBJECT RECOGNITION IN IMAGE WITH MACHINE LEARNING USING ARTIFICIAL INTELLIGENT}

The present invention relates to an unmanned payment system based on image recognition using artificial intelligence and big data, and provides a platform capable of product recognition and payment only by scanning images by training product images collected in various environments with artificial intelligence-based machine learning.

Convenience stores are easily accessible around us as a representative industry these days, as small self-employed people are struggling with protests against the minimum wage and protests are reported in the news. Convenience stores, which appear like stalks in every alley, are forced to share pies in a limited market, and the return on labor is bound to worsen due to rising labor costs. The reason for this problem was that the proportion of the labor cost burdened by hiring part-time jobs while operating for 24 hours increased. As a countermeasure against this, unmanned stores with no burden of labor costs are appearing one after another, and Wal-Mart, which employs a stock AI robot called Amazon Go and Bossa Nova, which uses a sensor to pay for goods, is available online in China. Hema, which adopted Alipay, which is linked to face recognition in the shopping mall Alibaba, is a representative example.

At this time, a method of acquiring information from the human body and operating an unmanned store has been researched and developed. In this regard, in Korean Patent Publication No. 2019-0008075 (published on January 23, 2019), a user's body Configuration for acquiring personal information, transmitting personal information to the first server, receiving authentication of personal information from the first server, and opening the gate to allow the user to enter or leave the store when the authentication device receives authentication , A configuration is disclosed in which personal information is obtained from a user's body part, personal information is transmitted to a second server, and payment of a product amount is received when authentication of the personal information is received from the second server.

However, even if all the barcodes are stamped on the product, it is difficult for an unfamiliar user to scan it, and eventually the salesman calls or returns to the scanning method. In addition, if all of the items are not barcoded, modeling and labeling must be carried out on their own. In addition, it is impossible in small stores, and because only the price tag is attached, manpower and time are wasted when settingtle. In addition, even when classifying items by image recognition, it is difficult because object recognition is significantly reduced when images are taken at different angles or lights, and when RFID tags are attached to all items, the recognition rate is good, but all products are RFID tags There is a problem of labor cost and cost increase since it must be attached.

According to an embodiment of the present invention, images acquired under various angles and lighting environments are trained using artificial intelligence algorithms to increase recognition of objects, and data that has undergone learning is stored in each store so that objects can be recognized through image scanning. By synchronizing with the store terminal, it is possible to recognize when an item is scanned in the store, and if the image is not recognized, the product receives an identification code of the product from the sales terminal and re-learns the image, so that it is robust to various environments including noise. An unmanned payment system based on image recognition using artificial intelligence and big data, which enables recognition and unmanned payment, can be provided. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention, a product for photographing, and a scanner for transmitting the photographed product image, recognizes an object included in the product image transmitted from the scanner, and is recognized A store terminal that outputs the product identification code stored by being mapped to an object, a sales terminal that recognizes the product name and product price with the product identification code output from the store terminal, and performs payment, and stores at least one product image taken from the scanner A receiving unit that receives via a terminal, a progressing unit that performs artificial intelligence learning for recognizing an object that is a product in a product image using at least one received product image, and provides a result of the learning to a store terminal to store products at the store terminal And an unmanned payment service providing server including a providing unit for recognizing an object in the image.

According to any one of the above-described problem solving means of the present invention, images acquired under various angles and lighting environments are learned with an artificial intelligence algorithm to recognize objects through image scanning, thereby increasing object recognition and learning. Synchronize the progressed data with the store terminal of each store so that it can be recognized when the goods are scanned in the store. If the image is not recognized, receive the product identification code from the sales terminal and re-learn the image to reduce noise. It enables strong image recognition and unattended payment in various environments, including automated sales of fresh foods that cannot attach 2D or 3D tags, including barcodes or QR codes, and reduces customer line-up time. Payment is possible without spending resources including original education and learning, and it can accommodate elderly people who are vulnerable to IT devices or even minors who do not have a mobile payment system.

1 is a view for explaining an unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment of the present invention.
FIG. 2 is a block diagram for explaining an unattended payment service providing server included in the system of FIG. 1.
3 is a diagram for explaining an embodiment in which an unmanned payment service based on artificial intelligence image object recognition according to an embodiment of the present invention is implemented.
4 is a diagram illustrating a process in which data is transmitted and received between each component included in the image recognition-based unmanned payment system using artificial intelligence and big data of FIG. 1 according to an embodiment of the present invention.
5 is an operation flowchart for explaining an unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present invention pertains may easily practice. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Throughout the specification, when a part is "connected" to another part, this includes not only "directly connected" but also "electrically connected" with another element in between. . Also, when a part is said to "include" a certain component, it means that the component may further include other components, not exclude other components, unless specifically stated otherwise. However, it should be understood that the existence or addition possibilities of numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

The terms "about", "substantially", and the like used throughout the specification are used in or at a value close to the value when manufacturing and substance tolerances unique to the stated meaning are given, and the understanding of the present invention. To aid, accurate or absolute figures are used to prevent unscrupulous use of the disclosed disclosure by unscrupulous infringers. The term "~(step)" or "step of" as used in the entire specification of the present invention does not mean "step for".

In the present specification, the term “unit” includes a unit realized by hardware, a unit realized by software, and a unit realized by using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

In the present specification, some of the operations or functions described as performed by the terminal, the device, or the device may be performed instead on a server connected to the corresponding terminal, device, or device. Similarly, some of the operations or functions described as being performed by the server may be performed in a terminal, apparatus, or device connected to the corresponding server.

In this specification, some of the operations or functions described as mapping or matching with the terminal means that the unique number of the terminal or identification information of the individual, which is identification data of the terminal, is mapped or matched. Can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a view for explaining an unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment of the present invention. 1, the image recognition-based unmanned payment system 1 using artificial intelligence and big data includes at least one store terminal 100, an unmanned payment service providing server 300, and at least one scanner 400, It may include at least one sales terminal 500. However, the unmanned payment system 1 based on image recognition using artificial intelligence and big data of FIG. 1 is only an embodiment of the present invention, and thus the present invention is not limitedly interpreted through FIG. 1.

At this time, each component of FIG. 1 is generally connected through a network (network, 200). For example, as illustrated in FIG. 1, at least one store terminal 100 may be connected to the server 300 for providing an unmanned payment service through the network 200. In addition, the unmanned payment service providing server 300 may be connected to at least one store terminal 100, at least one scanner 400, and at least one sales terminal 500 through the network 200. Also, the at least one scanner 400 may be connected to the store terminal 100 and the unmanned payment service providing server 300 through the network 200. Also, the at least one sales terminal 500 may be connected to the at least one store terminal 100 and the unmanned payment service providing server 300 through the network 200.

Here, the network means a connection structure capable of exchanging information between each node such as a plurality of terminals and servers, and examples of such a network include RF, 3GPP (3rd Generation Partnership Project) network, and LTE (Long Term) Evolution (Evolution) network, 5GPP (5th Generation Partnership Project) network, WIMAX (World Interoperability for Microwave Access) network, Internet (Internet), Local Area Network (LAN), Wireless Local Area Network (LAN), Wide Area Network (WAN) , PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, and the like.

In the following, the term at least one is defined as a term including singular and plural, and even if the term at least one does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, it may be said that each component is provided in a singular or plural form, depending on the embodiment.

The at least one store terminal 100 identifies an object that is an object in a product image using a web page, an app page, a program, or an application related to unmanned payment service based on artificial intelligence image object recognition and sells identification information to the sales terminal 500 It may be a terminal that is sent to the payment and settlement. To this end, the at least one store terminal 100 may be a terminal that models an object in a product image and performs labeling to identify it. Further, the at least one store terminal 100 may be a terminal that can recognize a product in synchronization with a result of learning and re-learning from the unmanned payment service providing server 300. In addition, the at least one store terminal 100 is connected to the scanner 400 and the sales terminal 500 to identify a product image scanned or photographed by the scanner 400 and transmits an identification code to the sales terminal 500 It may be a terminal that enables the product to be paid by the sales terminal 500. In addition, the at least one store terminal 100 may be a terminal that may be deleted or integrated with the sale terminal 500 when connected to the cloud with the unattended payment service providing server 300. At this time, the sales terminal 500, the function and operation of the store terminal 100 SaaS (Software as a Service), PaaS (Platform as a Service), IaaS (Infrastructure as a Service) any one or a combination of at least one It may be a terminal implemented by. The reverse is also true (Vice Versa).

Here, the at least one store terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser (WEB Browser), a desktop, a laptop, and the like. At this time, the at least one store terminal 100 may be implemented as a terminal that can access a remote server or terminal through a network. The at least one store terminal 100 is, for example, a wireless communication device in which portability and mobility are guaranteed, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet ) It may include all kinds of handheld-based wireless communication devices such as a terminal, a smartphone, a smartpad, and a tablet PC.

The unmanned payment service providing server 300 may be a server that provides an artificial intelligence image object recognition-based unmanned payment service web page, app page, program, or application. In addition, the unmanned payment service providing server 300 may be a server that collects at least one product image received from the scanner 400 via the store terminal 100. Also, the unmanned payment service providing server 300 may be a server that performs artificial intelligence-based machine learning to perform object recognition robust to noise or environmental changes when the collected product images are collected in various environments. In addition, the unmanned payment service providing server 300 may be a server that automatically identifies the product scanned by the scanner 400 in the store terminal 100 by sharing the learned result with the store terminal 100. In addition, when the unmanned payment service providing server 300 learns product images received from the plurality of store terminals 100 when the store terminals 100 are plural, when the store terminal 100 does not recognize, It may be a server that receives an unrecognized product image from the store terminal 100 and inputs an identification code of the product, thereby identifying the product and inputting it as an image of the identified product for re-learning and training. In addition, the unmanned payment service providing server 300 may be a server that allows the sales terminal 500 to communicate with a user terminal (not shown) using an O2O service (Online to Offline), and enables simple payment or automatic payment. have. And, the unattended payment service providing server 300, when the store terminal 100 is integrated into the sale terminal 500 in the cloud, the store terminal 100 from the sale terminal 500 using IaaS, PaaS, SaaS, etc. It may be a server that allows the function of.

Here, the unmanned payment service providing server 300 may be implemented as a computer that can access a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser (WEB Browser), a desktop, a laptop, and the like.

The at least one scanner 400 may be a device that photographs or scans an object using an artificial intelligence image object recognition-based unmanned payment service related web page, app page, program, or application. At this time, the at least one scanner 400 may be a device capable of weighing by further including a weight sensor on the plate on the lower surface, and store information such as a photographed or scanned image and weight in the store terminal 100 It may be a device transmitting to.

The at least one sales terminal 500 may be a terminal that performs a payment process and setstles using an artificial intelligence image object recognition-based unmanned payment service-related web page, app page, program, or application, for example, POS (Point of Sale). And, at least one sales terminal 500, when the scanner 400 returns the identification code of the product after the photographing is completed, the store terminal 100 receives this, and when the payment method is input, the product is sold and processed. It may be a terminal that reduces inventory quantity and issues receipts. Here, when the at least one sales terminal 500 performs payment with O2O from the user terminal, receives a payment event from the unattended payment service providing server 300 via the store terminal 100 and processes payment completion. It may be a terminal.

Here, the at least one sales terminal 500 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, navigation, a laptop equipped with a web browser (WEB Browser), a desktop, a laptop, and the like. At this time, the at least one sales terminal 500 may be implemented as a terminal capable of accessing a remote server or terminal through a network. The at least one sales terminal 500 is, for example, a wireless communication device in which portability and mobility are guaranteed, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), Personal Handyphone System (PHS), Personal Digital Assistant (PDA), International Mobile Telecommunication (IMT)-2000, Code Division Multiple Access (CDMA)-2000, W-Code Division Multiple Access (W-CDMA), Wireless Broadband Internet ) It may include all kinds of handheld-based wireless communication devices such as a terminal, a smartphone, a smartpad, and a tablet PC.

FIG. 2 is a block diagram illustrating an unmanned payment service providing server included in the system of FIG. 1, and FIG. 3 is an embodiment of an artificial intelligence image object recognition-based unmanned payment service according to an embodiment of the present invention It is a drawing for explaining.

Referring to FIG. 2, the unmanned payment service providing server 300 may include a receiving unit 310, a progressing unit 320, a providing unit 330, and a requesting unit 340.

Unattended payment service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one store terminal 100, at least one scanner 400, and at least one sale When transmitting the artificial intelligence image object recognition-based unmanned payment service application, program, app page, web page, etc. to the terminal 500, at least one store terminal 100, at least one scanner 400, and at least one The sales terminal 500 may install or open an artificial intelligence image object recognition-based unmanned payment service application, program, app page, web page, or the like. In addition, the service program may be driven in at least one store terminal 100, at least one scanner 400, and at least one sales terminal 500 using a script executed in a web browser. Here, the web browser is a program that enables the use of the world wide web (WWW) service, and refers to a program that receives and displays hypertext described in HTML (hyper text mark-up language), for example, Explorer , Chrome, and the like. Further, the application means an application on the terminal, and includes, for example, an app that is executed on a mobile terminal (smartphone).

Referring to FIG. 2, the receiver 310 may receive at least one product image photographed from the scanner 400 via the store terminal 100. At this time, the scanner 400 may photograph a product and transmit the photographed product image, but does not exclude scanning a barcode or a QR code. It also does not exclude scanning and recognizing images and barcodes or QR codes together. In addition, when the scanner 400 includes a weight sensor or a piezoelectric sensor, the weight sensed together with the product image is transmitted, and the receiver 310 may receive the image and data together.

The progress unit 320 may perform artificial intelligence learning for recognizing an object that is a product in a product image using at least one received product image. At this time, if the article is recognized using the shape of the article, for example, color or shape itself, the need to find the barcode disappears and there is no need to worry about the barcode damage or recognition problem. In addition, since the need to hold the scanner is eliminated, the problem of separating the scanning process from the packaging process can also be solved. For image recognition, it can be largely divided into a training step and an application step using an artificial intelligence algorithm, and the former explains the former. First, the progress unit 320 must collect data. In order to recognize images, first, a data set necessary for a deep learning model training process must be prepared. For this, it is necessary to select products to be applied to the system and collect data. For example, canned beverages, cigarettes, ramen, tissues, etc. can be used as representative products sold at convenience stores, but are not limited to those listed. Collection methods may include crawling or direct collection or shooting, and data screening can improve the reliability of the dataset. Secondly, the progress unit 320 performs data set alignment. In order to make it into a dataset, it is necessary to undergo a resize operation that makes the size of different photos constant. The reason is that if you resize photos with different aspect ratios in bulk, distortion may occur, which is low during training. Since it is a cause of accuracy, it can be resized after a square cropping operation. After that, the dataset is expanded by rotating and inverting the image.

Third, the progress unit 320 performs data set production. In one picture, each pixel is divided into RGB channels, and after all the pixels are separated, label information can be added. In this way, we work on all the photos to produce a dataset. Fourth, the progress unit 320 may perform deep learning model (Mobilenet) code conversion. If you run deep learning model implemented in Python in Raspberry Pi as it is, it is very slow, so there is a delay and it is difficult to meet the system goal. Therefore, it is necessary to convert Python code to C code to ensure the speed of the system. The training person uses the Python code to extract only the weight value from the high-performance PC and converts the part that determines the type of the product to implement it in Raspberry Pi. Fifth, the progress unit 320 performs deep learning model training. The dataset is trained in a deep learning model to generate weight values. At this time, in the code implemented in Python, the weight value is formed as a pickle file. Since the code converted to C takes a long time to load the pickle file, the weight value can be converted into a text file and saved to ensure speed. Of course, in addition to the above-described method, the image can be learned and trained in various ways, and it is obvious that variations and modifications according to various embodiments are not limited to those listed.

Here, the progress unit 320, when performing artificial intelligence learning, supervised learning (Supervised Learning), unsupervised learning (Unsupervised Learning), semi-supervised learning (Semi-Supervised Learning), and reinforcement learning (Reinforcement Learning) Learning may be conducted in one or at least one combination.

The providing unit 330 provides the result of learning to the store terminal 100 so that the store terminal 100 recognizes an object in the product image. Accordingly, the store terminal 100 may recognize an object included in the product image transmitted from the scanner 400, and may output a product identification code that is pre-mapped and stored in the recognized object, and the sales terminal 500 may , Recognizing the product name and product price with the product identification code output from the store terminal 100 and performing payment. At this time, the providing unit 330 proceeds to the latter, that is, the application step, which has not been described in the proceeding unit 320. When the article is recognized, the scanner 400 may automatically recognize the article. For example, two ultrasonic sensors connected to an Arduino can be activated by recognizing the existence of an item and transmitting a signal through serial communication with a Raspberry Pi. Then, the providing unit 330, the Raspberry Pi receives a signal from the Arduino that there is a bite, can capture the object recognized by the attached scanner 400, crop and resize it, and convert it into a text file and save it. Then, the providing unit 330 allows image recognition and data transmission through the store terminal 100. Raspberry Pi loads the weight into the deep learning model to determine what the recognized product is and the price of the product in the database. Loads detailed information such as a name, adds it to the bill, and outputs the bill from the sales terminal 500. Of course, it will be apparent that various methods are not limited to the above-described method.

The requesting unit 340, when the object in the product image received from the scanner 400 via the store terminal 100 cannot be recognized, the product identification code to the sales terminal 500 via the store terminal 100 You can ask to enter. At this time, when the product identification code is input, the product image that could not be recognized may be transmitted as input data for re-learning artificial intelligence. Here, a convolutional neural network, which is a CNN, can be used for re-learning. The convolutional neural network is composed of a convolutional layer that extracts the features of the input, a fully connected layer that infers the results with the extracted features, and a pooling layer that reduces the size of the feature map of the convolutional layer. Can be configured. In the convolution layer, the weight filter matrix and the input matrix are convolution operations. That is, after multiplying the filter matrix and the corresponding input, the result of adding the value (Partial Sum) may be sequentially stored in the output matrix. Because the convolutional layer is an iterative multiplication and addition operation, most of the computation time of the convolutional neural network is consumed in the convolutional layer. The pooling layer can reduce the size of the output matrix generated by the convolution layer, and the reduced output matrix can reduce the over-fitting problem. The method of reducing the output matrix consists of a method in which a weight filter having a size of 2×2 or 3×3 is moved by 2 or 3, and the largest value among the values of the input matrix crossing the weight filter matrix is stored in the output matrix. The fully connected layer exists at the end of the neural network, and the results obtained by synthesizing features obtained through the convolution layer are inferred. The size of the weight filter matrix is determined by the size of the input and output matrices.

Here, in an embodiment of the present invention, in the process of re-learning the image data that has been completely learned, it may be configured to reduce unnecessary weight and connection by applying a pruning process. That is, the size of the weight filter matrix can be reduced by a low-order approximation to the learned weight filter matrix, which is applied to the fully connected layer and not to the convolution layer. This is because the accuracy of the neural network is greatly reduced when pruning is applied to the convolution layer. Accordingly, in an embodiment of the present invention, a method of applying pruning after applying a low order approximation to a convolution layer first, and then converging a number of weights with a small number of weights may be used. As a result, the size of the neural network is reduced, reducing the storage and computation of weights, and the loss of accuracy can be minimized.

To this end, the acceleration algorithm, first, applies a low-order approximation to all the layers of the neural network where learning is completed, and the fully coupled layer can apply pruning 90% of the total weight from the value where the absolute value of the weight is close to 0. have. Next, to examine the scarcity of each layer, the reference value may be a value in which the magnitude of the absolute value of the weight in the first convolution layer becomes the lower x%. The initial value of x may be changed according to the memory size of the terminal or server to be retrained. Values below the determined reference value are regarded as 0, and scarcity may mean a ratio occupied by 0 of the weights of the layer. Pruning can be applied from the selected convolutional layer to pruning based on the scarcity of each layer examined by the reference value. Next, re-learning can be performed to restore Top-1 accuracy loss by low order approximation and pruning techniques.

The second low-order approximation is performed, and the low-order approximation is a method of improving the computation speed by reducing the size of the neural network without significant loss of Top-1 accuracy. The weight filter matrix of the convolution layer is small, but there are multiple weight filter matrices. Therefore, since many operations occur in the multi-layer convolutional layer, and the fully connected layer is concentrated in weight, affecting the overall size of the neural network, the size of the neural network can be reduced by using a low-order approximation technique for each of these layers. have. The low-order approximation can reduce the size of the matrix by decomposing the matrix through singular value decomposition and finding a small order (Rank) that is less different from the matrix.

Thirdly, the pruning technique is used. As described above, the pruning technique is a method of removing insignificant weights from the learned convolutional neural network. In the pruning technique, when the weight ratio to be left in each layer is determined, the weight closest to 0 can be removed. Since each layer has a different effect on Top-1 accuracy, if the pruning techniques are applied at the same weight ratio, the Top-1 accuracy may be significantly lowered. Therefore, in one embodiment of the present invention, after lower order approximation, the weights are pruned at different ratios so that the influence on Top-1 accuracy is small for each layer. The reference value for applying pruning may be set to a value such that the weight scarcity of the first convolution layer is 10%, but is not limited thereto.

The reason for setting the reference value based on the scarcity of the first convolution layer is that the first convolution layer recognizes the edge of the input image, so the weight sparse is the lowest among the convolution layers. When the operation is performed through the continuous upper convolution layer, the weights of the boundary portion are more concentrated with a small number of weights. In the case of the reference value in which the weight constancy of the first convolutional layer is 20%, since the full coupling layer recognizes most of the weights as close to 0, applying the pruning technique can significantly reduce the Top-1 accuracy. As a result of repeated experiments, the scarcity of the first convolution layer can be set as a reference value within 10%, but this is also not a fixed value. Next, it is possible to select a convolutional layer to perform pruning according to the memory size of the system. If the size of the memory is small, it is desirable to perform pruning from a low convolutional layer, but the Top-1 accuracy is reduced and the time for re-learning can be increased. Conversely, if the memory size is large, it is better to perform pruning from a high convolutional layer, but the top-1 accuracy is increased, and the time for re-learning can be shortened.

When re-learning, pruning in convolutional neural network-based deep learning can be performed within limited computing resources or networking resources, or within a limited time, so it can be shorter than the initial learning time and much less than the initial resources Just re-learning becomes possible. Since the acceleration algorithm of the present invention can efficiently reduce the size of a neural network by applying a low-order approximation and a weight pruning method, the required memory is reduced, the amount of computation is reduced, the speed of inference is improved, and the time for re-learning is greatly increased. It can be reduced to provide AI services faster and faster. Of course, the method of performing re-learning is not limited to the above-described method.

In addition, the store terminal 100 may perform a modeling and labeling process to recognize an object that is a product within at least one product image of the product. In addition, when a plurality of stores exist, the store terminal 100 includes a plurality of store terminals 100 installed in a plurality of stores, and the unmanned payment service providing server 300 includes a plurality of store terminals 100 And a cloud platform, and at least one product image recognized by the plurality of store terminals 100 may be collected as learning data. At this time, any one or a combination of IaaS, SaaS, and PaaS is available as described above.

Also, the unmanned payment service providing server 300 recognizes an object in the product image using any one or a combination of Hough Transformation, Pattern Recognition, and Template Matching. You may. Alternatively, the unattended payment service providing server 300 processes the objects in the product image with Open Source Computer Vision (OpenCV) to binarize the objects, and groups the adjacent regions in the pixel unit by using the labeling of the image made of pixels. Formation, the point where the value of the pixel exceeds a predetermined allowable threshold value, performs binarization, extracts and recognizes color through binarization labeling, dilates and expands (OCR) optical character recognition Objects may be recognized using but are not limited thereto.

Hereinafter, the operation process according to the configuration of the unmanned payment service providing server of FIG. 2 will be described in detail with reference to FIG. 3 as an example. However, the embodiment is only one of various embodiments of the present invention, it will be apparent that it is not limited thereto.

Referring to FIG. 3, (a) when the unattended payment service providing server 300 is photographed through at least one scanner 400 and collected through the store terminal 100, (b) from at least one product image The object and the background are separated, and the extracted object starts learning and training with input data. Then, the unmanned payment service providing server 300 synchronizes data so that learning contents can be transmitted to the store terminal 100, and (c) in the application step, when an object is detected and photographed in the scanner 400, the store terminal 100 ) To identify the product and make payment through the sales terminal 500. And, (d) O2O service (Offline to Online) can be used to facilitate payment, and in the case of a minor, payment can be completed by sending a payment request to the parent terminal or a legal representative terminal. The payment using the credit card in the sales terminal 500 is the same as the known technology, and thus detailed description is omitted.

In addition, the unattended payment service providing server 300 may be configured to automatically scan the scanned products when they pass through an opening and closing device such as a breaker, or may be configured to not pass through the opening and closing device when payment is not made. However, for this, the opening/closing device may be configured with an RFID reader, and each product may further include an RFID tag.

2 and 3, the image recognition-based unmanned payment system using artificial intelligence and big data is not described above. The image recognition-based unmanned payment system using artificial intelligence and big data was previously described with reference to FIG. 1. Since the contents can be easily inferred from the same or described contents, the following description will be omitted.

4 is a diagram illustrating a process in which data is transmitted and received between each component included in the image recognition-based unmanned payment system using artificial intelligence and big data of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process in which data is transmitted and received between each of the components will be described with reference to FIG. 4, but the present application is not limited to such an embodiment, and is illustrated in FIG. 4 according to various embodiments described above. It is apparent to those skilled in the art that the process of transmitting and receiving data can be changed.

4, when a product image is received from the scanner 400 to the store terminal 100 (S4100), the store terminal 100 models and labels the product image (S4200), and the product identification code is labeled. If it is granted, the identification code and the product image, if the identification code is not provided, transmits only the product image to the unattended payment service providing server 300 so that the unattended payment service providing server 300 attaches (S4300).

Then, when the unmanned payment service providing server 300 collects the product image (S4400), the background is separated to extract an object (S4410), and the extracted object is trained for image recognition (S4430), and the learned The product identification code is mapped to the object, or the product identification code is generated and assigned and mapped (S4450). At this time, image collection, background separation, identification code generation and granting are of course made in advance when building a database, but it will be described on the assumption that re-learning is performed using the results of testing in the store or the actual scan results. In addition, the unmanned payment service providing server 300 transmits the learned result to the store terminal 100 to synchronize so that the store terminal 100 can recognize each product (S4500).

On the other hand, when the payment is started from the sales terminal 500 (S4600), the store terminal 100, in conjunction with the scanner 400 (S4700) to take a product image, or the scanner 400, such as ultrasonic sensors Upon recognizing this location, a product image is photographed (S4710). Then, when the product image received from the scanner 400 is recognizable (S4750), the store terminal 100 transmits the product identification code to the sales terminal 500 (S4900), and proceeds with the payment process (S4920). , If recognition is not possible (S4750), requests the input of the product identification code to the sales terminal 500 (S4900), and re-learns the received product identification code and the product image that could not be recognized as input values. At this time, the re-learned result may be shared in real time with the store terminal 100.

The order between the above-described steps (S4100 to S4920) is only an example, and is not limited thereto. That is, the order between the above-described steps (S4100 ~ S4920) may be mutually variable, some of which may be executed or deleted simultaneously.

The above description of the unmanned payment system based on image recognition using artificial intelligence and big data of FIG. 4 is described above with reference to the image recognition based unmanned payment system using artificial intelligence and big data through FIGS. 1 to 3. Since the contents can be easily inferred from the same or described contents, the following description will be omitted.

5 is an operation flowchart for explaining an unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment of the present invention. 5, the unmanned payment service providing server receives at least one product image photographed from a scanner via a store terminal (S5100).

Then, using the unmanned payment service providing server, the received at least one product image to perform artificial intelligence learning for recognizing an object that is a product in the product image (S5200).

In addition, the unmanned payment service providing server recognizes the object in the product image at the store terminal by providing the result of learning to the store terminal (S5300).

The above description of the unmanned payment system based on image recognition using artificial intelligence and big data in FIG. 5 is described above with reference to the image recognition based unmanned payment system using artificial intelligence and big data through FIGS. 1 to 4. Since the contents can be easily inferred from the same or described contents, the following description will be omitted.

An unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment described with reference to FIG. 5 is in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer Can also be implemented as Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, the computer-readable medium may include any computer storage medium. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The unmanned payment system based on image recognition using artificial intelligence and big data according to an embodiment of the present invention described above may include an application installed in a terminal (which may include a program included in a platform or an operating system basically mounted in the terminal). It may be executed by an application (i.e., a program) installed directly on the master terminal through an application providing server such as an application store server, an application, or a web server related to a corresponding service. In this sense, the image recognition-based unmanned payment system using artificial intelligence and big data according to an embodiment of the present invention described above is implemented as an application (that is, a program) basically installed in a terminal or directly installed by a user, and is provided in the terminal. It may be recorded on a computer readable recording medium.

The above description of the present invention is for illustration only, and those skilled in the art to which the present invention pertains can understand that the present invention can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and it should be interpreted that all changes or modified forms derived from the meaning and scope of the claims and equivalent concepts thereof are included in the scope of the present invention. do.

Claims (7)

A scanner that photographs a product and transmits the photographed product image;
A store terminal recognizing an object included in the product image transmitted from the scanner, and outputting a product identification code stored in a mapping to the recognized object;
A sales terminal that recognizes a product name and a product price with a product identification code output from the store terminal and performs payment; And
A receiving unit for receiving at least one product image photographed from the scanner via the store terminal, and a progress unit for performing artificial intelligence learning to recognize an object as a product in the product image using the received at least one product image , An unmanned payment service providing server including a provision unit that provides the result of the learning to the store terminal so that the store terminal recognizes an object in a product image.
The unmanned payment service providing server, when the object in the product image received from the scanner via the store terminal cannot be recognized, requests a request unit to input a product identification code to the sales terminal via the store terminal In addition, when the product identification code is input, the unrecognized product image is transmitted as input data for AI re-learning,
The server for providing an unmanned payment service recognizes an object in the product image using any one or a combination of Hough Transformation, Pattern Recognition, and Template Matching,
The unmanned payment service providing server processes an object in the product image with Open Source Computer Vision (OpenCV) to binarize the object, and forms a group of adjacent regions in a pixel unit using labeling of an image made of pixels, The point where the value of the pixel exceeds a preset allowable threshold value performs binarization processing, extracts and recognizes color through binarization labeling, and uses dilation and expansion of optical character recognition (OCR). Unmanned payment system based on image recognition using artificial intelligence and big data to recognize objects.
According to claim 1,
The store terminal,
An unmanned payment system based on image recognition using artificial intelligence and big data, characterized in that modeling and labeling are performed to recognize the object that is the product within at least one product image of the product.
According to claim 1,
When a plurality of stores exist, the store terminal includes a plurality of store terminals installed in a plurality of stores,
The unmanned payment service providing server is connected to the plurality of store terminals and a cloud platform, and collects at least one product image recognized by the plurality of store terminals as learning data. Unmanned payment system based on image recognition.
delete delete delete According to claim 1,
The progress unit,
When performing the artificial intelligence learning, learning by any one or a combination of at least one of supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning Unmanned payment system based on image recognition using artificial intelligence and big data, characterized by proceeding.

Images