KR102323796B1 - Smart shopping cart, shopping system using the smart shopping cart and method for purchasing product using the smart shopping cart - Google Patents

Abstract

The present invention relates to a smart shopping cart, a shopping system using the smart shopping cart, and a method for purchasing a product using the smart shopping cart. The shopping cart in accordance with one embodiment of the present invention comprises: a cart for receiving products; a communication unit for communicating with a server; a sensor unit for detecting whether a product is put in the cart; a camera installed in the cart and capturing an image of a product put in the cart; a controller transmitting the image to the server, which recognizes the product using an artificial neural network, through the communication unit and acquiring information about the product put in the cart using a product recognition result on the image received from the server; a display unit for displaying the information on the product put in the cart; and a payment unit for performing payment for the product put in the cart based on the information about the product, wherein the controller obtains the image of the product put in the cart by outputting an operation signal for operating the camera when the sensor unit detects the product put in the cart.

Description

A smart shopping cart, a shopping system using the smart shopping cart, and a product purchase method using the smart shopping cart

The present application relates to a smart shopping cart, a shopping system using the smart shopping cart, and a product purchase method using the smart shopping cart. , a shopping system using the smart shopping cart, and a product purchase method using the smart shopping cart.

Artificial intelligence, a method that imitates human thinking, can be applied virtually infinitely to all industries, and as data traffic increases significantly with the development of computer technology, it has become a major trend leading future innovation.

Such artificial intelligence technology is being introduced on a trial basis in the field of shopping services such as self-driving carts, discount item recommendation, and product location guidance. Although a technology has been developed in which a shopping service system having an image sensor recognizes a product using artificial intelligence technology and uses it for payment, there is a problem in that the recognition rate of the product is low, which causes an erroneous payment. The technology for solving these problems is still insufficiently developed.

Therefore, there is a need to develop a technology for a technology that helps a user to accurately purchase a product by improving the product recognition performance.

An object to be solved is to provide a smart shopping cart for recognizing products using artificial intelligence, a shopping system using the smart shopping cart, and a product purchase method using the smart shopping cart.

Another object to be solved is to provide a smart shopping cart that provides a location-based service using an RFID reader installed in the cart, a shopping system using the smart shopping cart, and a product purchase method using the smart shopping cart.

The problem to be solved of the present application is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to an aspect of the present invention, as a smart shopping cart, a cart for placing a product, a communication unit for communicating with a server, a sensor unit for detecting whether a product is contained in the cart, is installed in the cart, and added to the cart A camera for photographing a product to obtain an image, and transmitting the image to the server that recognizes a product using an artificial neural network through the communication unit, and using the product recognition result for the image received from the server, to the cart A controller for acquiring information about the product to be loaded, a display unit for displaying information about the product to be added to the cart, and a payment unit for performing payment for the product to be added to the cart based on the information about the product and, when a product to be put in the cart is sensed by the sensor unit, the controller outputs an operation signal for an operation to the camera to provide a smart shopping cart configured to obtain an image of the product to be added to the cart. can

According to another aspect of the present invention, as a product purchase method, a cart for placing a product, a communication unit communicating with a server, a camera installed in the cart, and capturing an image of the product to be put in the cart, a sensor unit, a display preparing a smart shopping cart including a unit, a payment unit, and a controller; detecting whether a product is contained in the cart through the sensor unit; Acquiring an image about the product to be put in the cart through the camera controlled by the controller, transmitting the image to the server that recognizes the product using an artificial neural network through the communication unit and receiving from the server obtaining information about the product to be put in the cart by using the product recognition result for the image, displaying information about the product to be added to the cart through the display unit, and the product through the payment unit There may be provided a product purchase method comprising the step of performing payment for the product to be added to the cart based on the information.

Solutions of the present invention are not limited to the above-described solutions, and solutions not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. will be able

According to an embodiment of the present application, by recognizing a product using artificial intelligence, it is possible to provide an environment in which a customer can autonomously perform a shopping process ranging from product pickup to payment.

According to another embodiment of the present application, it is possible to provide a location-based service using an RFID reader installed in a cart to provide a mart a solution that enables location-based real-time marketing based on a customer's purchase pattern.

The effect of the present application is not limited to the above-described effects, and the effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and accompanying drawings.

1 is a diagram illustrating a shopping system according to an exemplary embodiment.
2 and 3 are diagrams illustrating a smart shopping cart according to an exemplary embodiment.
4 to 6 are diagrams for explaining a smart shopping cart according to an embodiment.
7 is a flowchart of a product purchase method according to an exemplary embodiment.
8 to 10 are diagrams for explaining imaging conditions based on a plurality of distance values according to an exemplary embodiment.
11 is a diagram of a machine learning model according to an embodiment.

The above-described objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. However, since the present invention may have various changes and may have various embodiments, specific embodiments will be exemplified in the drawings and described in detail below.

In the drawings, the thicknesses of layers and regions are exaggerated for clarity, and also, an element or layer may be referred to as “on” or “on” another component or layer. What is referred to includes all cases in which other layers or other components are interposed in the middle as well as directly on top of other components or layers. Throughout the specification, like reference numerals refer to like elements in principle. In addition, components having the same function within the scope of the same idea shown in the drawings of each embodiment will be described using the same reference numerals.

If it is determined that a detailed description of a known function or configuration related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are only identification symbols for distinguishing one component from other components.

In addition, the suffixes "module" and "part" for the components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have a meaning or role distinct from each other by themselves.

According to an aspect of the present invention, as a smart shopping cart, a cart for placing a product, a communication unit for communicating with a server, a sensor unit for detecting whether a product is contained in the cart, is installed in the cart, and added to the cart A camera for photographing a product to obtain an image, and transmitting the image to the server that recognizes a product using an artificial neural network through the communication unit, and using the product recognition result for the image received from the server, to the cart A controller for acquiring information about the product to be loaded, a display unit for displaying information about the product to be added to the cart, and a payment unit for performing payment for the product to be added to the cart based on the information about the product and, when a product to be put in the cart is sensed by the sensor unit, the controller outputs an operation signal for an operation to the camera to provide a smart shopping cart configured to obtain an image of the product to be added to the cart. can

In one embodiment, the sensor unit includes a distance sensor for measuring a distance to the product to be loaded into the cart, and the controller sends the camera to the camera when the detected product to be loaded in the cart is located within a specific distance range. An imaging signal for imaging a product to be added to the cart may be output.

In an embodiment, the distance sensor includes a plurality of distance sensors, and the controller is configured to capture an image to the camera based on a plurality of distance values measured by the plurality of distance sensors to image the product in the cart. signal can be output.

In one embodiment, the artificial neural network is learned through a running set in which a plurality of training images and identifiers reflecting which products are in the plurality of training images are tagged with each other, and added to the cart in the acquired images. It is possible to calculate a result value that reflects what kind of product the product is.

In an embodiment, when the calculated result value is less than a predetermined value, the controller may transmit another image regarding the product to the cart through the communication unit to the server.

In one embodiment, the smart shopping cart further includes a barcode reader for reading a barcode of a product contained in the cart, and the controller is a product recognition result for the acquired image and the barcode obtained through the barcode reader. Information about the product included in the cart may be acquired using at least one of the barcode reading results.

In one embodiment, when the barcode is read by the barcode reader, the controller may acquire information about the product to be added to the cart based on a result of reading the barcode.

In one embodiment, the artificial neural network detects the barcode of the product in the cart from the acquired image, and the server recognizes the product using the detected barcode to obtain information about the product. .

In an embodiment, the smart shopping cart further includes an RFID reader for recognizing an RFID tag installed in a store to obtain location information of the smart shopping cart, and based on the location information of the smart shopping cart through the communication unit Information on the determined recommended product may be obtained from the server, and information on the recommended product may be displayed through the display unit.

In an embodiment, the RFID reader includes a first antenna and a second antenna, and the first antenna and the second antenna may be installed on both sides of the cart or a lower frame located below the cart.

According to another aspect of the present invention, as a product purchase method, a cart for placing a product, a communication unit communicating with a server, a camera installed in the cart, and capturing an image of the product to be put in the cart, a sensor unit, a display preparing a smart shopping cart including a unit, a payment unit, and a controller; detecting whether a product is contained in the cart through the sensor unit; Acquiring an image about the product to be put in the cart through the camera controlled by the controller, transmitting the image to the server that recognizes the product using an artificial neural network through the communication unit and receiving from the server obtaining information about the product to be put in the cart by using the product recognition result for the image, displaying information about the product to be added to the cart through the display unit, and the product through the payment unit There may be provided a product purchase method comprising the step of performing payment for the product to be added to the cart based on the information.

1 is a diagram illustrating a shopping system according to an exemplary embodiment.

Referring to FIG. 1 , a shopping system 10 according to an exemplary embodiment may include a smart shopping cart 100 , a server 200 , an RFID tag 300 , and a network 400 .

The smart shopping cart 100 may provide an autonomous payment service for products. For example, the smart shopping cart 100 may recognize a product to be loaded into or taken out of the cart and perform a procedure for payment of the recognized product. For example, the smart shopping cart 100 receives the recognition result of a product put into or out of the cart from the product recognition unit 210 that recognizes the product using the artificial neural network of the server 200 , and adds the recognition result to the received recognition result. Based on the information about the recognized product, it is possible to support payment of the recognized product. As another example, the smart shopping cart 100 obtains a barcode recognition result of a product put into or out of the cart by using the barcode of the product, and provides information about the recognized product based on the obtained barcode recognition result, Payment of recognized products can be supported.

The smart shopping cart 100 may provide a location-based service. For example, the smart shopping cart 100 may provide predetermined information to the customer based on the location of the smart shopping cart 100 . As an example, the smart shopping cart 100 may guide the customer with information including a location of a product desired by the customer, a recommended product, a recommended route, and the like. Here, the smart shopping cart 100 may obtain information provided to the customer from the server 200 .

In one embodiment, in order to provide a location-based service, the smart shopping cart 100 is a space (eg, a store, etc.) in which the RFID reader 110 and the smart shopping cart 100 installed in the smart shopping cart 100 are used. This can be done according to the location of the smart shopping cart 100 using the RFID tags 300-1 and 300-2 installed at a predetermined location (eg, a store shelf, etc.).

RFID is a wireless communication system composed of an RFID reader for reading and decoding functions and an RFID tag for providing information. For example, the RFID reader can read information provided from an RFID tag provided on an object in a non-contact manner.

The smart shopping cart 100 recognizes the RFID tags 300-1 and 300-2 installed in various places of the store through the RFID reader 110 installed in a predetermined area of the smart shopping cart 100 to provide the smart shopping cart 100. ) can be found. For example, the smart shopping cart 100 checks the RFID tags 300-1 and 300-2 recognized through the RFID reader 110, and the identified RFID tags 300-1 and 300-2 are located. Based on the , the location of the smart shopping cart 100 may be confirmed. For example, when the smart shopping cart 100 recognizes the RFID tag 300-1 through the RFID reader 110 and does not recognize the RFID tag 300-2, the location of the smart shopping cart 100 is It may be determined to be around the location of the tag 300 - 1 .

Of course, the smart shopping cart 100 is not limited to the above description, and may provide a location-based service by providing components for a conventional positioning system such as a GPS sensor or a beacon instead of using RFID.

The server 200 may communicate with the smart shopping cart 100 through the network 400 to exchange various data. For example, the server 200 may obtain an image of a product from the smart shopping cart 100 and send information related to recognition of a product in the acquired image. For another example, the server 200 obtains information on the location of the smart shopping cart 100 from the smart shopping cart 100 , and provides a location-based service based on the acquired location of the smart shopping cart 100 . You can send information for

To this end, the server 200 may include a product recognition unit 210 and a product recommendation unit 220 .

The product recognition unit 210 may recognize the product from the image.

For example, the product recognition unit 210 may acquire an image of a product from the smart shopping cart 100 , and analyze the acquired image to recognize a product in the image. For example, the product recognition unit 210 may recognize the product in the image using an artificial neural network. A method of recognizing a product in an image using an artificial neural network will be described later. The product recognition unit 210 is not limited to the description described above, and may recognize the product in another way, such as recognizing the product using the difference image technique.

The product recommendation unit 220 may recommend a product to the smart shopping cart 100 .

For example, the product recommendation unit 220 may recommend a product based on the location of the smart shopping cart 100 . For example, the product recommendation unit 220 may preferentially recommend a product close to the location of the smart shopping cart 100 among a plurality of recommended products.

As another example, the product recommendation unit 220 may recommend a product based on information about the customer. For example, the product recommendation unit 220 may recommend a product in consideration of the customer's past purchase history, the customer's purchasing propensity, and the customer's characteristics (eg, gender, age, etc.). Information about the customer may be obtained from the smart shopping cart 100 or stored in a memory (not shown) of the server 200 . The product recommendation unit 220 is not limited to the above description, and may recommend products in other ways, such as recommending products using a big data analysis technique.

In addition, the server 200 shown in FIG. 1 is merely an example for convenience of description and is not limited thereto. According to some embodiments, a configuration may be added to or excluded from the server 200 of FIG. 1 , and may also be subdivided.

For example, the server 200 may include a memory (not shown) for storing various data. As an example, the server 200 may include a memory for storing a machine learning model for recognizing a product or information about a customer. For another example, the server 200 controls each configuration of the communication module (not shown) and the server 200 for communicating with an external device (eg, the smart shopping cart 100 ) or processes various information. And it may include a controller (not shown) for calculating and the like.

The network 400 may be understood as encompassing all wired/wireless networks such as a general communication network such as the Internet or an intranet, a mobile network, Bluetooth, and WiFi.

The shopping system 10 shown in FIG. 1 is only an example, and the configuration of the shopping system 10 is not limited thereto, and functions performed by each configuration of the shopping system 10 are not necessarily performed by the corresponding configuration. It may be performed by other configurations. For example, the product recognition unit 210 may be included in the smart shopping cart 100 instead of the server 200 to perform a function for product recognition.

2 and 3 are diagrams illustrating a smart shopping cart according to an exemplary embodiment.

2 and 3 , the smart shopping cart 100 includes a cart 101 , a lower frame 103 , a handle 105 , a wheel 107 , a camera 120 , a display unit 150 , and a payment unit. (160).

The cart 101 is a storage space for storing products, and one component of the smart shopping cart 100 may be installed therein. For example, a camera 120 , a sensor unit 140 , etc. may be installed in the cart 101 . A detailed description thereof will be provided later.

The lower frame 103 is a frame positioned below the cart 101 , and one component of the smart shopping cart 100 may be installed therein. For example, the antenna 111 may be installed in the lower frame 103 . A detailed description thereof will be provided later. In addition, the lower frame 103 may be used as an auxiliary storage space for storing products.

The handle 105 may be installed in a predetermined area of the smart shopping cart 100 to help a customer turn off the smart shopping cart 100 . For example, the handle 105 may be installed on the upper portion of the cart 101 to be easily gripped by the customer.

The wheel 107 may serve as a moving means for moving the smart shopping cart 100 . For example, the wheels 107 are installed to be connected to the lower frame 103 to move the smart shopping cart 100 .

The camera 120 may be installed in a predetermined area of the cart 101 to image the inside of the cart. For example, the camera 120 may acquire an image of products loaded or taken out of the cart 101 . For example, the camera 120 may be installed on the inner rim of the cart 101 . The camera 120 may have a protective housing (eg, urethane material) installed in a portion of the cart 101 in order to prevent damage or trapping of foreign substances, and the camera 120 may be installed in the protective housing.

The display unit 150 may be installed in a predetermined area of the smart shopping cart 100 to display various information on the screen. For example, the display unit 150 may be installed around the handle 105 so that a customer can easily view the screen.

The payment unit 160 may be installed in a predetermined area of the smart shopping cart 100 to perform a payment procedure. For example, the payment unit 160 may be installed around the display unit 150 so that a customer can pay while viewing a screen for payment for convenience of payment.

However, FIGS. 2 and 3 are only examples for convenience of description, and the present invention is not limited thereto, and the shape of each component may be changed, some components may be excluded according to selection, or various additional components may be added.

4 to 6 are diagrams for explaining a smart shopping cart according to an embodiment.

Referring to FIG. 4 , a smart shopping cart 100 according to an embodiment includes an RFID reader 110 , a camera 120 , a barcode reader 130 , a sensor unit 140 , a display unit 150 , and a payment unit ( 160 ), a communication unit 170 , and a controller 180 .

The RFID reader 110 may include an antenna, an RF unit, a control unit, a communication unit, and a switch capable of selectively requesting data transmission from the RFID tag 300 to obtain data stored in the RFID tag 300 . can At least one RFID reader 110 may be installed in the smart shopping cart 100 . Since the RFID reader 110 is a known technology, a detailed description thereof will be omitted.

The RFID reader 110 may be installed in a predetermined area of the smart shopping cart 100 . For example, the RFID reader 110 may be installed in the cart 101 or installed at the bottom or rear of the cart 101 . In addition, the smart shopping cart 100 is not limited to the above description, such as including a plurality of RFID readers (110).

In one embodiment, the RFID reader 110 may include an antenna 111 installed separately from the main body of the RFID reader 110 . Referring to FIG. 5 , the smart shopping cart 100 may include antennas 111-1 and 111-2 installed separately from the main body of the RFID reader 110 . Here, the antennas 111-1 and 111-2 installed separately from the main body of the RFID reader 110 may be installed in a predetermined area of the smart shopping cart 100 .

For example, the antenna 111 may be installed under the cart 101 .

Referring to FIG. 6( a ), the antenna 111 may be installed in the lower frame 103 positioned under the cart 101 . In addition, a plurality of antennas 111 may be installed. For example, the antenna 111 may include first and second antennas 111 , and the first and second antennas 111 may be installed on both sides of the lower frame 103 .

As another example, the antenna 111 may be installed on both sides of the cart 101 . Referring to FIG. 6( b ), the cart 101 is shown as viewed from above, and the antenna 111 may be installed on at least one of both sides of the cart 101 . In addition, a plurality of antennas 111 may be installed. For example, the antenna 111 includes a first antenna 111-1 and a second antenna 111-2, and the first antenna 111-1 and the second antenna 111-2 include a cart ( 101) can be installed on both sides. Here, the main body of the RFID reader 110 may be installed in the cart 101 or installed at the bottom or rear of the cart 101 .

The recognition rate of the RFID system can be improved through a structure in which the antenna 111 is installed separately from the main body of the RFID reader 110 and/or a structure in which a plurality of antennas 111 are installed on both sides of the cart 101. have.

The camera 120 may acquire an image. For example, the camera 120 may be implemented as a monocular camera, a binocular camera, a visible light camera, an IR camera, a depth camera, and the like, but is not limited thereto.

In one embodiment, the camera 120 may operate only when a product to be loaded into the cart 101 is detected by the sensor unit 140 . This is to reduce power consumption by the camera 120 because the power consumption of the camera 120 is large due to real-time video streaming, and the power consumed by operating the camera 120 only when a product is placed in the cart 101 . can reduce For example, when a product to be loaded into the cart 100 is detected by the sensor unit 140 , the controller 180 outputs an operation signal for an operation to the camera 120 to add the product to the cart 100 . image can be obtained. For example, the camera 120 may operate when a product contained in the cart 101 is sensed by the sensor unit 140 to image the product for a predetermined period of time. Here, an image suitable for product recognition using an artificial neural network may be selected from among a plurality of images acquired for a predetermined time.

In one embodiment, a plurality of cameras 120 may be installed on the cart 101 . For example, the plurality of cameras 120 may be installed in different positions of the cart 101 to acquire various images with respect to the products.

Referring to FIG. 5 , a front image of products loaded or taken out of the cart 101 through a first camera 120-1 and a second camera 120-2 installed to face each other inside the cart 101 and A back image can be acquired together. The server 200 may recognize the product by receiving the first image acquired through the first camera 120-1 and the second image acquired through the second camera 120-2. Here, the server 200 may recognize a product based on an image having a higher probability of being a product output as an output value of the artificial neural network used in the server 200 among the first image and the second image.

The plurality of cameras 120 are not limited to the above description, such as being installed at various locations or three or more cameras to obtain images of various angles regarding products.

The barcode reader 130 may read a barcode located in an area of a product loaded or loaded in the cart 101 . For example, the barcode reader 130 may read the barcode to obtain information about the product. Information about the product obtained by the barcode reader 130 may be used for payment in the payment unit 160 . That is, the barcode reader 130 may function as an auxiliary means for product recognition.

The barcode reader 130 may refer to a series of input devices that irradiate light such as an electron beam to a barcode printed on one area of an object to be recognized, and read an electronic signal generated according to the intensity of the reflected light. A barcode is a one-dimensional barcode made by combining a plurality of bars of various thicknesses or a two-dimensional form consisting of X-axis and Y-axis in a certain rectangle by arranging braille or mosaic symbols. It may include a two-dimensional barcode representing information. Since the barcode reader 130 is a known technology, a detailed description thereof will be omitted.

Referring to FIG. 5 , the barcode reader 130 is installed in a predetermined area of the cart 101 , and can read a barcode located in one area of a product loaded or loaded in the cart 101 .

The sensor unit 140 may sense information on the smart shopping cart 100 or the product loaded into or taken out of the cart 101 . For example, the sensor unit 140 may include a distance sensor, a weight sensor, a contact sensor, a temperature sensor, a humidity sensor, an infrared sensor, an ultrasonic sensor, an acceleration sensor, and the like.

The sensor unit 140 may detect a product loaded into or taken out of the cart 101 . That is, the sensor unit 140 may detect whether a product is contained in the cart 101 . For example, the sensor unit 140 may include a weight sensor (not shown) and detect whether a product is loaded in the cart 101 based on a change in the weight of the cart 101 . The sensor unit 140 is not limited thereto, such as being composed of other sensors capable of detecting whether a product is contained in the cart 101 .

In an embodiment, the sensor unit 140 may be implemented as a distance sensor 141 and installed around the camera 120 . For example, the distance sensor 141 may be implemented and installed in a direction toward the inside of the cart 101 around the camera 120 . For example, the sensor unit 140 may include, but is not limited to, a distance sensor 141 such as an ultrasonic sensor, an infrared sensor, a LIDAR sensor, a radar (RADAR) sensor, and a camera sensor.

In an embodiment, the distance sensor 141 may detect a product loaded or taken out of the cart 101 by measuring the distance to the product loaded or taken out of the cart 101 . For example, if the distance to the product contained in the cart 101 measured by the distance sensor 141 is located within a specific distance range, the controller 180 determines that the product is contained in the cart 101 . can That is, the controller 180 may determine the intention of scanning the product based on the distance to the product contained in the cart 101 measured by the distance sensor 141 .

In an embodiment, the sensor unit 140 may include a plurality of distance sensors 141 . For example, the controller 180 may measure a plurality of distance values for a product loaded or taken out of the cart 101 by using the plurality of distance sensors 141 . As an example, the controller 180 may determine the intention of scanning the product based on a plurality of distances to the products contained in the cart 101 measured by the plurality of distance sensors 141 .

Referring to FIG. 5 , the sensor unit 140 may include a plurality of distance sensors 141-1, 141-2, 141-3, and 141-4 installed around the camera 120 . A plurality of distance values measured by the plurality of distance sensors 141-1, 141-2, 141-3, and 141-4 may be used to detect a product loaded or taken out of the cart 101 . More details on this will be provided later.

The display unit 150 may output various types of visual information. For example, the display unit 150 may display information related to payment of a product. For example, the display unit 150 may be recognized by the server 200 or the barcode reader 130 to display information about a product for which payment is to be performed, and may provide a payment window. In addition, the display unit 150 may display a screen for inquiry of shopping cart details, inquiry of product information, and the like.

As another example, the display unit 150 may display information for providing a location-based service. For example, the display unit 150 may display information including a location of the smart shopping cart 100 , a location of a product desired by a customer, a recommended product, a recommended route, and the like.

The display unit 150 may be, for example, an LCD, an OLED, or an AMOLED display. When the display unit 150 is provided as a touch screen, the display unit 150 may perform the function of the input unit.

In an embodiment, the display unit 150 may be implemented as a terminal. For example, the display unit 150 is a smart phone, a tablet PC (Personal Computer), a notebook PC, a portable game terminal, a mobile terminal such as a portable / dongle type mobile router or digital camera, M2M (Machine To Machine) communication It may be implemented in the form of a vehicle-mounted terminal such as a mobile terminal or a car navigation device.

When the display unit 150 is implemented in the form of a terminal, a POS program for payment of goods may be stored in the terminal, so that it is also possible to perform the functions of the payment unit 160 together. That is, the smart shopping cart 100 may include a terminal such as a tablet PC that performs the functions of the display unit 150 and the payment unit 160 together. The POS program may refer to a program for intensively managing information related to sales by recording information on sold products at the time of sale.

The payment unit 160 may perform payment for a product. For example, the payment unit 160 may include a POS terminal. The POS terminal may refer to a terminal in which a POS program used in a Point Of Sales system (POS system) is stored. The POS terminal may be implemented as a terminal such as a tablet PC together with the display unit 150 .

The payment unit 160 may recognize the customer's payment method in order to pay for the product based on the information about the products included in the cart 101 . For example, the payment unit 160 may recognize a credit card for payment of products included in the cart 101 . Of course, the payment unit 160 is not limited to the above description, such as being implemented as a terminal that recognizes an electronic payment means, and may be implemented by a known technology for product payment.

The communication unit 170 may communicate with an external device. For example, the communication unit 170 may communicate with the server 200 and the like. For example, the smart shopping cart 100 may transmit an image related to a product to the server 200 through the communication unit 170 and receive a product recognition result from the server 200 . As another example, the smart shopping cart 100 may transmit the location of the smart shopping cart 100 to the server 200 through the communication unit 170 , and receive information for providing a location-based service from the server 200 . have.

The communication unit 170 may perform communication mainly according to wireless communication standards, but BLE (Bluetooth Low Energy), Bluetooth (Bluetooth), WLAN (Wireless LAN), WiFi (Wireless Fidelity), WiFi Direct, NFC (Near Field Communication) ), infrared communication (Infrared Data Association; IrDA), UWB (Ultra Wide Band), Zigbee, 3G, 4G or 5G, including mobile communication modules and other various communication standards including wired and wireless modules that transmit data can do.

The controller 180 may control each configuration of the smart shopping cart 100 or process and calculate various types of information.

The controller 180 may be implemented by software, hardware, or a combination thereof. For example, in terms of hardware, the controller 180 may be implemented as a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a semiconductor chip, and other various types of electronic circuits. , in software, the controller 180 may be implemented as a logic program executed according to the above-described hardware, various computer languages, etc. In addition, the controller 180 may include a microprocessor, a microcontroller, a digital signal processing device (DSP) or its It may be of any type, including, but not limited to, any combination.

In the following description, unless otherwise stated, it may be understood that the operation of the smart shopping cart 100 is performed under the control of the controller 180 .

In one embodiment, the smart shopping cart 100 may include a terminal that performs at least one function of each component. For example, the display unit 150 and/or the payment unit 160 may be provided in the form of a terminal. For example, the terminal is a smart phone, a tablet PC (Personal Computer), a notebook PC, a portable game terminal, a mobile terminal such as a portable/dongle-type mobile router or digital camera, a M2M (Machine to Machine) communication terminal, or a car navigation device. It may be implemented as a vehicle-mounted terminal such as a device. Here, the provided terminal may be detachable, and may be implemented to perform functions of other components of the smart shopping cart 100, such as performing the role of the communication unit 170 together.

In addition, the smart shopping cart 100 illustrated in FIGS. 4 to 6 is merely an example for convenience of description and is not limited thereto. According to some embodiments, components may be added to, excluded from, or subdivided in the smart shopping cart 100 of FIGS. 4 to 6 . For example, the smart shopping cart 100 applies power to an input unit (not shown) that receives various inputs from customers, a speaker (not shown) for outputting audible information, and components of the smart shopping cart 100 . It may further include a battery (not shown) provided.

7 is a flowchart of a product purchase method according to an exemplary embodiment.

Referring to FIG. 7 , in the method of purchasing a product according to an embodiment, acquiring at least one of an image and a barcode reading result related to a product in a cart ( S1000 ), based on at least one of the acquired image and barcode reading result to obtain information about the product ( S2000 ) and may include a step ( S3000 ) of paying for the product to be added to the cart.

The smart shopping cart 100 may acquire at least one of an image and a barcode reading result related to a product contained in the cart 101 ( S1000 ).

The smart shopping cart 100 may acquire an image of a product included in the cart 101 through the camera 120 . The acquired image is transmitted to the server 200 through the communication unit 170 , and the product recognition unit 210 of the server 200 may use the artificial neural network to recognize the product.

In an embodiment, the smart shopping cart 100 may acquire an image about the product based on the distance to the product measured through the distance sensor 141 .

For example, the camera 120 may image a product based on the distance sensor 141 . As an example, the camera 120 receives an imaging signal for imaging the product from the controller 180 when the product loaded in the cart 101 is located within a specific distance range, and captures the product loaded in the cart 101 to obtain an image. can be obtained. In this case, when a product is photographed with the camera 120 mounted on the cart 101, if the distance between the product and the camera 120 is too close, only a part of the product is photographed, and the distance between the product and the camera 120 is too close. This is because it is difficult to grasp the characteristics of a product if it is far away.

The camera 120 may image the product for a predetermined time when the product contained in the cart 101 is located within a specific distance range. Here, an image suitable for product recognition using an artificial neural network may be selected from among a plurality of images obtained by the camera 120 capturing images for a predetermined time.

In an embodiment, the smart shopping cart 100 may acquire an image about the product based on a plurality of distance values to the product measured through the plurality of distance sensors 141 . That is, the smart shopping cart 100 may use the plurality of distance sensors 141 to identify the customer's product scanning intention and select an image suitable for use in the product recognition unit 210 of the server 200 . An image suitable for use in the product recognition unit 210 of the server 200 may be an image of the product being photographed from the front or an image in which the product is located within the viewing angle of the camera 120 . Through this, the recognition rate of the product through the product recognition unit 210 of the server 200 may be improved.

To this end, the smart shopping cart 100 may determine whether a predetermined imaging condition is satisfied based on a plurality of distance values obtained by the plurality of distance sensors 141 . The predetermined imaging condition may mean a condition for determining an image suitable to be used in the product recognition unit 210 of the server 200, for example, considering the product's inclination, the viewing angle of the camera 120, and the like. Thus, it can be used to determine the intention to scan the product, and to instruct the camera 120 to take an image. For example, the smart shopping cart 100 may determine whether a predetermined imaging condition is satisfied in consideration of the existence or nonexistence of a plurality of distance values measured by the plurality of distance sensors 141 and differences between the distance values.

8 to 10 are diagrams for explaining imaging conditions based on a plurality of distance values, according to an exemplary embodiment.

In one embodiment, the smart shopping cart 100 may take an image of the product in consideration of the inclination of the product. For example, the smart shopping cart 100 may include distance values measured through the first distance sensor 141-1 and the second distance sensor 141-2 installed at regular intervals around the camera 120. It is possible to judge whether the product is tilted based on the existence or non-existence of the product.

Referring to FIG. 8 , in the smart shopping cart 100 , the first distance value to the product is measured through the first distance sensor 141-1, and the distance to the product is measured through the second distance sensor 141-2. If the second distance value is not measured, it is determined that the product is tilted and does not satisfy a predetermined imaging condition, and the product may not be captured through the camera 120 . Unlike FIG. 8 , in the smart shopping cart 100 , the first distance value to the product is measured through the first distance sensor 141-1, and the first distance value to the product is measured through the second distance sensor 141-2. 2 When the distance value is also measured, it is determined that the product is not inclined and a predetermined imaging condition is satisfied, and the product can be imaged through the camera 120 .

As another example, in the smart shopping cart 100 , the distance value measured through the first distance sensor 141-1 and the second distance sensor 141-2 installed at regular intervals around the camera 120 . Based on the difference in , it is possible to determine whether the product is tilted or not.

Referring to FIG. 9 , in the smart shopping cart 100 , the first distance value d1 to the product is measured through the first distance sensor 141-1, and the product is measured through the second distance sensor 141-2. If the difference between d1 and d2 is greater than or equal to a predetermined value even though the second distance value d2 to it may not be When the difference between d1 and d2 is less than a predetermined value, the smart shopping cart 100 determines that the product is not tilted and satisfies the predetermined imaging condition, and may image the product through the camera 120 .

In an embodiment, the smart shopping cart 100 may image a product in consideration of a field of view (FOV) of the camera 120 . The viewing angle may mean to what extent the image is included in the image horizontally or vertically, and may generally be expressed as an angle (degree, degree). When the product is located within the viewing angle of the camera 120 , the smart shopping cart 100 captures the product through the camera 120 to obtain an image suitable for use in the product recognition unit 210 of the server 200 . have.

For example, a plurality of distance sensors 141 are installed in the smart shopping cart 100 to measure a distance in an area equal to the viewing angle of the camera 120 , and the controller 180 includes the plurality of distance sensors ( Based on the distance value measured through 141), it may be determined whether the product is located within the viewing angle of the camera 120 .

For another example, the controller 180 includes a first distance sensor 141-1, a second distance sensor 141-2, and a third distance sensor 141- which are installed at regular intervals around the camera 120. 3) and the fourth distance sensor 141-4 may determine whether the product is located within the viewing angle of the camera 120 based on the presence or absence or difference of the distance value measured through the sensor 141-4.

Referring to FIG. 10 , the smart shopping cart 100 provides a third distance value to the product through the third distance sensor 141-3 and a fourth distance value to the product through the fourth distance sensor 141-4. If this is not measured and the first distance value to the product through the first distance sensor 141-1 and the second distance value to the product through the second distance sensor 141-2 are measured, the camera 120 ), it is determined that the product is not located within the viewing angle and does not satisfy the predetermined imaging condition, and the product may not be captured through the camera 120 .

In contrast, in the smart shopping cart 100 , the first distance value to the product through the first distance sensor 141-4 and the fourth distance value to the product through the fourth distance sensor 141-4 are not measured. , when the second distance value to the product through the second distance sensor 141-2 and the third distance value to the product through the third distance sensor 141-3 are measured, within the viewing angle of the camera 120 It is determined that the product is located and satisfies a predetermined imaging condition, and the product can be imaged through the camera 120 .

The first distance sensor 141-1 and the fourth distance sensor 141-4 may be installed to measure a distance with respect to an area outside the range of the viewing angle of the camera 120, and the second distance sensor 141-2 and the third distance sensor 141-3 may be installed to measure a distance with respect to an area within the range of the viewing angle of the camera 120, but is not limited thereto.

The examples shown in FIGS. 8 to 10 are only examples for convenience of description and are not limited thereto. As an example, although the distance sensor 141 is expressed as using a single scanning line for distance measurement in FIGS. 8 to 10 , it is not limited thereto, such as using a plurality of scanning lines or using a scanning area.

In one embodiment, when the smart shopping cart 100 acquires an image satisfying a predetermined imaging condition through the camera 120 based on the distance sensor 141, the display unit 150 or An alarm may be provided through a speaker (not shown) or the like.

Returning to FIG. 7 again, it will be described.

The smart shopping cart 100 may obtain a barcode reading result by reading a barcode on a product contained in the cart 101 through the barcode reader 130 . For example, the barcode reader 130 may obtain a barcode reading result by scanning the barcode of the product when the barcode of the product is readable because the product is located at a close distance. The barcode reading result may be used to obtain information about the product.

The smart shopping cart 100 may acquire information about the product based on at least one of the acquired image and the barcode reading result ( S2000 ).

The smart shopping cart 100 may acquire information about the product based on the image acquired through the camera 120 . For example, the smart shopping cart 100 transmits an image acquired through the camera 120 to the server 200 , and the server 200 receives the image and recognizes the product in the image through the product recognition unit 210 . can recognize

In one embodiment, the product recognition unit 210 of the server 200 may recognize the product in the image using an artificial neural network.

11 is a diagram of a machine learning model according to an embodiment.

Referring to FIG. 11 , the artificial neural network may include an input layer, a hidden layer, and an output layer. The hidden layer may have a plurality of nodes connecting the input layer and the output layer. Such an artificial neural network is highly trained by adjusting the weight value of the hidden layer through a learning set in which the input data to be input to the input layer and the pre-confirmed output value are tagged with each other, and can be learned to derive an accurate output value from the input data. In the input layer, an image of a product may be used as input data. For example, the artificial neural network is learned through a running set in which a plurality of training images and an identifier reflecting the product in the plurality of training images are tagged with each other, and the product included in the cart in the image is a product. It is possible to calculate a result value that reflects

As an example, the artificial neural network may be designed such that an image of a product is input to the input layer as input data, and a related result value is output to the output layer. For example, an image in which a product is captured may be input as input data, and a type of product may be output as an output value. Here, the output value may be a probability related to which product the product in the image is, and the artificial neural network may recognize the product as the product having the highest probability. That is, the result value output from the artificial neural network may be a probability value of which product is the product in the image.

As another example, the artificial neural network may be designed such that a plurality of captured images are input as input data to an input layer, and a result value related thereto is output to an output layer. For example, an image captured for a predetermined time may be input as input data, and a product type may be output as an output value. Here, the output value may be a probability related to which product the product in the image is, and the artificial neural network may recognize the product as the product with the highest probability. That is, the result value output from the artificial neural network may be a probability value of which product the product in the image is.

In one embodiment, the product recognition unit 210 or the controller 180 may determine that the product has not been recognized when the probability associated with the product, which is the result value calculated by the artificial neural network, is less than a predetermined value. . For example, the product recognition unit 210 determines that recognition is not performed properly when the probability value of the product in the image calculated by the artificial neural network is too low, and sets another image for product recognition into the smart shopping cart 100 . ) can be obtained from

In one embodiment, the artificial neural network may be designed to detect barcodes of goods in carts from images. For example, the artificial neural network may be designed such that a plurality of captured images are input to an input layer as input data, and data about a barcode is output to an output layer. Through this, the server can detect the barcode of the product to be put in the cart from the image, and recognize the product using the detected barcode. The smart shopping cart 100 may receive information about a product obtained by using the barcode detected from the server 200 .

Types of artificial neural networks include a convolutional neural network (CNN) that extracts features using a filter, and a recurrent neural network (RNN) that has a structure in which the output of a node is fed back as an input. Various structures such as restricted Boltzmann machine (RBM), deep belief network (DBN), generative adversarial network (GAN), relational network (RN), etc. can be applied and have limitations. it is not

Returning to FIG. 7 again, it will be described.

The smart shopping cart 100 may acquire information about the product based on the barcode read through the barcode reader 130 . For example, the barcode reader 130 may scan a barcode of a product to obtain a barcode reading result, and use this to acquire information about the product.

In an embodiment, the smart shopping cart 100 may determine whether to obtain information about a product by using either an image or a barcode.

For example, the smart shopping cart 100 may acquire information about a product included in the cart in consideration of whether the barcode is read by the barcode reader 130 . For example, when the barcode of the product to be loaded into the cart is read by the barcode reader 130 , the controller 180 may acquire information about the product to be loaded into the cart by using the result of reading the barcode.

As another example, the smart shopping cart 100 may acquire information about a product included in the cart in consideration of an output value output through an artificial neural network used in the product recognition unit 210 . As an example, the controller 180 obtains information about a product to be added to the cart from the server 200 only when the probability of a specific product output as an output value from the product recognition unit 210 as an output value is greater than or equal to a predetermined value. can When the probability of a specific product output from the product recognition unit 210 as an output value through an artificial neural network is less than a predetermined value, the smart shopping cart 100 displays a different image (for example, an image captured at a different angle or at a different time). can be transmitted to the server 200, and information about the product to be added to the cart can be obtained. To this end, the smart shopping cart 100 may give the customer an alarm notifying the re-shooting of the product through the display unit 150 or a separate output unit (not shown), and may provide other information regarding the product to be added to the cart. image can be obtained. Alternatively, the smart shopping cart 100 may transmit to the server 200 another image among a plurality of images of the product captured for a predetermined time.

Of course, the server 200 may determine whether the probability of a specific product output as an output value through the artificial neural network reaches a predetermined value. For example, when the probability of a specific product output as an output value through an artificial neural network is greater than or equal to a predetermined value, the server 200 transmits information about the product to the smart shopping cart 100 to the smart shopping cart 100, and through the artificial neural network When the probability of a specific product output as an output value is less than a predetermined value, another image may be requested to the smart shopping cart 100 or a message indicating that product recognition accuracy is poor may be transmitted.

In an embodiment, the smart shopping cart 100 may verify information about the acquired product. For example, the smart shopping cart 100 may verify the product recognition result for the image received from the server 200 based on the barcode reading result obtained through the barcode reader 130 . This is because the accuracy of the barcode reading result is higher than the accuracy of the product recognition result for the image.

For example, when the product recognition result for the image received from the server 200 and the barcode reading result obtained through the barcode reader 130 are different, the smart shopping cart 100 uses the barcode reading result to add the product to the product. information can be obtained.

As another example, when the product recognition result for the image received from the server 200 and the barcode reading result obtained through the barcode reader 130 are different, the smart shopping cart 100 may display a different image (eg, a different angle). Alternatively, an image captured at another time, etc.) may be transmitted to the server 200 , and information about a product to be added to the cart may be obtained.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Smart Shopping Cart 100
RFID Reader 110
camera 120
barcode reader 130
sensor unit 140
display unit 150
Payment Department 160
communication department 170
controller 180
server 200
RFID tag 300
network 400

Claims (14)

As a smart shopping cart,
carts for goods;
a communication unit communicating with the server;
a sensor unit for detecting whether a product is contained in the cart;
a camera installed on the cart;
a controller that transmits an image to the server for recognizing a product using an artificial neural network through the communication unit and acquires information about a product to be added to the cart using a product recognition result for the image received from the server; and
Including; a display unit for displaying information about the product to be put in the cart;
the controller is
When the product to be put in the cart is detected by the sensor unit, an image of the product to be added to the cart is obtained by outputting an operation signal for operation to the camera,
The sensor unit includes one or more distance sensors for obtaining a distance value to the product put in the cart,
the controller is
Determining whether to output the operation signal based on at least one distance value,
The operation signal is output when the at least one distance value satisfies a predetermined imaging condition, so that the camera acquires an image of a product to be added to the cart.
Smart shopping cart.
According to claim 1,
the controller is
Outputting the operation signal when the product in the cart sensed by the at least one distance value is located within a specific distance range
Smart shopping cart.
3. The method of claim 2,
The sensor unit includes a plurality of distance sensors,
the controller is
determining whether to output the operation signal based on a plurality of distance values measured by the plurality of distance sensors
Smart shopping cart.
According to claim 1,
The artificial neural network is learned through a running set in which a plurality of training images and an identifier reflecting which product is a product in the plurality of training images are tagged with each other, and what product is the product included in the cart in the acquired image to produce a result that reflects
Smart shopping cart.
5. The method of claim 4,
the controller is
When the calculated result value is less than a predetermined value, transmitting another image about the product to be added to the cart through the communication unit to the server
Smart shopping cart.
According to claim 1,
Further comprising; a barcode reader for reading the barcode of the product put in the cart;
the controller is
Using at least one of a product recognition result for the acquired image and a result of reading the barcode obtained through the barcode reader to obtain information about the product to be added to the cart
Smart shopping cart.
7. The method of claim 6,
the controller is
When the barcode is read by the barcode reader, information about the product to be put in the cart is obtained based on the reading result of the barcode.
Smart shopping cart.
According to claim 1,
The artificial neural network detects the barcode of the product in the cart from the acquired image,
The server recognizes the product using the detected barcode to obtain information about the product.
Smart shopping cart.
According to claim 1,
an RFID reader for recognizing an RFID tag installed in a store to obtain location information of the smart shopping cart; further comprising,
Obtaining information about a recommended product determined based on location information of the smart shopping cart from the server through the communication unit,
Displaying information about the recommended product through the display unit
Smart shopping cart.
10. The method of claim 9,
The RFID reader includes a first antenna and a second antenna,
The first antenna and the second antenna are installed on both sides of the lower frame or the cart located below the cart
Smart shopping cart.
A method of purchasing a product, comprising:
preparing a smart shopping cart including a cart for storing products, a communication unit communicating with a server, a camera installed in the cart, a sensor unit, a display unit, a payment unit, and a controller;
detecting whether a product is contained in the cart through the sensor unit, wherein the sensor unit includes one or more distance sensors for obtaining a distance value to the product stored in the cart;
Determining whether or not to output an operation signal for allowing the camera to acquire an image of a product to be added to the cart based on at least one distance value - In this case, the operation signal is the at least one distance value Outputs when the imaging conditions are satisfied - ;
acquiring an image of the product to be added to the cart through the camera controlled by the controller when the product to be added to the cart is detected by the sensor unit;
transmitting the image to the server for recognizing a product using an artificial neural network through the communication unit and obtaining information about a product to be put in the cart using a product recognition result for the image received from the server;
displaying information about the product to be added to the cart through the display unit; and
Containing, comprising:;
How to buy a product.
According to claim 1,
A payment unit for performing payment for the product to be added to the cart based on the information about the product; further comprising
Smart shopping cart.
According to claim 1,
When the at least one distance value does not satisfy the predetermined imaging condition, the operation signal is not output until the at least one distance value satisfies the predetermined imaging condition.
Smart shopping cart.
According to claim 1,
The controller determines whether the predetermined imaging condition is satisfied by determining the inclination of the product in the cart based on the at least one distance value.
Smart shopping cart.

Images