KR20190093733A - Items recognition system in unmanned store and the method thereof - Google Patents

Abstract

The present invention relates to an item recognition system in an unmanned store and a method thereof. The present invention relates to a technique which uses a sensor module which includes a photo sensor detecting a row location for a plurality of items displayed in a display stand and a load cell sensor detecting weights of the plurality of items and an image module obtaining image information for the plurality of items to recognize an event for the plurality of items and compare transaction data generated by the event and inventory data so as to manage items.

Description

ITEMS RECOGNITION SYSTEM IN UNMANNED STORE AND THE METHOD THEREOF}

The present invention relates to an item recognition system and a method thereof in an unmanned store, and more particularly, by using a sensor module and an image module to recognize events for a plurality of items displayed on the shelf, the change data and inventory according to the event A technique for managing items based on data.

Retailers, wholesalers, and distributors of other products generally maintain inventory of a variety of products that can be ordered and purchased by customers or customers. For example, if you look at a typical mart, a user (customer) enters the mart, selects a product, and lines up a cash register with a cart containing the product. Thereafter, the mart employee recognizes the barcode of the product selected by the user and performs the calculation.

The fourth industrial revolution, which has recently been spotlighted, includes representative keywords such as big data, the Internet of Things, artificial intelligence, and deep learning. The technology of the Fourth Industrial Revolution can have a direct impact on our real life and is evolving through learning and analysis beyond being a convenient tool in our lives.

Recently, with the development of these technologies, interest in offline shopping to which the fourth industrial revolution is applied has increased.

Amazon has launched a new offline mart, Amazon Go, in response to these technological changes. Amazon Go is not much different from a grocery store that sells groceries and sundries. However, users in Amazon Go do not need to perform any calculation after selecting a product. Amazon Go automatically recognizes the product selected by the user and bills for the recognized product using a payment method corresponding to the user.

Accordingly, US Patent Publication No. US2015 / 0012396, filed by Amazon, discloses a method of 'transitioning items from a materials handling facility', which updates a user's product list by tracking the removal or placement of goods.

However, U.S. Patent Application Publication No. 2015/0012396 recognizes the movement of a product using RFID (Radio Frequency IDentification) and recognizes the movement of the user by using a camera. The product (hereinafter referred to as 'item') There was a limit to tracking the removal or placement of a).

That is, the patent describes the use of various sensors, but the embodiment or the right for the application of the sensor is not specified, and mainly uses Wi-Fi, NFC, RFID and camera. In addition, the conventional unmanned store system mainly uses a variety of technologies, such as artificial intelligence, computer vision or sensors, but there is a limit that the accuracy of item management in various situations that may occur when a user purchases a product in real life exists It was.

US published patent US2015 / 0012396 published January 1, 2015, "Transitioning items from a materials handling facility"

An object of the present invention is an item in which an event of pick up or put down according to a user's selection is generated using a photo sensor, a camera, and a load cell sensor positioned near a plurality of items displayed on a display stand. To provide a technology that can recognize more accurately.

In addition, an object of the present invention is a system for managing an item by determining whether the item is moving, opening, correct position or return by using the variation data for the item on which the event occurred and the inventory data related to the plurality of items. To provide a more stable and efficient technology to design.

Item recognition system according to an embodiment of the present invention is a sensor module including a photo sensor for detecting the position of the row for a plurality of items displayed on the display rack, a load cell sensor for sensing the weight of the plurality of items, the plurality of items Recognizing events for the plurality of items using the image module and the sensor module and the image module to obtain image information for the comparison, comparing the variation data generated by the event with the inventory data associated with the plurality of items And an inventory management unit to manage the items.

The sensor module detects a weight at a bottom of the photo sensor and the plurality of items and the photo sensor for detecting a row position for each of the plurality of items displayed in a plurality of columns on the display rack partitioned by a plurality of rows and a plurality of columns. It may include the load cell sensor.

The photo sensor may measure a distance of an item in which an event occurs among a plurality of items displayed in a plurality of columns by using a light emitting unit for emitting light and a light receiving unit spaced apart from the light emitting unit at a predetermined interval and forming a pair. have.

The image module may be a camera. The camera may include inventory data of items displayed on the display rack, items for which events have occurred, and parts of a user's body accessing the items. Can be obtained.

The inventory manager may update the inventory data by combining the event recognition with the sensor module and the image module-based recognition system.

The inventory manager may recognize an event in which a pick up or a put down movement occurs according to a user's selection using the sensor module and the image module.

The inventory manager compares the inventory data including at least one of display position, brand, category, size, weight, number, and price of the plurality of items with the change data generated by an event to compare the items. It can be determined whether the movement of the, opening, correct position or return.

The inventory manager may update the inventory data based on a determination result of comparing the inventory data with the change data.

The item recognition system according to another embodiment of the present invention uses a photo sensor that detects a row position of a plurality of items displayed on a display stand, a load cell sensor that senses a weight, and an image module that acquires image information. The item is managed by comparing the change data and inventory data of the generated item.

In the method of operating an item recognition system for managing items in accordance with an embodiment of the present invention, using the sensor module and the image module to recognize the event for a plurality of items displayed on the shelves and the fluctuation caused by the event And managing the items by comparing data with inventory data related to the plurality of items, wherein the sensor module includes a photo sensor for detecting a row position of the plurality of items, and a weight of the plurality of items. And a load cell sensor, wherein the image module obtains image information about the item.

Recognizing events for the plurality of items may recognize an event in which a pick up or a put down movement occurs according to a user's selection using the sensor module and the image module. have.

The managing of the item may include comparing the inventory data including at least one of display position, brand, category, size, weight, number, and price of the plurality of items with the change data generated by an event. To determine whether the item is moved, opened or not, or whether it is returned or not.

The managing of the item may update the inventory data based on a determination result of comparing the inventory data with the variation data.

According to an embodiment of the present invention, an event of pick up or put down according to a user's selection is performed using a photo sensor, a camera, and a load cell sensor positioned near a plurality of items displayed on a display rack. The generated items can be recognized more accurately.

In addition, according to an embodiment of the present invention, by using the variation data for the item on which the event occurred and the inventory data related to the plurality of items, the item is determined by determining whether the item is moved, opened or not, or whether it is returned or not. You can design your system more reliably and efficiently.

1 is a block diagram of a detailed configuration of an item recognition system according to an embodiment of the present invention.
2 is a diagram for conceptually explaining communication between an item recognition system and a detailed configuration of a sensor module and an image module according to an exemplary embodiment of the present invention.
3 illustrates an application example of an item recognition system according to an exemplary embodiment of the present invention.
4 and 5 illustrate an example of detecting an item using a photo sensor and a camera according to an embodiment of the present invention in detail.
6a to 6c illustrate an example of detecting an item using a load cell sensor according to an embodiment of the present invention in detail.
7A and 7B illustrate an application example of a photo sensor according to an embodiment of the present invention.
8 illustrates an application example of a load cell sensor according to an exemplary embodiment of the present invention.
9 shows an application example of a camera according to an embodiment of the present invention.
10 is a flowchart illustrating an item recognition method according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

Also, the terminology used herein is a term used to properly express a preferred embodiment of the present invention, which may vary depending on a viewer, an operator's intention, or customs in the field to which the present invention belongs. Therefore, the definitions of the terms should be made based on the contents throughout the specification.

1 is a block diagram of a detailed configuration of an item recognition system according to an embodiment of the present invention.

Referring to FIG. 1, an item recognition system according to an embodiment of the present invention recognizes an event for a plurality of items (eg, goods, goods, etc.) displayed on a display rack using a sensor module and an image module. The item is managed based on the change data and the inventory data.

To this end, the item recognition system 100 according to an embodiment of the present invention includes a sensor module 110, an image module 120 and the inventory management unit 130.

The item recognition system 100 according to the exemplary embodiment of the present invention may detect a plurality of items displayed on each of the plurality of shelves by using the sensor module 110 and the image module 120 positioned near the display rack in a specific space. have. In this case, the specific space may be a specific space of a shopping mall, a large mart, a store, a restaurant, or another commercial facility, and represents a space for selling goods or goods. In addition, the display rack may be a kind of furniture displaying a refrigerator or an item (goods).

The sensor module 110 includes a photo sensor that detects a row position of a plurality of items displayed on a display rack, and a load cell sensor that senses the weight of the plurality of items.

The sensor module 110 includes a photo sensor that senses a row position for each of a plurality of items displayed in a plurality of columns on a display stand partitioned by a plurality of rows and a plurality of columns, and a load cell that is positioned at the bottom of the plurality of items to sense weight. It may include a sensor.

For example, the photo sensor may include a light emitting unit for emitting light, such as a light emitting diode (LED) or a laser, and a light receiving unit that is spaced apart from the light emitting unit at predetermined intervals and paired with a light receiving unit. The distance to the body part of the user who selects the item or item in which the event is generated may be measured at each row position and column position of the plurality of items. In this case, the photo sensor may be a photo sensor or an infrared sensor, and in some embodiments, the number of photo sensors may be larger than that of the load cell sensor.

The image module 120 obtains image information about a plurality of items. The image module 120 may be a camera, and may acquire image information about an item in which an event occurs and a body part of a user accessing the item among the plurality of items displayed on the display rack. In addition, the image module 120 may obtain the image information based on inventory data of items displayed on the shelves.

According to an exemplary embodiment, the image module 120 may obtain image information about an item where an event is generated and a body part of the user based on a signal sensed by the sensor module 110. For example, the image module 120 may be in an off state or a sleep mode state, and a sensing signal received from the sensor module 110 or an item recognition system according to an embodiment of the present invention ( On the basis of the signal of 100, the signal may be activated to obtain image information at a corresponding position. Accordingly, the present invention provides the effect of minimizing image processing time mainly by using a sensor and minimizing camera use.

The item recognition system 100 according to an exemplary embodiment of the present invention uses a plurality of items using at least two of the plurality of sensor modules 110 and the plurality of image modules 120 positioned near the plurality of items displayed on the display rack. You can detect events for, and the combination of sensors and cameras can more accurately recognize the user's actions on items, and can quickly detect malfunctions and errors of the device.

The inventory manager 130 recognizes an event for a plurality of items using the sensor module 110 and the image module 120, compares the variation data generated by the event with inventory data related to the plurality of items, and selects the item. Manage. In addition, the inventory management unit 130 may update the inventory data by combining the recognition of the event and the recognition system based on the sensor module 110 and the image module 120.

The inventory manager 130 may recognize an event in which a pick up or a put down or return movement according to a user's selection is generated using the sensor module 110 and the image module 120. In addition, shelf information, location information, and weight information of an item in which an event occurs may be recognized. In this case, the pick-up may be an operation in which the user picks up an item, and the foot down may be an operation in which an item is held down by the user. The inventory manager 130 may be configured to detect a signal sensed by a photo sensor, a camera, and a load cell sensor. The combination may recognize pick up, foot down, shelf information of the item, position information, and weight information of the item.

According to an embodiment, the user may pick up any item of the plurality of items displayed on the shelf, and may put the item back on the shelf. In addition, the user may pick up items displayed on the shelves and then put them down in place, or put them down in a position other than the home position. In addition, the movement of the item may occur due to carelessness of the user. That is, the inventory manager 130 may recognize the event for the above-described situation through the sensor module 110 and the image module 120.

Thereafter, the inventory management unit 130 compares the inventory data including at least one or more of the display position, brand, category, size, weight, number, and price of the plurality of items with the variation data generated by the event to compare the items. It can be determined whether the movement of the, opening, correct position or return.

For example, the item recognition system 100 according to the embodiment of the present invention may further include a database unit 140, and the database unit 140 may include a plurality of items displayed on each of the plurality of shelves in a specific space. About display location, brand, category, size, weight, number, price, discount information, expiration date, receipt date, inventory date, identification information, distributor information, manager information, notice information, pre-opening image, image after opening and Inventory data including at least one of the up / down / left / right images may be stored and maintained. At this time, the inventory data stored and maintained in the database unit 140 may be updated such as deletion, change, addition, warning, etc. by an event generated in real time.

Accordingly, the inventory management unit 130 compares the inventory data stored and maintained in the database unit 140 with the variation data for the corresponding item generated by an event of a pick up or put down. In accordance with the comparison result, it is possible to determine whether the item is moved, opened or not, whether it is positioned or returned.

For example, when a pick-up event of an item occurs, the inventory manager 130 may determine a movement by comparing variation data and inventory data of the corresponding item according to the pick-up event.

As another example, when a put down event of the item occurs, the inventory management unit 130 determines the return position of the corresponding item according to the foot down event, and compares the variation data according to the return position with the inventory data to determine the return, At the same time, it can be determined whether it is opened. According to an embodiment, assuming that the item is bottled water, the user may open the bottled water and consume it in a portion, and then return the bottled water to a position other than the correct position. Accordingly, when the foot down event occurs, the inventory management unit 130 may determine whether to return, open and correct position.

Thereafter, the inventory manager 130 may update the inventory data based on the determination result of the comparison between the inventory data and the variation data. For example, the inventory manager 130 may update the inventory data such as deletion, change, addition, or warning based on whether the item is moved, opened, or not or returned according to the comparison result.

According to an embodiment, when the user selects an item, the inventory manager 130 may delete or change the corresponding item from the inventory data, and when the user puts the item down to a position other than the correct position, the inventory manager ( 130 may add or warn the corresponding item in the inventory data. When an opening event of an item or an event that is returned to a location other than the correct location occurs, the inventory management unit 130 may update the inventory data in the database unit 140 with a warning phrase for the corresponding item, and in a specific space An administrator for managing the device may send a warning signal for the corresponding item to a mobile device, a server in a specific space, or an item management server.

At this time, the mobile device possessed by the administrator is a terminal device such as a personal computer, a laptop computer, a smart phone, a tablet PC, a wearable computer, and the like. It may refer to all terminal devices that can communicate with the recognition system 100 to install a site dedicated to a site related to a specific space or to execute a service-only application. The mobile device may perform overall service operations such as service screen configuration, data input, data transmission, data storage, payment, and billing under the control of a web / mobile site or a dedicated application.

In addition, the item recognition system 100 according to an embodiment of the present invention may transmit information and signals related to inventory data updated to a mobile device, a server in a specific space, or an item management server possessed by an administrator through a communication module. . At this time, the communication module may be a cellular telephone network, a wireless local area network (LAN) and / or a wireless network such as a metropolitan area network (MAN), a network such as an intranet and / or the Internet, also called the World Wide Web (WWW), and the like. The device may communicate with a device by wireless communication, and the wireless communication may include Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), wideband code division multiple access (WCDMA), code division multiple access (CDMA), and TDMA. (time division multiple access), Wireless Fidelity (such as IEEE 802.11a, IEEE 802.11b, IEEE 802.11g and / or IEEE 802.11n), voice over Internet Protocol (VoIP), WiMAX, Long Term Evolution (LTE) , E-mail protocols such as Zigbee, Zwave, Bluetooth Low Energy (BLE), Beacon, Internet Message Access Protocol (IMAP) and / or Post Office Protocol (POP), eXtensible Messaging and Presence Protocol (XMPP) , SIMPLE (Session Initiation P instant messaging, such as rotocol for Instant Messaging and Presence Leveraging Extensions (IMPS), Instant Messaging and Presence Service (IMPS), or Short Message Service (SMS), or a communication protocol not developed at the time of filing of the present application. However, a plurality of communication standards, protocols, and technologies, which are not limited thereto, may be used in the above-described wireless communication.

2 is a diagram for conceptually explaining communication between an item recognition system and a detailed configuration of a sensor module and an image module according to an exemplary embodiment of the present invention.

The item recognition system 100 according to the exemplary embodiment of the present invention may include a photo sensor 10, a camera 20, and a load cell sensor 30. In FIG. 2, the photo sensor 10, the camera 20, and the load cell sensor 30 are illustrated in the singular but may be plural in number.

The item recognition system 100 according to the exemplary embodiment of the present invention may accurately recognize an act of purchasing or returning an item using the photo sensor 10, the camera 20, and the load cell sensor 30.

Referring to FIG. 2, the item recognition system 100 according to an exemplary embodiment of the present invention may include a photo sensor 10 using a USB, a universal asynchronous receiver / transmitter (USBUART), or a Bluetooth communication scheme. ), Location information, weight information, event information, and image information of the item may be received from each of the camera 20 and the load cell sensor 30, and the event of the item and the movement of the user's body part using the received information. Can be recognized. In addition, the item recognition system 100 according to an embodiment of the present invention may be connected to the photo sensor 10, the camera 20, and the load cell sensor 30 through a wired / wireless communication channel to transmit and receive information.

According to an embodiment, the item recognition system 100 according to an embodiment of the present invention may check in advance whether the sensor module and the image module operate before the operation.

For example, when the photo sensor 10 is not operated, the item recognition system 100 may use the load cell sensor 30 to detect an event (pick up or foot down) of an item in a specific shelf and a specific column (vertical column). Recognize and use the weight information in the inventory data to recognize the product group (for example, 500ml or 350ml, etc.) for which item occurred. In addition, the item recognition system 100 may re-recognize an event (pick up or foot down) of the item using the image information obtained from the camera 20.

As another example, when the camera 20 is not operated, the item recognition system 100 may use the photo sensor 10 and the load cell sensor 30 to display an event (pickup) of an item on a specific shelf and a specific column (vertical column). Or foot down), in particular, the load cell sensor 30 may be used to detect one event or two or more events for the item. In addition, the item recognition system 100 may recognize an item to be put down at a position other than the correct position using the photo sensor 10.

As another example, when the load cell sensor 30 does not operate, the item recognition system 100 may use the photo sensor 10 and the inventory data to store an event (pickup or Foot down), and using the image information obtained from the camera 20 to recognize the movement path of the item to select the item (pick up) or returning (foot down) can be recognized. have. For example, the item recognition system 100 may detect whether the item has moved in the shelf direction or the shelf direction by using the image information obtained from the camera 20.

3 illustrates an application example of an item recognition system according to an exemplary embodiment of the present invention.

Referring to FIG. 3, an item recognition system according to an exemplary embodiment of the present invention includes a plurality of photo sensors (not shown), a camera 20, and a load cell sensor (not shown) included in a display stand in a specific space 1. .

For example, the display rack located in the specific space 1 may include a plurality of shelves, and each shelf may include a plurality of items 40 arranged in a plurality of rows. In this case, a plurality of areas for dividing the various types of items 40 may be positioned on a single shelf, and the plurality of items 40 may be displayed in rows according to the types. In this case, the area may be a tray or a partition for dividing the item 40 by a category, type, or trademark on a shelf, and may mean a space classified by the partition wall 830 illustrated in FIG. 8.

According to an embodiment, the item 40 may be not only displayed as shown in FIG. 3 but also stacked.

In addition, a load cell sensor for detecting the weight of the plurality of items 40 is positioned between each shelf of the display rack in the specific space 1 and the plurality of items 40, that is, the lower portion of the area where the plurality of items 40 are displayed. can do. In addition, an upper portion of the area where the plurality of items 40 are displayed, image information of the plurality of items 40, image information of the item 41 in which an event occurs, and a body part of the user who selects the item 40. The camera 20 for acquiring image information may be located. In addition, a photo sensor may be positioned on a side of each shelf of the display rack to detect the position of the first row of the plurality of items 40 displayed in a plurality of columns and to measure the distance of the item 41 in which an event occurs.

At this time, the item recognition system 100 according to an embodiment of the present invention may be characterized by including a greater number of photosensors than the load cell sensor.

4 and 5 illustrate an example of detecting an item using a photo sensor and a camera according to an embodiment of the present invention in detail.

More specifically, FIG. 4 illustrates an example of an item displayed on a display stand, an arrangement example of a sensor module and an image module, and FIG. 5 illustrates an example of sensing a sensor module for an item on which an event has occurred. It is.

A plurality of display racks are located in a specific space, and the display racks may be divided into a plurality of shelves by the plurality of shelves 410. In this case, an area for classifying and displaying the plurality of items 40 by category, type, or trademark is present on the shelf 410, and the area may be a tray or a partition.

Taking the shelf 410 as an example with reference to FIG. 4, the plurality of items 40 may be classified according to different types, categories, brands, numbers, and sizes, and displayed in a plurality of rows. The plurality of cameras 20 may be positioned at an upper end thereof, and the photo sensor 10 at a row position may be positioned at side surfaces of the plurality of items 40. At this time, the position and the number of the camera 20 and the photo sensor 10 is not limited.

The camera 20 may include a left camera 20 _L , a center camera 20 _C , and a right camera 20 _R , and image information and events of the plurality of items 40 displayed on the shelf 410. At least one of the image information on the generated item 41 and the image information on the body part of the user who selects the item 40 may be obtained.

Furthermore, the item recognition system according to the embodiment of the present invention, as well as the photo sensor 10 of the row position located on the side of the shelf 410, the photo sensor 10 for detecting the column position of the plurality of items 40 It may further include, the photo sensor 10 of the row position and the photo sensor 10 of the column position detects the item 41, the event occurred in the row position and column position, the item 41, the event occurred Measure a distance) and detect movement of a part of the user's body to select an item.

Referring to FIG. 5, it can be seen that the photo sensor 10 is located in the row position direction of the plurality of items 40 sorted by regions and displayed in a plurality of columns. According to the exemplary embodiment, the photo sensor 10 may include a light emitting unit for emitting light, such as a light emitting diode (LED) or a laser, and a light receiving unit that is spaced apart from the light emitting unit at predetermined intervals, and the event is By using the signal reflected to the generated item 41 and received by the light receiving unit, the motion of the item 41 and the body part of the user who selects the item are detected, and the event of the item 41 is generated. You can measure the distance.

The item recognition system according to an embodiment of the present invention adds different identification information to each of a plurality of shelves of the display rack and the photo sensor 10, the camera 20, and a load cell sensor (not shown) positioned on the shelves 410. And event and variation data of the item 40 based on information received from the photo sensor 10, the camera 20, and a load cell sensor (not shown), and respective identification information. Shelf information, location information, weight information, image information, etc.) can be recognized more quickly and accurately.

6a to 6c illustrate an example of detecting an item using a load cell sensor according to an embodiment of the present invention in detail.

The load cell sensor 30 according to an exemplary embodiment of the present invention may be positioned between the shelf 410 and the plurality of items 40, that is, at the bottom of the plurality of items 40 in FIG. 4.

The load cell sensor 30 may be a weight sensor and may detect the weight of the plurality of items 40. For example, the item recognition system according to an embodiment of the present invention stores and maintains inventory data related to a plurality of items, and the inventory data may include weight information for each of the plurality of items 40. Accordingly, the load cell sensor 30 may detect the weight of the plurality of items 40 in real time, the item recognition system according to an embodiment of the present invention by comparing the weight information and inventory data detected in real time item It is possible to determine whether or not the opening of the 40 (40).

The load cell sensor 30 according to the embodiment of the present invention may be located at the bottom of the plurality of items 40 displayed on the same shelf as shown in FIG. 6A, and as shown in FIG. 6B, the category, It may be located at the bottom of the item 40 classified according to the type or trademark. In addition, referring to FIG. 6C, the load cell sensor 30 may be located at the lower end of each column of the plurality of items 40 displayed. According to an embodiment, since the load cell sensors 30 may be stacked on a shelf, and the plurality of items 40 may be displayed on the stacked load cell sensors 30, the number and arrangement of the load cell sensors 30 may be different. Is not limited to this.

The item recognition system according to the embodiment of the present invention is characterized by including the number of load cell sensors 30 less than the number of photosensors and cameras. This is to reduce the number of load cell sensor 30, to provide an effect that can recognize the event of a more accurate item using a camera and a photo sensor.

7A and 7B illustrate an application example of a photo sensor according to an embodiment of the present invention.

More specifically, FIG. 7A illustrates an example of a sensor board including a photo sensor, and FIG. 7B illustrates an example of a display rack to which a sensor board is applied.

Referring to FIG. 7A, the sensor board 710 may include a light emitting unit 711 and a light receiving unit 712. In the present invention, the sensor board 710 may include a light emitting unit 711 and a light receiving unit 712.

The light emitting unit 711 emits light such as a light emitting diode (LED) or a laser, and the light receiving unit 712 is spaced apart from the light emitting unit 711 at predetermined intervals to form a pair, and the light emitting unit 711 is provided. It is irradiated by and receives light that is reflected by the item or body part of the user and is incident. In this case, the light emitter 711 and the light receiver 712 may be repeatedly arranged at a predetermined interval, and the predetermined interval may be variously applied according to the size and type of the display rack or the type of the item. At this time, the type, shape and number of the light emitting part 711 and the light receiving part 712 are not limited, and the material, substrate type and shape of the sensor board 710 are not limited.

Referring to FIG. 7B, the sensor board 710 may be attached to all sides of the display rack and may be connected and fixed by the harness 720.

The item recognition system 100 according to an exemplary embodiment of the present invention detects an item in which an event is generated from the photo sensors 711 and 712 included in each of the plurality of sensor boards 710, and measures the distance of the item in which the event is generated. The measurement may be performed to detect a movement and a distance of the body part of the user who wants to select an item.

According to an exemplary embodiment, each of the plurality of sensor boards 710 may include different identification information, and the item recognition system according to an exemplary embodiment of the present invention may include sensing information and distance information received from the photo sensors 711 and 712. And it is possible to more accurately detect the variation data for the item on which the event occurred based on the identification information. The variation data may include at least one of shelf information, position information, weight information, and image information of an item in which an event occurs.

8 illustrates an application example of a load cell sensor according to an exemplary embodiment of the present invention.

Referring to FIG. 8, display shelves are classified into a plurality of shelves 810, and a plurality of load cell sensors 820 may be disposed on the plurality of shelves 810. In this case, each shelf 810 may include a partition 830 for classifying items, which may be formed in plurality on the shelf 810. According to an exemplary embodiment, the load cell sensor 820 may be disposed on the shelf 810 in a large area, and may be disposed below the region classified by the partition wall 830. That is, as illustrated in FIGS. 3 to 5, the plurality of items arranged in a plurality of columns may mean a space classified by the partition wall 830, and the area is included according to an embodiment to which the present invention is applied. It may or may not be included.

The load cell sensor 820 may detect the weight of a plurality of items displayed on the shelf 810, and the item recognition system 100 according to an embodiment of the present invention may display a plurality of items displayed on the shelf 810. The weight according to the number of can be recognized. For example, assuming that the shelf 810 item is bottled water, when the user ingests one bottled water and then puts it down, or when the user picks up two or more bottled water at the same time, the load cell sensor 820 Detects a change in weight, and the item recognition system 100 according to an embodiment of the present invention determines whether the item is moved, opened or not, by using information received from the camera and the photo sensor based on the change in weight. Alternatively, it may be determined whether to return, bill the user according to the determination result, or update the inventory data.

The load cell sensor 820 may detect movement of an item by a user. For example, when a user picks up the bottled water displayed on the top shelf and puts it down to the bottom shelf, the load cell sensor 820 of the top shelf detects a decrease in weight, and the load cell sensor 820 of the bottom shelf is weighed. It can detect the increase of. However, in this case, the item recognition system 100 according to an embodiment of the present invention further recognizes the information received from the camera and the photo sensor in addition to the weight information to determine whether the item is moving, whether or not the position is correct or return In addition, it may be determined that the user does not have the item and may not be treated as a purchase.

The item recognition system 100 according to an exemplary embodiment of the present invention determines whether an item is evented by using a camera and a photo sensor in addition to the weight information received from the load cell sensor 820, and the item is not displayed in the correct position or the weight. If there is a change, information of the corresponding item may be transmitted to an administrator or a server managing a specific space.

9 shows an application example of a camera according to an embodiment of the present invention.

Referring to FIG. 9, a plurality of cameras 910 may be disposed on the wall surface of the display rack. According to an embodiment, the camera 910 may be Microsoft's 720P-class PC CAM.

The camera 910 is in an off or sleep mode, and is based on a sensing signal received from a photo sensor and a load cell sensor or a signal of an item recognition system 100 according to an embodiment of the present invention. On the basis of this, it can be activated to obtain image information at the corresponding position.

The item recognition system 100 according to the embodiment of the present invention minimizes the use of the camera 910 and mainly uses a sensor module (photo sensor and a load cell sensor), thereby minimizing the image processing time of the camera. Provide effect.

More specifically, the item recognition system 100 according to an embodiment of the present invention detects an item on which a movement and an event of a body part of a user are generated using a photo sensor and a load cell sensor, and a corresponding position where the event is generated. The camera 910 is activated to obtain image information of an item on which a movement and an event of a user's body part are generated. That is, the camera 910 according to an embodiment of the present invention may be activated when a movement of a body part of a user who selects an item and an event of the item are generated, thereby improving the speed of image processing. The processing capacity can be managed efficiently.

10 is a flowchart illustrating an item recognition method according to an embodiment of the present invention.

The method shown in FIG. 10 is performed by the item recognition system according to the embodiment of the present invention shown in FIG.

Referring to FIG. 10, in step 1010, an event for a plurality of items displayed on a shelf is recognized using a sensor module and an image module.

In this case, the sensor module may include a photo sensor for detecting a row position of a plurality of items, and a load cell sensor for sensing the weight of the plurality of items. The image module may be a camera that acquires image information about an item.

In operation 1020, the item is managed by comparing the variation data generated by the event with inventory data related to the plurality of items.

In operation 1020, the sensor module and the image module are used to recognize an event in which a pick up or put down movement occurs according to a user's selection, and display position, brand, and category of the plurality of items. , Inventory data including at least one of size, weight, number and price, and fluctuation data generated by the event are compared, and whether the item is moved, opened, placed or returned according to the comparison result. Determining and updating the inventory data.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the devices and components described in the embodiments may be, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to the execution of the software. For convenience of explanation, one processing device may be described as being used, but one of ordinary skill in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiment may be embodied in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

10: photo sensor
20, 910: image module (or camera)
30, 820: load cell sensor
40: items
41: Item that triggered the event
410, 810 shelves
710: sensor board
711: light emitting unit
712: light receiver
720: harness
830: bulkhead

Claims (14)

A sensor module including a photo sensor detecting a row position of a plurality of items displayed on a display rack, and a load cell sensor sensing a weight of the plurality of items;
An image module for obtaining image information about the plurality of items; And
An inventory management unit that recognizes events for the plurality of items using the sensor module and the image module, and manages the items by comparing the variation data generated by the event with inventory data related to the plurality of items.
Item recognition system comprising a. The method of claim 1,
The sensor module
The photo sensor for detecting a row position for each of the plurality of items displayed in a plurality of columns on the display stand partitioned by a plurality of rows and a plurality of columns, and the load cell sensor positioned at the bottom of the plurality of items to sense weight Item recognition system comprising a. The method of claim 2,
The photo sensor is
And a light emitting unit for emitting light and a light receiving unit spaced apart from the light emitting unit at a predetermined interval to form a pair to measure a distance of an item in which an event occurs among the plurality of items displayed in a plurality of columns. The method of claim 1,
The image module
Characterized in that the camera,
And the camera acquires the image information of the displayed item among the plurality of items displayed on the display rack, the item on which the event occurs, and the body information of the user's body part accessing the item. The method of claim 4, wherein
The inventory management unit
An item recognition system that updates the inventory data by combining the event recognition with the sensor module and the image module based recognition system. The method of claim 1,
The inventory management unit
An item recognition system using the sensor module and the image module to recognize the event that the pick-up or put down movement occurs according to the user's selection. The method of claim 6,
The inventory management unit
Comparing the inventory data including at least one of display position, brand, category, size, weight, number, and price of the plurality of items with the change data generated by an event; Item recognition system for determining whether to open, in-place or return. The method of claim 7, wherein
The inventory management unit
And updating the inventory data based on a determination result of the comparison of the inventory data and the change data. Comparison of inventory data with fluctuation data on the item where an event occurred by using a photo sensor for detecting a row position of a plurality of items displayed on a display stand, a load cell sensor for sensing a weight, and an image module for obtaining image information Item management system, characterized in that for managing the item. In the operation method of the item recognition system for managing the item,
Recognizing events for the plurality of items displayed on the display rack using the sensor module and the image module; And
Comparing the change data generated by the event with inventory data related to the plurality of items to manage the items,
The sensor module includes a photo sensor for detecting a row position of the plurality of items and a load cell sensor for sensing the weight of the plurality of items, wherein the image module obtains image information about the item. Item recognition method to do. The method of claim 10,
Recognizing events for the plurality of items
And recognizing an event in which a pick-up or a put-down movement occurs according to a user's selection by using the sensor module and the image module. The method of claim 11,
Managing the item
Comparing the inventory data including at least one of display position, brand, category, size, weight, number, and price of the plurality of items with the change data generated by an event; Item recognition method for determining whether to open, in place, or return. The method of claim 12,
Managing the item
And updating the inventory data based on a determination result of comparing the inventory data and the change data. A computer program stored in a computer readable recording medium for carrying out the method of claim 10.

Images