JP2020119381A - Shelf allocation information generating apparatus and shelf allocation information generating program - Google Patents

Abstract

To reduce an error of recognition of commodities and prices from shelf allocation images in generation of shelf allocation information being the arrangement information on the exhibited commodities, and thereby improve the accuracy of the shelf allocation information acquired.SOLUTION: A shelf allocation information generating apparatus 10 includes: an acquisition unit 11 for acquiring shelf allocation images comprising commodity images and price tag images of commodities arranged for sale; a commodity recognition unit 12 for recognizing the commodities on the basis of the commodity images; a price tag recognition unit 13 for recognizing prices on the basis of the price tag images; a commodity information acquisition unit 14 for acquiring prices associated with the recognized commodities, with reference to the commodity information; a determination unit 15 for determining whether or not the recognized price falls under the price of the commodity acquired by the commodity information acquisition unit 14; and an output unit 16 for associating the commodity with the recognized price to output it as the shelf allocation information, when determined that the price recognized by the price tag recognition unit 13 falls under the price of the commodity by determination unit.SELECTED DRAWING: Figure 1

Description

The present invention relates to a shelving allocation information generation device and a shelving allocation information generation program.

The so-called shelving allocation information including the information on the products arranged on each shelf and the price of the products is acquired by recognizing the products from the shelving allocation image which is an image of the display shelves of the products. There is. For example, a technique is known that generates shelving allocation information representing a product display state based on a product recognized from an image including a display shelf and information on a designated position of the product (for example, see Patent Document 1).

JP, 2016-224831, A

In the generation of the shelving allocation information based on the shelving allocation image, since the product and the price are recognized by the image recognition processing for the shelving allocation image, the recognition may be incorrect. On the other hand, it is required to correct the recognition error. In addition, there is a technique for determining whether a product or a price tag arranged on a display shelf or the like is aligned with a valid product or a price tag on the assumption that the recognition from the image is performed without error. Such techniques, of course, do not correct recognition errors from the image.

Therefore, the present invention has been made in view of the above problems, and in the generation of shelving allocation information, which is array information of displayed products, reduces errors in recognizing products and prices from shelving allocation images. An object of the present invention is to provide a shelving allocation information generation device and a shelving allocation information generation program that can improve the accuracy of the acquired shelving allocation information.

In order to solve the above problems, a shelving allocation information generation device according to an aspect of the present invention is a shelving allocation information generation device that generates shelving allocation information that is array information of displayed products, and displays shelving allocation images. In the acquisition unit for acquiring, the shelving allocation image is an image of products arranged for sale, and includes a product image representing each product and a price tag image displaying the price of each product, An acquisition unit, a product recognition unit that recognizes a product based on a product image, a price tag recognition unit that recognizes the price of a product based on a price tag image having a predetermined relationship with the product recognized by the product recognition unit, and a product The product information acquisition unit that acquires the price associated with the product recognized by the product recognition unit and the price recognized by the price tag recognition unit are The determination unit that determines whether or not the price of the product acquired by the information acquisition unit is determined, and the price tag recognition unit when the determination unit determines that the price recognized by the price tag recognition unit corresponds to the price of the product. And an output unit that associates the price recognized by the product with the product and outputs it as shelving allocation information.

Further, the shelving allocation information generation program according to an aspect of the present invention causes shelving allocation information to cause a computer to function as a shelving allocation information generation device that generates shelving allocation information that is arrangement information of products arranged on a product shelf. The generating program is an acquisition function of acquiring a shelving allocation image in a computer, and the shelving allocation image is an image of products arranged for sale, and a product image showing each product and Based on an acquisition function including a price tag image showing the price of each product, a product recognition function for recognizing a product based on the product image, and a price tag image having a predetermined relationship with the product recognized by the product recognition function. A price tag recognition function for recognizing the price of a product, and a product information acquisition function for acquiring the price associated with the product recognized by the product recognition function based on the product information including the price information of the product associated with the product And the judgment function to judge whether the price recognized by the price tag recognition function corresponds to the price of the product acquired by the product information acquisition function, and the price recognized by the price tag recognition function corresponds to the price of the product. Then, when it is determined by the determination unit, an output function of associating the price recognized by the price tag recognition function with the product and outputting it as shelving allocation information is realized.

According to the above aspect, the product and the price are recognized from the product image and the price tag image included in the acquired shelving allocation image, and the recognized price corresponds to the price associated in advance with the recognized product. It is determined whether or not. If it is determined to be applicable, it is highly likely that both the recognition of the product and the recognition of the price are valid, so by outputting the product and the price determined to be applicable in association with each other, highly accurate shelving allocation information is output. Can be generated.

When generating the shelving allocation information which is the arrangement information of the displayed commodities, it is possible to reduce the error in recognizing the products and the prices from the shelving allocation image and improve the accuracy of the acquired shelving allocation information.

It is a block diagram which shows the functional structure of the shelving allocation information generation apparatus of this embodiment. It is a hardware block diagram of a shelving allocation information generation device. It is a figure which shows the example of a shelf allocation image. It is a figure which shows typically the example of the extraction of the product image and price tag image from a shelf allocation image, and the recognition of the product and price from each of a product image and a price tag image. It is a figure which shows the example of the product information which linked|related the product and the price of the product. It is a figure which shows the example of extraction of the price of goods based on recognition of goods and price, and calculation of an index value. It is a figure showing an example of extraction of the price of goods based on recognition of goods and price, calculation of an index value, and judgment of whether the recognized price corresponds to the price of the recognized goods. It is a figure showing an example of extraction of the price of goods based on recognition of goods and price, calculation of an index value, and judgment of whether the recognized price corresponds to the price of the recognized goods. It is a figure which shows the example of acquisition of price likelihood information based on the price information of the goods contained in goods information. It is a figure showing an example of extraction of the price of goods based on recognition of goods and price, calculation of an index value, and judgment of whether the recognized price corresponds to the price of the recognized goods. It is a figure which shows typically the example of the product image and price tag image extraction from a shelf allocation image, the recognition of the product from a product image, and the recognition of the price and the product name from the price tag image. It is a figure which shows the example of correction|amendment of the likelihood of recognition of goods using the likelihood of recognition of the goods name from a price tag image. It is a flowchart which shows the processing content of the shelf allocation information generation method of this embodiment. It is a flowchart which shows the processing content of the shelf allocation information generation method of this embodiment. It is a figure which shows the structure of a shelf allocation information generation program.

An embodiment of a shelving allocation information generation device according to the present invention will be described with reference to the drawings. It should be noted that, if possible, the same portions will be denoted by the same reference symbols and redundant description will be omitted.

FIG. 1 is a diagram showing a functional configuration of a shelving allocation information generation system 1 including a shelving allocation information generation device 10 according to this embodiment. The shelving allocation information generation device 10 is a device that generates shelving allocation information that is array information of the displayed products.

As shown in FIG. 1, the shelving allocation information generating system 1 includes a shelving allocation information generating device 10, an image pickup/display device 20, a product information storage unit 30, and a shelving allocation information storage unit 40.

The shelving allocation information generation system 1 may be configured as one device, or may be one of the shelving allocation information generating device 10, the imaging/display device 20, the product information storage unit 30, and the shelving allocation information storage unit 40. A plurality of devices may each constitute one device. As an example, the imaging/display device 20 may be configured as one terminal, and the shelving allocation information generating device 10, the product information storage unit 30, and the shelving allocation information storage unit 40 may be configured by a server. Further, as another example, the shelving allocation information generation device 10 and the imaging/display device 20 may be configured as one terminal. Each storage unit of the product information storage unit 30 and the shelving allocation information storage unit 40 may be configured in any form as long as it is accessible from the shelving allocation information generating device 10. When the imaging/display device 20 is configured as one terminal, the photographer can refer to the display screen while photographing the display shelves of the products and refer to the shelving allocation information generated through image recognition. Further, in the example of the terminal including only the display (display device) 22, it is assumed that the operator or the like confirms and corrects the shelving allocation information generated by the image recognition in the office or the like, not in the actual store.

In the present embodiment, the terminal configuring the imaging/display device 20, or the terminals configuring the shelving allocation information generation device 10 and the imaging/display device 20, is, for example, a mobile phone such as a high-performance mobile phone (smartphone) or a mobile phone. Configured as a terminal.

As shown in FIG. 1, the shelving allocation information generation device 10 functionally includes an acquisition unit 11, a product recognition unit 12, a price tag recognition unit 13, a product information acquisition unit 14, a determination unit 15, and an output unit 16. The image pickup/display device 20 also includes a camera 21 as an image pickup device and a display 22 as a display device. Each of these functional units will be described in detail later.

The block diagram shown in FIG. 1 shows blocks of functional units. These functional blocks (components) are realized by an arbitrary combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device physically or logically coupled, or directly or indirectly (for example, two or more devices physically or logically separated). , Wired, wireless, etc.) and may be implemented using these multiple devices. The functional blocks may be realized by combining the one device or the plurality of devices with software.

Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, resolution, selection, selection, establishment, comparison, assumption, expectation, observation, Broadcasting, notifying, communicating, forwarding, configuration, reconfiguring, allocating, mapping, assigning, etc., but not limited to these. I can't. For example, a functional block (component) that causes transmission to function is called a transmitting unit or a transmitter. In any case, as described above, the implementation method is not particularly limited.

For example, the shelving allocation information generation device 10 according to the embodiment of the present invention may function as a computer. FIG. 2 is a diagram showing an example of a hardware configuration of the shelving allocation information generation device 10 according to the present embodiment. The shelving allocation information generation device 10 may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the shelving allocation information generation device 10 may be configured to include one or a plurality of each device illustrated in the figure, or may be configured not to include some devices.

Each function in the shelving allocation information generation device 10 causes a predetermined software (program) to be loaded on hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs an operation, communication by the communication device 1004, and memory 1002. It is realized by controlling the reading and/or writing of data in the storage 1003.

The processor 1001 operates an operating system to control the entire computer, for example. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, a calculation device, a register, and the like. For example, the functional units 11 to 16 shown in FIG. 1 may be realized by the processor 1001.

Further, the processor 1001 reads a program (program code), a software module, and data from the storage 1003 and/or the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least part of the operations described in the above-described embodiments is used. For example, the functional units 11 to 16 of the shelving allocation information generation device 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001. Although it has been described that the various processes described above are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from the network via an electric communication line.

The memory 1002 is a computer-readable recording medium, and is configured by, for example, at least one of ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), and the like. May be done. The memory 1002 may be called a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store an executable program (program code), a software module, etc. for implementing the shelving allocation information generating method according to the embodiment of the present invention.

The storage 1003 is a computer-readable recording medium, for example, an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disc, a magneto-optical disc (for example, a compact disc, a digital versatile disc, a Blu-ray disc). (Registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like. The storage 1003 may be called an auxiliary storage device. The storage medium described above may be, for example, a database including the memory 1002 and/or the storage 1003, a server, or another appropriate medium.

The communication device 1004 is hardware (transmission/reception device) for performing communication between computers via a wired and/or wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that performs output to the outside. The input device 1005 and the output device 1006 may be integrated (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured with a single bus or different buses among devices.

The shelving allocation information generating device 10 is a hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). May be included, and a part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented with at least one of these hardware.

Referring back to FIG. 1, each functional unit of the shelving allocation information generation device 10 will be described. The acquisition unit 11 acquires a shelving allocation image. The shelving allocation image is an image of products arranged for sale, and includes a product image representing each product and a price tag image displaying the price of each product. Specifically, the shelving allocation image is, for example, an image of a product display shelf and a vending machine in a store or the like.

FIG. 3 is a diagram showing an example of the shelving allocation image PM0 of the display shelves of the products acquired by the acquisition unit 11. Specifically, when the display shelf of the product is captured by the camera 21, the acquisition unit 11 acquires the image captured by the camera 21 as a shelf allocation image. As shown in FIG. 3, the shelf allocation image PM0 represents a product shelf, a plurality of products arranged on each shelf, and a price tag of each product. That is, the shelving allocation image includes a product image in which each product is displayed and a price tag image in which the price of each product is displayed.

The product recognition unit 12 recognizes a product based on the product image. Specifically, the product recognition unit 12 detects a product image showing a product from the shelving allocation image acquired by the acquisition unit 11. The product recognition unit 12 recognizes an object extracted from the shelving allocation image by using, for example, a well-known method such as edge detection, as a product image representing a product. Further, the product recognition unit 12 learns the shape of each product in advance by a well-known method such as deep learning, and detects the product image from the shelving allocation image using the learned data. The method used to detect the product image from the shelving allocation image is not limited to the above example, and any method may be used as long as each product image can be detected.

The product recognition unit 12 recognizes the product represented by the detected product image based on the information about the product image stored in advance. For example, the product recognition unit 12 may recognize the product represented in the product image based on the product appearance data that represents the appearance of the product. The product appearance data can include a plurality of image data representing the appearance of one product from various directions.

The product recognition unit 12 can recognize the product represented by the product image by matching the product appearance data with the detected product images by a well-known matching technique. Further, the recognition of the product is not limited to the above matching technique and the like, and any method may be used. For example, the product recognition unit 12 may learn the appearance of various products in advance by a method such as deep learning, and recognize the products represented by each product image.

The price tag recognition unit 13 recognizes the price of the product based on the price tag image. Specifically, the price tag recognition unit 13 detects a price tag image in which the price of each product is displayed from the shelving allocation image acquired by the acquisition unit 11. For example, the price tag recognition unit 13 extracts, from the shelving allocation image, a rectangular area having a high possibility of being an area representing a price tag by a known method such as edge detection. The price tag recognition unit 13 may also learn the shape of the price tag in advance by a well-known method such as deep learning, and use the learned data to detect the price tag image from the shelving allocation image. The method used to detect the price tag image from the shelving allocation image is not limited to the above example, and any method can be used as long as each price tag image can be detected.

The price tag recognition unit 13 recognizes the price of the product from the detected price tag image. Specifically, the price tag recognition unit 13 recognizes the price from the price tag image by using, for example, a well-known image character recognition technique.

FIG. 4 is a diagram schematically illustrating an example of extracting a product image and a price tag image from a shelving allocation image and recognizing a product and a price from each of the product image and the price tag image. FIG. 4 illustrates an example of recognition of products and prices based on the shelving allocation image pm1 which is a part of the shelving allocation image acquired by the acquisition unit 11.

The product recognition unit 12 detects the product image pm11 from the shelf allocation image pm1. Further, the price tag recognition unit 13 may extract a price tag image having a predetermined relationship with the product recognized by the product recognition unit 12. In the example illustrated in FIG. 4, the price tag recognition unit 13 extracts the price tag image pm12 located within a predetermined distance in the shelf allocation image from the product image pm11 recognized by the product recognition unit 12.

In the present embodiment, in the shelving allocation image, in view of the high possibility that the price tag of the product is arranged close to the corresponding product, the price tag recognition unit 13 determines a predetermined value from the product image of the recognized product. The price tag image located within the distance of is extracted, and the price is recognized based on the extracted price tag image. As a result, the price is recognized from the price tag image that is highly likely to have a corresponding relationship with the product recognized from the product image.

If the identification information (for example, the product name) for identifying the product based on the price tag image is recognizable, the price tag recognition unit 13 has the identification information for identifying the product recognized by the product recognition unit 12. The price tag image may be extracted and the price may be recognized based on the extracted price tag image. In this case, it is possible to recognize the price from the price tag image having a high possibility of having a correspondence relationship with the product recognized from the product image.

As shown in FIG. 4, the product recognition unit 12 recognizes a product based on the product image pm11, and outputs a product recognition result ra01 including the recognized product candidate product name. In addition, the product recognition unit 12 acquires a product recognition likelihood indicating the likelihood of recognition for each product candidate. In the example illustrated in FIG. 4, the product recognition result ra01 includes “drink A 500 ml”, “drink A 1.0 L”, and “drink C 500 ml” as the product names of the product candidates, and each product candidate has a product. The recognition likelihoods “0.6”, “0.2”, and “0.1” are associated.

Further, as shown in FIG. 4, the price tag recognition unit 13 recognizes the price based on the price tag image pm12, and outputs the price recognition result rb01 including the recognized price candidate. Further, the price tag recognition unit 13 acquires the price recognition likelihood indicating the recognition certainty for each price candidate. In the example illustrated in FIG. 4, the price recognition result rb01 includes “120”, “1200”, and “140” as price candidates, and the price recognition likelihoods “0.8” and “0” are included in the price candidates, respectively. . 1” and “0.05” are associated.

Referring again to FIG. 1, the product information acquisition unit 14 acquires the price associated with the product recognized by the product recognition unit 12 based on the product information including the price information of the product associated with the product. To do. In the present embodiment, the product information is stored in the product information storage unit 30, and the product information acquisition unit 14 refers to the product information that associates the product with the price of the product in advance, and the product recognition unit 12 The price associated with the product recognized by is acquired.

FIG. 5 is a diagram showing an example of product information stored in the product information storage unit 30. The product information pd1 in the present embodiment includes information indicating what product was sold at what time at what price, and includes the product name, date, and price in association with each other. As shown in FIG. 5, the product information pd1 includes, for example, product information “product name: drink A 350 ml, date: 2018/1/1, price: 110”.

For example, when the product name of the product recognized by the product recognition unit 12 is “drink A 350 ml”, the product information acquisition unit 14 refers to the product information pd1 and associates the product name with “drink A 350 ml”. Get all the prices given.

In the present embodiment, the product information acquisition unit 14 recognizes by the product recognition unit 12 with reference to the stored product information in the product information storage unit 30 that associates the product with the price of the product in advance. Although the price associated with the created product is to be acquired, the present invention is not limited to such an example. For example, when the user inputs product information that associates a product (product name) with the price of the product to the shelving allocation information generating device 10 at the time of acquisition of a shelving allocation image, before acquisition, or the like, the product is The information acquisition unit 14 may acquire the price associated with the product recognized by the product recognition unit 12 based on the product information input by the user.

Further, the product information acquisition unit 14 may acquire the price associated with the product recognized by the product recognition unit 12 through the following first to third steps. That is, as the first step, the product recognition unit 12 recognizes the product (product name) of each product image based on the plurality of product images included in the shelving allocation image acquired by the acquisition unit 11. As the second step, the price tag recognition unit 13 is based on the price tag image corresponding to each product image (for example, the price tag image having a predetermined positional relationship with each product image, the price tag image closest to each product image, etc.). Then, the price of the product corresponding to each product image is recognized. As the third step, the product information acquisition unit 14 acquires the price of the price tag image corresponding to each of the plurality of product images recognized to be the same product, and various statistical methods for the distribution of the acquired price. Then, the price of the product is calculated.

For example, the product information acquisition unit 14 may extract the most included price from the acquired plurality of prices and acquire the extracted price as the price associated with the product. Further, the product information acquisition unit 14 may acquire the average price of the acquired plurality of prices as the price associated with the product. In addition, if the product information acquisition unit 14 can calculate a likely price as the price of the same product, the product information acquisition unit 14 adopts various other known statistical methods and based on the plurality of acquired prices. The price associated with may be obtained.

The determination unit 15 determines whether the price recognized by the price tag recognition unit 13 corresponds to the price of the product acquired by the product information acquisition unit 14. The price of the product acquired by the product information acquisition unit 14 is the price of the product recognized by the product recognition unit 12. The output unit 16 determines the price recognized by the price tag recognition unit 13 and the product recognition unit when it is determined that the price recognized by the price tag recognition unit 13 corresponds to the price of the product acquired by the product information acquisition unit 14. The product recognized by 12 is associated and output as shelving allocation information. Further, the determination unit 15 may perform the determination using at least one of the product recognition likelihood and the price recognition likelihood.

For the determination, the determination unit 15 may calculate the index value using at least one of the product recognition likelihood of the product candidate and the price recognition likelihood of the price candidate. The determination unit 15 determines the combination of the product candidate and the price candidate that is most likely to be the recognition result among the combination of the product candidate and the price candidate whose price candidate corresponds to the price associated with the product candidate in the product information. In order to do so, calculate the index value.

FIG. 6 is a diagram schematically illustrating an example of extracting the price of a product and calculating an index value based on the recognition of the product and the price. As shown in FIG. 6, the product recognition unit 12 outputs a product recognition result ra11 including a plurality of product candidates. Each product candidate included in the product recognition result ra11 includes the product name of the product candidate and the product recognition likelihood. Further, the price tag recognition unit 13 includes a price recognition result rb11 including a plurality of price candidates. A price recognition likelihood is associated with each price candidate included in the price recognition result rb11.

The determination unit 15 acquires the price information associated with the product of each product candidate in the product information, and generates the product recognition result ra12 associated with the acquired price information. Then, the determination unit 15 calculates an index value indicating the certainty of recognition in the combination of the product candidate and the price candidate. In the example illustrated in FIG. 6, the determination unit 15 obtains the index value by calculating the product of the product recognition likelihood of the product candidate and the price recognition likelihood of the price candidate, as shown in the determination result ra13. .. Specifically, the determination unit 15 calculates the product of the product recognition likelihood “0.6” of the product candidate “drink A 500 ml” and the price recognition likelihood “0.8” of the price candidate “120”, An index value “0.48” indicating the certainty as a combination of those product candidates and price candidates is obtained.

In the present embodiment, the index value is calculated by the product of the product recognition likelihood and the price recognition likelihood, but the magnitude of the likelihood value may be reflected in the index value. However, it is not limited to products. For example, the index value may be the sum of the likelihood of product recognition and the likelihood of price recognition, or may be calculated by a function having those likelihoods as variables. A value obtained by multiplying the power by the price recognition likelihood may be used.

FIG. 7 is a diagram illustrating an example of extracting a price of a product based on the recognition of the product and the price, calculating an index value, and determining whether the recognized price corresponds to the price of the recognized product. As shown in FIG. 7, the product recognition unit 12 outputs a product recognition result ra21 including a plurality of product candidates. Each product candidate included in the product recognition result ra21 includes the product name of the product candidate and the product recognition likelihood. The price tag recognition unit 13 also includes a price recognition result rb21 including a plurality of price candidates. A price recognition likelihood is associated with each price candidate included in the price recognition result rb21.

In the example shown in FIG. 7, in the product information, a plurality of price information is associated with one product (commodity candidate), and the product information acquisition unit 14 sets a plurality of price information for each product candidate. Is acquired. In such a case, the determination unit 15 acquires the price range based on the plurality of price information associated with each product candidate, and generates the product recognition result ra22 in which the acquired price range is associated with each product candidate. To do.

Then, as shown in the determination result ra23, the determination unit 15 calculates the index value indicating the certainty of recognition in the combination of the determination result product candidate and the price candidate, and the price recognized by the price tag recognition unit 13 is , It is determined whether or not it falls within the price range of the corresponding product candidate.

In the example illustrated in FIG. 7, the determination unit 15 calculates the product of the product recognition likelihood “0.6” of the product candidate “drink A 500 ml” and the price recognition likelihood “0.8” of the price candidate “120”. Then, the index value “0.48” indicating the certainty as the combination of the product candidate and the price candidate is obtained. At the same time, the determination unit 15 determines whether or not the price candidate “120” falls within the price range “98-138” and obtains the determination result “1”.

Further, the determination unit 15 calculates the product of the product recognition likelihood “0.2” of the product candidate “drink A 1.0L” and the price recognition likelihood “0.8” of the price candidate “120”, and calculates them. The index value “0.16” indicating the certainty as the combination of the product candidate and the price candidate is obtained. At the same time, the determination unit 15 determines whether or not the price candidate “120” falls within the price range “125-160”, and obtains the determination result “0”.

The output unit 16 outputs, as shelving allocation information, the product candidates and the price candidates included in the combination in which the price recognized by the price tag recognition unit 13 corresponds to the price range and has the largest index value, are associated with each other. In the example of the combination of the commodity candidate and the price candidate shown in the determination result ra23, the output unit 16 outputs the commodity candidate “drink A 500 ml” and the price candidate “120” in association with each other as shelving allocation information.

FIG. 8 is a diagram showing an example of extraction of a price of a product based on recognition of a product and a price, calculation of an index value, and determination of whether or not the recognized price corresponds to the price of the recognized product. As shown in FIG. 8, the product recognition unit 12 outputs a product recognition result ra31 including a plurality of product candidates. Each product candidate included in the product recognition result ra31 includes the product name of the product candidate and the product recognition likelihood. Further, the price tag recognition unit 13 includes a price recognition result rb31 including a plurality of price candidates. A price recognition likelihood is associated with each price candidate included in the price recognition result rb31.

In the price recognition result rb21 in FIG. 7, the price recognition likelihood of the price “120” was 0.8 and the price recognition likelihood of the price “1200” was 0.1. In the recognition result rb31, the price recognition likelihood of the price “1200” is 0.8, and the price recognition likelihood of the price “120” is 0.1.

In the example illustrated in FIG. 8, as in the example of FIG. 7, the product information acquisition unit 14 acquires information about a plurality of prices for each product candidate. The determination unit 15 acquires a price range based on information on a plurality of prices associated with each product candidate, and generates a product recognition result ra32 in which the acquired price range is associated with each product candidate. Then, as indicated by the determination result ra33, the determination unit 15 calculates the index value indicating the certainty of recognition in the combination of the determination result product candidate and the price candidate, and the price recognized by the price tag recognition unit 13 is determined. , It is determined whether or not it falls within the price range of the corresponding product candidate.

In the example illustrated in FIG. 8, the determination unit 15 calculates the product of the product recognition likelihood “0.6” of the product candidate “drink A 500 ml” and the price recognition likelihood “0.8” of the price candidate “1200”. Then, the index value “0.48” indicating the certainty as the combination of the product candidate and the price candidate is obtained (first line of the determination result ra33). At the same time, the determination unit 15 determines whether or not the price candidate “1200” falls within the price range “98-138” and obtains the determination result “0”.

In addition, the determination unit 15 calculates the product of the product recognition likelihood “0.6” of the product candidate “drink A 500 ml” and the price recognition likelihood “0.1” of the price candidate “120”, and those products An index value “0.06” indicating the certainty as a combination of the candidate and the price candidate is obtained (the fourth line of the determination result ra33). At the same time, the determination unit 15 determines whether or not the price candidate “120” falls within the price range “98-138” and obtains the determination result “1”.

The output unit 16 outputs, as shelving allocation information, the product candidates and the price candidates included in the combination in which the price recognized by the price tag recognition unit 13 corresponds to the price range and has the largest index value, are associated with each other. In the example of the combination of the product candidate and the price candidate shown in the determination result ra33, the output unit 16 outputs the product candidate “drink A 500 ml” and the price candidate “120” in association with each other as shelving allocation information.

Next, an example in which the product information indicating the price of the product is obtained as the price likelihood information indicating the likelihood that a certain price is set for the product will be described. FIG. 9 is a diagram showing an example of acquisition of price likelihood information based on price information of a product included in product information. The product information acquisition unit 14 acquires the product information pd2. Similar to FIG. 5, the product information pd2 includes information indicating what product was sold at what time at what price, and includes the product name, date, and price in association with each other. ..

The determination unit 15 can refer to the product information pd2 and generate a histogram pd3 for each price based on, for example, information on a plurality of prices associated with the product “drink A 350 ml”. Furthermore, the determination unit 15 outputs a likelihood function representing the likelihood of each price of the product “drink A 350 ml” as price likelihood information pd4. That is, an example of the price likelihood information pd4 is a probability distribution of each price, and is represented as a function that outputs the likelihood with the price as a variable.

FIG. 10 is a diagram showing an example of extraction of a price of a product based on recognition of a product and a price, calculation of an index value, and determination of whether or not the recognized price corresponds to the price of the recognized product. In the example shown in FIG. 10, the price of the product is represented by price likelihood information.

As shown in FIG. 10, the product recognition unit 12 outputs a product recognition result ra41 including a plurality of product candidates. Each product candidate included in the product recognition result ra31 includes the product name of the product candidate and the product recognition likelihood. The price tag recognition unit 13 also includes a price recognition result rb41 including a plurality of price candidates. A price recognition likelihood is associated with each price candidate included in the price recognition result rb41.

In the example shown in FIG. 10, the product information acquisition unit 14 acquires information on a plurality of prices for each product candidate. Then, the determination unit 15 acquires price likelihood information composed of a price likelihood function for each product based on the prices acquired for each product. The determination unit 15 generates the product recognition result ra42 in which the acquired price likelihood information is associated with each product candidate. In the product recognition result ra42, for example, the likelihood function “likelihood=f ₁ (price)” is associated with the product candidate “drink A 1.0L” as price likelihood information.

Then, as indicated by the determination result ra43, the determination unit 15 calculates the index value indicating the certainty of recognition in the combination of the determination result product candidate and the price candidate, and the price recognized by the price tag recognition unit 13 The likelihood is calculated based on the price likelihood information.

As indicated by the determination result ra43, the determination unit 15 determines that the product recognition likelihood “0.6” of the product candidate “drink A 1.0L” and the price recognition likelihood “0.8” of the price candidate “120”. Is calculated, and an index value “0.48” indicating the certainty as a combination of the product candidate and the price candidate is obtained (first line of the determination result ra43). At the same time, the determination unit 15 determines, based on the price likelihood information “likelihood=f ₁ (price)” associated with the product candidate “drink A 1.0L”, the likelihood “0. 10" is calculated. Further, the determination unit 15 multiplies the index value “0.48” by the price likelihood “0.1” to obtain the determination value “0.048”.

Similarly, the determination unit 15 calculates the product of the product recognition likelihood “0.2” of the product candidate “drink A 500 ml” and the price recognition likelihood “0.8” of the price candidate “120”, and calculates the product of them. An index value “0.16” indicating the certainty as a combination of the product candidate and the price candidate is obtained (second line of the determination result ra43). At the same time, the determination unit 15 determines the likelihood “0.40” of the price candidate “120” based on the price likelihood information “likelihood=f ₂ (price)” associated with the product candidate “drink A 500 ml”. To calculate. Furthermore, the determination unit 15 multiplies the index value “0.16” by the price likelihood “0.40” to obtain the determination value “0.064”.

The output unit 16 associates the product candidate and the price candidate included in the combination having the largest calculated determination value with each other and outputs them as shelving allocation information. In the example of the combination of the product candidate and the price candidate indicated by the determination result ra43, the output unit 16 outputs the product candidate “drink A 500 ml” and the price candidate “120” in association with each other as shelving allocation information.

As described with reference to FIGS. 9 and 10, for example, in the case where the product information includes the prices of a plurality of stores and the product over a specific period, a certain price is set for the product. The likelihood of a thing can be represented by a probability distribution with respect to the price of a product. In such a case, it is determined whether or not the recognized price corresponds to the price of the corresponding product based on the likelihood of the recognized price. Therefore, the appropriateness of the recognized price can be appropriately determined.

Next, another example of generation of shelving allocation information when a shelving allocation image including a plurality of product images and a plurality of price tag images having a predetermined relationship with products of each product image is acquired will be described. The acquisition unit 11 acquires a shelf allocation image including a plurality of product images and a plurality of price tag images having a predetermined relationship with the products of each product image, and the product recognition unit 12 recognizes the product images as the same type of product. When a plurality of shelving allocation images are included in the shelving allocation image, a plurality of different prices are set as the prices of the products of the respective product images for each of the plurality of product images recognized by the determination unit 15 as the same type of product. It may be determined to be applicable. In such a case, the output unit 16 may determine that the price calculated statistically based on a plurality of different prices determined to correspond to the price of the product of each product image is the same type of product. It is output as shelving allocation information for each of the recognized plurality of product images.

For example, the output unit 16 recognizes that the most frequent price, which is the most included price among a plurality of different prices determined to correspond to the price of the product of each product image, is the same type of product. It is output as shelving allocation information for each of a plurality of product images. The most frequent price is a price likely to be the price of a plurality of product images recognized to be the same type of product, and is likely to correspond to the price of the product. Further, the output unit 16 calculates by weighting each of a plurality of different prices determined to correspond to the price of the product of each product image based on the number of product images determined to correspond to each price. The average price may be output as shelving allocation information for each of a plurality of product images recognized to be the same type of product. In addition, the output unit 16 adopts various other known statistical methods as long as it is possible to calculate a plausible price as the prices of the products of a plurality of product images recognized to be the same type of product. be able to.

Next, with reference to FIG. 11 and FIG. 12, an example of the correction of the product recognition likelihood regarding the recognition of the product based on the product image will be described. FIG. 11 is a diagram schematically illustrating an example of extracting a product image and a price tag image from a shelving allocation image, recognizing a product from the product image, and recognizing a price and a product name from the price tag image. FIG. 11 shows an example of recognition of products and prices based on the shelving allocation image pm2 which is a part of the shelving allocation image acquired by the acquisition unit 11.

The product recognition unit 12 detects the product image pm21 from the shelving allocation image pm2. Further, the price tag recognition unit 13 extracts the price tag image pm22 located within a predetermined distance in the shelf allocation image pm2 from the product image pm21 recognized by the product recognition unit 12.

Further, the product recognition unit 12 recognizes the product based on the product image pm21 and outputs the product recognition result ra02 including the recognized product name of the product candidate. In addition, the product recognition unit 12 acquires a product recognition likelihood indicating the likelihood of recognition for each product candidate. In the example illustrated in FIG. 11, the product recognition result ra02 includes “drink A 500 ml”, “drink A 1.0 L”, and “drink C 500 ml” as the product names of the product candidates, and each product candidate has a product. The recognition likelihoods “0.6”, “0.2”, and “0.1” are associated.

Further, the price tag recognition unit 13 recognizes the price based on the price tag image pm22, and outputs the price recognition result rb02 including the recognized price candidate. Further, the price tag recognition unit 13 acquires the price recognition likelihood indicating the recognition certainty for each price candidate. In the example shown in FIG. 11, the price recognition result rb02 includes “120”, “140”, and “1200” as price candidates, and the price recognition likelihoods “0.8” and “0” are included in the price candidates, respectively. . 1” and “0.05” are associated.

Further, the price tag recognition unit 13 recognizes a product name which is identification information for identifying a product based on the price tag image pm22, and outputs a product name recognition result rc02 including the recognized product name. In addition, the price tag recognition unit 13 acquires the identification information recognition likelihood indicating the certainty of recognition of the product name from the price tag image for each product name. In the example illustrated in FIG. 11, the product name recognition result rc02 includes “drink A”, “drink a”, and “drink α” as product name candidates, and each product name candidate has the identification information recognition likelihood “ “0.5”, “0.3”, and “0.1” are associated.

FIG. 12 is a diagram illustrating an example of correcting the likelihood of recognizing a product using the likelihood of recognizing a product name from a price tag image. FIG. 12 shows the product recognition result ra02 and the product name recognition result rc02 described with reference to FIG. The determination unit 15 associates the price information of each product acquired by the product information acquisition unit 14 with each product candidate of the product recognition result ra02, and generates the product recognition result ra52 associated with the price information.

The determination unit 15 corrects the product recognition likelihood of the product candidate corresponding to the product identified by the product name by using the identification information recognition likelihood included in the product name recognition result rc02, and the product recognition likelihood is corrected. The generated product recognition result ra53 is generated.

Specifically, the determination unit 15 corrects by multiplying the product recognition likelihood “0.6” of the product “drink A 500 ml” by the identification information recognition likelihood “0.5” of the product name “drink A”. The obtained product recognition likelihood “0.3” is obtained (first line of the product recognition result ra53). In addition, the determination unit 15 multiplies the product recognition likelihood “0.2” of the product “drink A 1.0L” by the identification information recognition likelihood “0.5” of the product name “drink A” to make the correction. The product recognition likelihood “0.1” is obtained (the second line of the product recognition result ra53).

Note that the product name “drink C” is not recognized based on the price tag image pm22 as shown in the product name recognition result rc02, or even if it is recognized, a predetermined higher rank of the price recognition likelihood. Not included in n cases. In this case, the determination unit 15 multiplies the product recognition likelihood “0.1” of the product “Drink C 500 ml” by the identification information recognition likelihood “0.0” to correct the product recognition likelihood “0”. 0” is obtained (the third line of the product recognition result ra53).

The determination unit 15 can calculate the index value described with reference to FIGS. 7 and 8 using the product name recognition result rc02 in which the product recognition likelihood is corrected in this way. In this way, the product recognition likelihood in the product recognition based on the product image is corrected based on the acquired identification information recognition likelihood, so that the accuracy of the likelihood with respect to the recognition result of the product recognition is improved.

Next, with reference to FIG. 13, a method of generating shelving allocation information in the shelving allocation information generating apparatus 10 will be described. FIG. 13 is a flowchart showing the processing contents of the shelving allocation information generation method of this embodiment.

In step S<b>1, the acquisition unit 11 acquires an image of the display shelf on which the products are arranged, which is captured by the camera 21, as a shelf allocation image. In subsequent step S2, the product recognition unit 12 extracts a product image representing a product from the shelving allocation image acquired by the acquisition unit 11. Further, the price tag recognition unit 13 extracts a price tag image displaying the price of each product from the shelving allocation image.

In step S3, the product recognition unit 12 recognizes a product (product candidate) from each product image. In step S4, the price tag recognition unit 13 recognizes the price (price candidate) from the price tag image.

In step S5, the determination unit 15 associates the product image and the price tag image having a predetermined relationship with each other. For example, the determination unit 15 associates the product image and the price tag image that are located within a predetermined distance with each other in the shelving allocation image. In step S6, the determination unit 15 determines whether or not the determination has been performed for all combinations of product images and price tag images. If it is determined that all combinations have been determined, the process proceeds to step S12. On the other hand, when it is not determined that the determination has been performed for all the combinations, the process proceeds to step S7, and the processes up to step S11 are repeated.

In step S7, the determination unit 15 selects a combination of a set of product image and price tag image. In step S8, the product information acquisition unit 14 refers to the product information stored in the product information storage unit 30 and acquires the price information associated with the product recognized based on the product image.

In step S9, the determination unit 15 calculates an index value indicating the likelihood of recognition in the combination of the product candidate and the price candidate for each combination. The index value is calculated, for example, as the product of the product recognition likelihood and the price recognition likelihood. In step S10, the determination unit 15 determines in step S9 the combination of the price candidate and the product candidate whose price candidate based on the price tag image corresponds to the price associated with the product candidate recognized based on the product image. The combination with the largest calculated index value is adopted.

Then, in step S11, the output unit 16 associates the product candidates and the price candidates included in the combination adopted in step S10, and outputs them as shelving allocation information regarding the combination of the product image and the price tag image selected in step S7. To do.

In step S12, the output unit 16 outputs the shelving allocation information on the products and the price tags shown in the shelving allocation image. The output unit 16 outputs the shelving allocation information by, for example, storing the shelving allocation information in the shelving allocation information storage unit 40 or displaying it on the display 22.

Next, with reference to FIG. 14, another example of the shelving allocation information generating method in the shelving allocation information generating apparatus 10 will be described. FIG. 13 is a flowchart showing the processing contents of the shelving allocation information generation method of this embodiment.

In step S21, the acquisition unit 11 acquires an image of the display shelf on which the products are arranged, which is captured by the camera 21, as a shelf allocation image. In subsequent step S22, the product recognition unit 12 extracts a product image representing a product from the shelving allocation image acquired by the acquisition unit 11. Further, the price tag recognition unit 13 extracts a price tag image displaying the price of each product from the shelving allocation image.

In step S23, the product recognition unit 12 recognizes a product (product candidate) from each product image. In step S24, the price tag recognition unit 13 recognizes the price (price candidate) and the product name from the price tag image.

In step S25, the determination unit 15 associates the product image and the price tag image having a predetermined relationship with each other. For example, the determination unit 15 associates the product image and the price tag image that are located within a predetermined distance with each other in the shelving allocation image. In step S26, the determination unit 15 determines whether or not the combination of all the product images and the price tag images has been determined. If it is determined that all combinations have been determined, the process proceeds to step S33. On the other hand, if it is not determined that the determination has been made for all the combinations, the process proceeds to step S27, and the processes up to step S32 are repeated.

In step S27, the determination unit 15 selects a combination of a set of product image and price tag image. In step S28, the product information acquisition unit 14 refers to the product information stored in the product information storage unit 30 and acquires the price information associated with the product recognized based on the product image.

In step S29, the determination unit 15 corrects the likelihood of product recognition based on the product image (product recognition likelihood) using the likelihood of recognition of the product name based on the price tag image (identification information recognition likelihood). ..

In step S30, the determination unit 15 calculates an index value indicating the likelihood of recognition in the combination of the product candidate and the price candidate for each combination. The index value is calculated, for example, as the product of the corrected product recognition likelihood and the price recognition likelihood.

In step S31, the determination unit 15 determines in step S30 among the combinations of the price candidates and the product candidates that the price candidates based on the price tag image correspond to the prices associated with the product candidates recognized based on the product image. The combination with the largest calculated index value is adopted.

Then, in step S32, the output unit 16 associates the product candidate and the price candidate included in the combination adopted in step S31, and outputs them as shelving allocation information related to the combination of the product image and the price tag image selected in step S27. To do.

In step S33, the output unit 16 outputs the shelving allocation information regarding the products and the price tags shown in the shelving allocation image. The output unit 16 outputs the shelving allocation information by, for example, storing the shelving allocation information in the shelving allocation information storage unit 40 or displaying it on the display 22.

Next, a shelving allocation information generation program for causing the computer to function as the shelving allocation information generation device 10 of the present embodiment will be described. FIG. 15 is a diagram showing the configuration of the shelving allocation information generation program P1.

The shelving allocation information generation program P1 is a main module m10 that integrally controls shelving allocation information generation processing in the shelving allocation information generating device 10, an acquisition module m11, a product recognition module m12, a price tag recognition module m13, a product information acquisition module m14, The determination module m15 and the output module m16 are provided. The modules m11 to m16 realize the functions of the acquisition unit 11, the product recognition unit 12, the price tag recognition unit 13, the product information acquisition unit 14, the determination unit 15, and the output unit 16 in the shelving allocation information generation device 10. To be done. The shelving allocation information generation program P1 may be transmitted via a transmission medium such as a communication line, or may be stored in the recording medium M1 as shown in FIG. Good.

In the shelving allocation information generation device 10, the shelving allocation information generation method, and the shelving allocation information generation program P1 of the present embodiment described above, the product and the price are recognized from the product image and the price tag image included in the acquired shelving allocation image. Then, it is determined whether or not the recognized price corresponds to the price associated with the recognized product in advance. If it is determined to be applicable, it is highly likely that both the recognition of the product and the recognition of the price are valid, so by outputting the product and the price determined to be applicable in association with each other, highly accurate shelving allocation information is output. Can be generated.

Further, in the shelving allocation information generation device according to another aspect, the product recognizing unit, for each of a plurality of product candidates obtained as a result of recognizing the product for one product image, a product recognition likelihood indicating the likelihood of recognition. The price tag recognition unit acquires the price recognition likelihood indicating the certainty of recognition for each of the plurality of price candidates obtained as a result of the price recognition for one price tag image, and the determination unit determines the product recognition. The determination may be performed using at least one of the likelihood and the price recognition likelihood.

According to the above-described embodiment, at least one of the product recognition likelihood and the price recognition likelihood is used for the determination, so that it is possible to output the product and the price related to the recognition that are more likely to be relevant in association with each other.

Further, in the shelving allocation information generation device according to another aspect, the determination unit determines the product candidate among the combination of the price candidate and the product candidate such that the price candidate corresponds to the price associated with the product candidate in the product information. Of the product recognition likelihood and the price candidate likelihood of price recognition, the output unit determines the combination having the largest index value, and the output unit determines the combination of the price candidate and the product included in the combination determined by the determination unit. The candidates may be associated with each other and output as shelf allocation information.

According to the above aspect, the index value calculated using at least one of the product recognition likelihood and the price recognition likelihood is acquired for the combination of the recognized price candidate and the product candidate, and the product candidate product is obtained. The corresponding price candidate corresponds to the price associated in advance in the product information, and the product candidate and the price candidate included in the combination having the largest index value are associated and output as shelving allocation information. Therefore, since the combination of the product and the price with high likelihood of recognition is output as the shelving allocation information, the accuracy of the shelving allocation information is improved.

Further, in the shelving allocation information generation device according to another aspect, the price tag recognition unit recognizes the identification information for identifying the product based on the one price tag image, and the identification information indicating the recognition accuracy of each identification information. The determination unit may acquire the recognition likelihood and correct the product recognition likelihood of the product candidate corresponding to the product identified by the identification information, using the identification information recognition likelihood.

According to the above mode, when the identification information of the product can be recognized from the price tag image, the identification information is recognized and the identification information recognition likelihood indicating the certainty of the recognition is acquired. Then, since the product recognition likelihood in the product recognition based on the product image is corrected based on the acquired identification information recognition likelihood, the accuracy of the likelihood with respect to the recognition result of the product recognition is improved.

Further, in the shelving allocation information generation device according to another aspect, the product information includes information on a plurality of prices associated with one product, and the product information acquisition unit identifies the product recognized by the product recognition unit. The plurality of associated prices are acquired, and the determination unit determines whether or not the price recognized by the price tag recognition unit falls within the price range indicated by the plurality of prices acquired by the product information acquisition unit, and outputs it. When the determination unit determines that the price recognized by the price tag recognition unit falls within the price range, the unit may output the price recognized by the price tag recognition unit and the product as shelving allocation information in association with each other. ..

According to the above aspect, for example, when the product information includes a plurality of stores and the prices of the products over a specific period, a plurality of prices associated with the recognized products are acquired, and the plurality of acquired prices are acquired. It is determined whether the recognized price falls within the price range indicated by the price. Therefore, the validity of the recognized price is appropriately determined.

Further, in the shelving allocation information generation device according to another aspect, the product information includes price likelihood information indicating the likelihood that a certain price is set for the product as the price information of the product. The product information acquisition unit acquires the price likelihood information about the price of the product recognized by the product recognition unit, and the determination unit calculates the price recognized by the price tag recognition unit using the price likelihood information. It is also possible to determine whether or not the price recognized by the price tag recognition unit corresponds to the price of the product acquired by the product information acquisition unit by further using the likelihood.

According to the above aspect, for example, when the product information includes a plurality of stores and the price of the product over a specific period, the likelihood that a certain price is set for the product is Can be expressed as a probability distribution for the price of. In such a case, it is determined whether or not the recognized price corresponds to the price of the corresponding product based on the likelihood of the recognized price. Therefore, the validity of the recognized price is appropriately determined.

Further, in the shelving allocation information generating device according to another embodiment, the price tag recognizing unit is based on the price tag image located within a predetermined distance in the shelving allocation image from the product image of the product recognized by the product recognizing unit, The price of the product may be recognized.

According to the above aspect, in the shelving allocation image, the price tag of the product is positioned within a predetermined distance from the product image of the recognized product in view of the high possibility that the price tag of the product is arranged close to the corresponding product. The price is recognized based on the displayed price tag image. As a result, the price is recognized from the price tag image that is highly likely to have a corresponding relationship with the product recognized from the product image.

Further, in the shelving allocation information generation device according to another aspect, the price tag recognition unit identifies the product recognized by the product recognition unit when the identification information for identifying the product based on the price tag image can be recognized. The price of the product may be recognized based on the price tag image having the identification information.

According to the above aspect, the price is recognized based on the price tag image having the identification information for identifying the recognized product. As a result, the price is recognized from the price tag image that is highly likely to have a corresponding relationship with the product recognized from the product image.

Further, in the shelving allocation information generating device according to another aspect, the product recognizing unit recognizes a product of each product image based on the plurality of product images included in the shelving allocation image, and the price tag recognizing unit recognizes each product image. The price of the product corresponding to each product image is recognized based on the price tag image corresponding to, and the product information acquisition unit determines the price tag corresponding to each of the plurality of product images recognized as the same product. The price of the image may be acquired, and the price calculated by a predetermined statistical method based on the acquired plurality of price distributions may be acquired as the product information regarding the products recognized as the same product.

According to the above aspect, even if the product information in which the product and the price of the product are associated with each other is not acquired in advance, the plurality of prices corresponding to each of the plurality of product images recognized as the same product are acquired. Based on the distribution of, it is possible to obtain the price information of the products recognized as the same product.

Although the present embodiment has been described in detail above, it is obvious to those skilled in the art that the present embodiment is not limited to the embodiment described in this specification. The present embodiment can be implemented as modified and changed modes without departing from the spirit and scope of the present invention defined by the description of the claims. Therefore, the description of the present specification is for the purpose of exemplifying explanation, and does not have any restrictive meaning to the present embodiment.

Each aspect/embodiment described in this specification is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA. (Registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), It may be applied to a system using Bluetooth (registered trademark) or other appropriate system and/or a next-generation system extended based on these.

As long as there is no contradiction, the order of the processing procedure, sequence, flowchart, etc. of each aspect/embodiment described in this specification may be changed. For example, the methods described herein present elements of the various steps in a sample order, and are not limited to the specific order presented.

The input/output information and the like may be stored in a specific place (for example, a memory) or may be managed by a management table. Information to be input/output may be overwritten, updated, or added. The output information and the like may be deleted. The input information and the like may be transmitted to another device.

The determination may be performed by a value represented by 1 bit (whether 0 or 1), may be performed by a Boolean value (Boolean: true or false), and may be performed by comparing numerical values (for example, a predetermined value). (Comparison with the value).

The aspects/embodiments described in the present specification may be used alone, in combination, or may be switched according to execution. Further, the notification of the predetermined information (for example, the notification of “being X”) is not limited to the explicit notification, and is performed implicitly (for example, the notification of the predetermined information is not performed). Good.

Although the present disclosure has been described in detail above, it is obvious to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modifications and changes without departing from the spirit and scope of the present disclosure defined by the description of the claims. Therefore, the description of the present disclosure is for the purpose of exemplification, and does not have any restrictive meaning to the present disclosure.

Software, whether called software, firmware, middleware, microcode, hardware description language, or any other name, instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules , Application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc. should be construed broadly.

Moreover, software, instructions, etc. may be transmitted and received via a transmission medium. For example, the software may use a wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and/or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wireline and/or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description include voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any of these. May be represented by a combination of

Note that the terms described in the present disclosure and/or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

As used herein, the terms “system” and “network” are used interchangeably.

In addition, the information, parameters, and the like described in this specification may be represented by absolute values, relative values from predetermined values, or may be represented by other corresponding information. ..

The terms "determining" and "determining" as used in this disclosure may encompass a wide variety of actions. "Judgment", "decision" means, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigating (investigating), searching (looking up, search, inquiry) (Eg, searching in a table, database, or another data structure), considering ascertaining as “judging” or “deciding”, and the like. In addition, "decision" and "decision" include receiving (eg, receiving information), transmitting (eg, transmitting information), input (input), output (output), access (accessing) (for example, accessing data in a memory) may be regarded as “judging” and “deciding”. In addition, "judgment" and "decision" are considered to be "judgment" and "decision" when things such as resolving, selecting, choosing, establishing, establishing, and comparing are done. May be included. That is, the “judgment” and “decision” may include considering some action as “judgment” and “decision”. In addition, "determination (decision)" may be read as "assuming," "expecting," "considering," and the like.

As used in this disclosure, the phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

When the designations "first," "second," etc. are used herein, any reference to that element does not generally limit the amount or order of those elements. These designations may be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not imply that only two elements may be employed therein, or that the first element must precede the second element in any way.

As long as the terms “include”, “including”, and variations thereof are used in the present specification or claims, these terms are the same as the term “comprising”. It is intended to be comprehensive. Furthermore, the term "or" as used in the specification or claims is not intended to be an exclusive OR.

In the present specification, a plurality of devices are also included unless the context or technology clearly indicates that only one device exists.

Throughout this disclosure, the plural includes the plural unless the context clearly indicates the singular.

1... Shelving allocation information generating system, 10... Shelving allocation information generating device, 11... Acquisition unit, 12... Product recognition unit, 13... Price tag recognition unit, 14... Product information acquisition unit, 15... Judgment unit, 16... Output unit, 20... Imaging/display device, 21... Camera, 22... Display, 30... Product information storage unit, 40... Shelving allocation information storage unit, M1... Recording medium, m10... Main module, m11... Acquisition module, m12... Product recognition module , M13... Price tag recognition module, m14... Product information acquisition module, m15... Judgment module, m16... Output module, P1... Shelving allocation information generation program.

Claims (10)

A shelving allocation information generating device that generates shelving allocation information that is array information of displayed products,
An acquisition unit that acquires a shelving allocation image, wherein the shelving allocation image is an image of products arranged for sale, and a product image showing each product and a price of each product are displayed. An acquisition unit that includes a price tag image,
A product recognition unit that recognizes a product based on the product image;
A price tag recognition unit that recognizes the price of the product based on the price tag image having a predetermined relationship with the product recognized by the product recognition unit,
A product information acquisition unit that acquires a price associated with the product recognized by the product recognition unit based on product information that includes price information of the product associated with the product;
A determination unit that determines whether or not the price recognized by the price tag recognition unit corresponds to the price of the product acquired by the product information acquisition unit,
When the price recognized by the price tag recognition unit corresponds to the price of the product by the determination unit, the price recognized by the price tag recognition unit and the product are output as shelving allocation information in association with each other. Department,
Shelving allocation information generating device. The product recognition unit, for each of a plurality of product candidates obtained as a result of recognition of the product for one product image, acquires a product recognition likelihood indicating the likelihood of recognition,
The price tag recognition unit, for each of a plurality of price candidates obtained as a result of price recognition for one price tag image, acquires a price recognition likelihood indicating the certainty of recognition,
The determination unit performs determination using at least one of the product recognition likelihood and the price recognition likelihood,
The shelving allocation information generation device according to claim 1. The determination unit, among the combinations of the price candidate and the product candidate, such that the price candidate corresponds to the price associated with the product candidate in the product information, the product recognition likelihood of the product candidate, and Determining a combination having the largest index value calculated using at least one of the price recognition likelihoods of the price candidates,
The output unit associates the price candidate and the product candidate included in the combination determined by the determination unit with each other and outputs the price candidate as shelving allocation information.
The shelving allocation information generation device according to claim 2. The price tag recognition unit, based on one of the price tag image, while recognizing the identification information for identifying the product, to obtain the identification information recognition likelihood indicating the likelihood of recognition of each identification information,
The determination unit corrects the product recognition likelihood of the product candidate corresponding to the product identified by the identification information, using the identification information recognition likelihood,
The shelving allocation information generating device according to claim 2. The product information includes information on a plurality of prices associated with one product,
The product information acquisition unit acquires a plurality of prices associated with the product recognized by the product recognition unit,
The determination unit determines whether or not the price recognized by the price tag recognition unit falls within a price range indicated by the plurality of prices acquired by the product information acquisition unit,
The output unit associates the price recognized by the price tag recognition unit with the product when the price recognized by the price tag recognition unit falls within the price range by the determination unit, and shelving allocation information. Output as
The shelving allocation information generation device according to any one of claims 1 to 4. The product information has price likelihood information indicating the likelihood that a certain price is set for the product, as information on the price of the product,
The product information acquisition unit acquires price likelihood information regarding the price of the product recognized by the product recognition unit,
The determination unit further uses the likelihood of the price recognized by the price tag recognition unit calculated using the price likelihood information, and the price recognized by the price tag recognition unit is the product information acquisition unit. Determine whether or not the price of the product acquired by
The shelving allocation information generation device according to any one of claims 1 to 4. The price tag recognition unit recognizes the price of the product based on a price tag image located within a predetermined distance in the shelf allocation image from the product image of the product recognized by the product recognition unit,
The shelving allocation information generation device according to any one of claims 1 to 6. The price tag recognition unit, when the identification information for identifying the product based on the price tag image is recognizable, based on the price tag image having the identification information for identifying the product recognized by the product recognition unit, Recognize the price of the product,
The shelving allocation information generation device according to any one of claims 1 to 7. The product recognition unit recognizes a product of each product image based on a plurality of product images included in the shelving allocation image,
The price tag recognition unit recognizes the price of the product corresponding to each product image based on the price tag image corresponding to each product image,
The product information acquisition unit acquires a price of a price tag image corresponding to each of a plurality of product images recognized as the same product, and a predetermined statistical method based on the distribution of the acquired plurality of prices. The price calculated by is acquired as the product information regarding the products recognized as the same product,
The shelving allocation information generation device according to any one of claims 1 to 8. A shelving allocation information generation program for causing a computer to function as a shelving allocation information generation device that generates shelving allocation information that is arrangement information of products arranged on product shelves,
On the computer,
An acquisition function of acquiring a shelving allocation image, wherein the shelving allocation image is an image of products arranged for sale, and a product image showing each product and a price of each product are displayed. Acquisition function including price tag image,
A product recognition function for recognizing a product based on the product image,
A price tag recognition function for recognizing the price of the product based on the price tag image having a predetermined relationship with the product recognized by the product recognition function,
A product information acquisition function for acquiring the price associated with the product recognized by the product recognition function based on the product information including the price information of the product associated with the product;
A determination function of determining whether or not the price recognized by the price tag recognition function corresponds to the price of the product acquired by the product information acquisition function,
When the price recognized by the price tag recognition function corresponds to the price of the product by the determination unit, the output recognized as shelving allocation information by associating the price recognized by the price tag recognition function with the product Function,
Shelving allocation information generation program that realizes.

Images