JP7224931B2 - LEARNING MODEL GENERATOR, PROGRAM, AND METHOD OF MANUFACTURING TERMINAL DEVICE - Google Patents

Description

The present invention relates to a learning model generation device, a program, and a method of manufacturing a terminal device.

Japanese Unexamined Patent Application Publication No. 2002-200001 discloses a method of purchasing a product using a mobile device. In this method, the user scans the barcode of the product using the camera of the mobile device and transmits the identification information of the product contained in the barcode to the server. Subsequently, the server acquires product information such as the product name and price based on the product identification information, and stores the acquired product information in the virtual shopping cart. After that, settlement processing is performed based on the product information stored in the virtual shopping cart.

Japanese Patent Publication No. 2013-541107

In the method of Patent Document 1, when confirming product names and prices with a portable device, it is necessary to scan a barcode for each product on the premise of purchase. Therefore, when confirming the product name and price of products on display shelves or products whose purchase has not been confirmed, it is necessary to scan the barcode, confirm the product name and price, and then cancel the purchase. There is

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a learning model generation device, a program, and a method of manufacturing a terminal device that enable easy confirmation of product names and prices.

A learning model generation device according to an aspect of the present invention includes a receiving unit that receives images of a product placed in front of a background of a specific color to be removed during chromakey synthesis, photographed from a plurality of angles; Further, as product information registered in association with each of the images, an input reception unit that receives input of at least the product name and price of the product corresponding to the image; a background removal unit that removes a background of a specific color; and the image from which the background of the specific color is removed by the background removal unit for a plurality of background images that may serve as a background of the product when the product is sold in a store. an image synthesizing unit that generates a synthesized image by synthesizing each of the above, the synthetic image generated by the image synthesizing unit, and teacher data including a combination of the product name and price included in the corresponding product information and a learning unit that generates a learning model that learns using and outputs product names and prices corresponding to one or more input product images.

In the above aspect, the learning unit further comprises a synthetic image growing unit that adjusts attribute information corresponding to each of the synthetic images generated by the image synthesizing unit and generates a plurality of the synthetic images having different attribute information. may be learned using teacher data including a combination of the synthetic image generated by the image synthesizing unit and the synthetic image growing unit and the product name and price included in the corresponding product information.

In the above aspect, the attribute information may include at least one of contrast, size, rotation angle, and noise.

In the above aspect, the background image may include at least an image with the inside of the shopping basket as a background.

A program according to another aspect of the present invention provides a computer with a function of displaying one or more products photographed by an imaging unit on a display, and inputting an image of one or more products displayed on the display into a learning model. Function, function to display the product name and price output from the learning model on the display in association with the product, and display the total amount corresponding to one or more products displayed on the display according to the user's total instruction. Realize the function to display.

A method of manufacturing a terminal device according to another aspect of the present invention includes a process of displaying one or more products photographed by a photographing unit on a display of the terminal device, and an image of the one or more products displayed on the display. to the learning model, processing to display the product name and price output from the learning model on the display in correspondence with the product, and correspondence to one or more products displayed on the display according to the user's aggregation instruction. A terminal device is manufactured by distributing an application program for installing an application program for executing a process of displaying a total amount to be paid on a display.

According to the present invention, it is possible to provide a learning model generation device, a program, and a method of manufacturing a terminal device that enable easy confirmation of product names and prices.

1 is a diagram illustrating the configuration of a learning model generation device according to an embodiment; FIG. It is a figure which illustrates the structure of the terminal device which concerns on embodiment. (A) to (B) and (C) to (D) are schematic diagrams showing an example of images of products placed in front of a green background photographed from different angles. 3A to 3D are schematic diagrams illustrating images obtained by removing the green background from the images of FIGS. 3A to 3D; FIG. 4A is a schematic diagram illustrating an image of a product display shelf in a store, and FIG. 4B is a schematic diagram illustrating a composite image created by synthesizing the image of FIG. 4A on the image of FIG. 4A. FIG. It is a schematic diagram which shows an example of the screen displayed on the touch panel of a terminal device. It is a schematic diagram which shows an example of the screen displayed on the touch panel of a terminal device. 4 is a flow chart explaining an example of the operation procedure of the learning model generation device according to the embodiment; 6 is a flowchart illustrating an example of an operation procedure of a terminal device according to an embodiment;

Preferred embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that, in each figure, the same reference numerals have the same or similar configurations.

The configuration of the learning model generation device according to the embodiment will be described with reference to FIG. The learning model generation device 1 is a server device that receives an image of a product photographed by a camera of a terminal device, which will be described later, as an input, and generates a learning model that outputs the product name and price corresponding to the image.

The learning model generation device 1 has a physical configuration including, for example, a control device 10 including a CPU (processor) and memory, a communication device 20, a storage device 30, an input device 40, an output device (eg, display, speaker) 50, and the like. configured with The camera 9 can be connected to the learning model generation device 1 using wired communication or wireless communication. When the CPU executes a predetermined program stored in the memory or the storage device 30, each of the following functions is realized.

The learning model generating device 1 has, for example, a receiving unit 11, an input receiving unit 12, a background removing unit 13, an image synthesizing unit 14, a synthetic image growing unit 15, and a learning unit 16 as functional configurations. Each function is described below.

The receiving unit 11 receives from the camera 9 images of a product placed in front of a background of a specific color to be removed during chromakey synthesis, which are photographed from a plurality of angles. Chroma key synthesis is one of techniques for synthesizing video (images), and is a technique for fitting another video (image) into an area where a specific color is displayed. Green and blue are commonly used as particular colors. In this embodiment, a case where green is used as the specific color will be exemplified. The learning effect of the learning model can be enhanced as the number of different shooting angles increases. On the other hand, as the number of angles to be photographed increases, the trouble of photographing and the learning time increase. Therefore, it is desirable to appropriately determine the number of shooting angles in consideration of the learning effect and effort.

FIG. 3 exemplifies images of products photographed at different angles. FIG. 3(A) is an image of A coffee Ma placed upright in front of a green background (green background) B, photographed almost from the front. FIG. 3B is an image of the A coffee Ma of FIG. FIG. 3(C) is an image of the B coffee Mb placed upright in front of the green background B and photographed almost from the front. FIG. 3(D) is an image of the B coffee Mb in FIG. 3(C) photographed from near directly above.

The input reception unit 12 shown in FIG. 1 receives input of product information registered in association with each image received by the reception unit 11 . The product information is information about the product corresponding to the image, and includes, for example, the product name, price, production area of the product, expiration date of the product, evaluation of the product, and the like. In this embodiment, the product information includes at least the product name and price. The product information can be input by, for example, the administrator operating the input device 40 . By registering the input product information in association with the image, it is possible to reduce labor for inputting the product information when generating a learning model to be described later.

The background remover 13 removes the background of a specific color from each image received by the receiver 11 . FIG. 4 illustrates an image from which a background of a specific color has been removed. FIGS. 4A and 4B are images of A coffee Ma obtained by removing the green background B from the images of FIGS. 3A and 3B. FIGS. 4C and 4D are images of B coffee Mb obtained by removing the green background B from the images of FIGS. 3C and 3D.

The image synthesis unit 14 shown in FIG. 1 generates a synthesized image by synthesizing the image from which the background of the specific color has been removed by the background removal unit 13 with each of the plurality of background images. As the background image used for synthesis, an image that can serve as the background of the product when the product is sold at the store is used. For example, it is preferable to use an image with the background of the inside of the shopping basket into which the products are put, or an image with the background of the shelves on which the products are displayed. Background images can enhance the learning effect of the learning model as the number of background images increases. On the other hand, as the number of background images increases, the time required for synthesis and learning time increases. Therefore, it is desirable to appropriately determine the number of background images in consideration of the learning effect and effort.

FIG. 5 shows an example of the background image and the composite image. FIG. 5A is a background image showing a shelf on which beverages are displayed. FIG. 5(B) is a synthesized image created by synthesizing the image of A coffee Ma in FIG. 4(A) on the background image in FIG. 5(A).

The synthetic image multiplication unit 15 shown in FIG. 1 adjusts the attribute information corresponding to each of the synthetic images generated by the image synthesizing unit 14, and generates a plurality of synthetic images with different attribute information. The attribute information includes, for example, the contrast of the synthesized image, the size of the synthesized image, the angle at which the entire image is rotated from the original synthesized image, and the noise contained in the synthesized image. Since the number of synthesized images can be increased by changing the attribute information, it is possible to speed up the process of generating different synthesized images.

The more synthetic images are grown, the more the learning effect of the learning model can be enhanced. On the other hand, as the number to be multiplied increases, the adjustment and learning time will increase. Therefore, it is desirable to appropriately determine the number of synthesized images to be multiplied in consideration of the learning effect and effort.

The learning unit 16 learns teacher data including a combination of the synthetic image generated by the image synthesizing unit 14 and the synthetic image multiplication unit 15 and the product name and price included in the product information corresponding to the synthetic image. , to generate a learning model. Based on the learned teacher data, the learning unit 16 outputs product names and prices corresponding to one or more product images input to the learning model.

The function of the learning unit 16 is realized using a deep learning model for object detection such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Regions with CNN features), etc. can be done.

The configuration of the terminal device according to the embodiment will be described with reference to FIG. The terminal device 6 has a physical configuration including, for example, a control device 61 including a CPU (processor) and memory, a storage device 62, a touch panel 63 including an input device and a display, a communication device 64, a camera (photographing device) 65, and the like. configured with.

An application program loaded with a learning model generated by the learning unit 16 is installed in the terminal device 6 . When the CPU executes the application program stored in the memory or the storage device 62, each of the following functions is realized.

The terminal device 6 has, for example, a product image display function, an input function, a product information display function, and a total amount display function as functional configurations. The product image display function is a function of displaying one or more products photographed by the camera 65 on the display. The input function is a function of inputting one or more product images displayed on the display to the learning model.

The product information display function is a function of displaying the product name and price output from the learning model on the display in association with the product. FIG. 6 shows an example of a screen displaying product names and prices corresponding to products. The display 63 of the terminal device 6 displays the inside of the shopping basket photographed by the camera 65 . Inside the shopping basket, A coffee Ma, B coffee Mb, onion Mc, apple Md, and banana Me are stored.

In FIG. 6, the product name "A coffee" and the price "¥67" are displayed corresponding to the A coffee Ma, and the product name "B coffee" and the price "¥95" are displayed corresponding to the B coffee Mb. is displayed. Similarly, the product name “onion” and the price “¥100” are displayed corresponding to the onion Mc, the product name “apple” and the price “¥109” are displayed corresponding to the apple Md, and the banana Me is displayed. Correspondingly, the product name "banana" and the price "¥201" are displayed. When the button T at the bottom of the screen is clicked, a tally instruction is sent, and the screen transitions to the screen shown in FIG. 7, which will be described later.

The total price display function is a function of displaying the total price and the like corresponding to one or more products displayed on the display in accordance with a user's tally instruction. FIG. 7 shows an example of a screen displaying the total price corresponding to each product. The display 63 of the terminal device 6 displays detailed information and lowest price information of each product photographed by the camera 65, as well as the total and the total lowest price.

The itemized information is the product name and price of each product photographed by the camera 65, and the lowest price information is the lowest price and the name of the store selling the product in the surrounding stores. The total is the sum of the prices of the products photographed by the camera 65, and the total lowest price is the sum of the lowest prices of the products. In FIG. 7, "572 yen" is displayed as the total, and "468 yen" is displayed as the total lowest price.

In this way, when one or more products are displayed on the display using the camera 65 of the terminal device 6, the total price of each product and the lowest price of each product at stores including nearby stores are displayed. shopping can be efficiently supported.

Next, an example of the operation of the learning model generation device 1 according to the embodiment will be described with reference to FIG.

First, the receiving unit 11 of the learning model generation device 1 receives from the camera 9 images of a product placed in front of a green background, which are captured from multiple angles (step S101).

Subsequently, the input receiving unit 12 receives an input of a product name and price as product information registered in association with each image received in step S101 (step S102).

Subsequently, the background removal unit 13 removes the green background from each image received in step S101 (step S103).

Subsequently, the image synthesis unit 14 generates a synthesized image by synthesizing the image from which the green background has been removed in step S103 with each of the plurality of background images (step S104).

Subsequently, the composite image growing unit 15 adjusts the attribute information corresponding to each composite image generated in step S104, and generates a plurality of composite images having different attribute information (step S105).

Subsequently, the learning unit 16 generates a learning model using teacher data including combinations of the synthetic images generated in steps S104 and S105 and the product names and prices included in the product information corresponding to the synthetic images. Generate (step S106). Then, this operation ends.

Next, an example of the operation of the terminal device 6 according to the embodiment will be described with reference to FIG.

First, the terminal device 6 displays one or more products photographed by the camera 65 on the display (step S201).

Subsequently, the terminal device 6 inputs one or more product images displayed on the display to the learning model (step S202).

Subsequently, the terminal device 6 displays the product name and price output from the learning model on the display in association with the product (step S203).

Subsequently, the terminal device 6 determines whether or not the user has issued a tally instruction (by clicking the button T) (step S204). If this determination is NO (step S204; NO), it waits until a totaling instruction is issued. Here, when the product photographed by the camera 65 is changed while waiting for the issuance of the tallying instruction, the process proceeds to step S201, and the changed product is processed from step S201 described above. The processing up to step S203 is executed.

On the other hand, if it is determined in step S204 that the tallying instruction has been issued (step S204; YES), the terminal device 6 displays the total amount corresponding to each product displayed on the display (step S205). ). Then, this operation ends.

As described above, according to the learning model generation device 1 of the embodiment, images of products placed in front of a green background photographed from a plurality of angles are received, and products registered in association with each of the images are received. It accepts input of names and prices, and generates composite images by synthesizing images with green backgrounds removed from multiple background images that can serve as backgrounds for products when they are sold in stores. Then, a learning model can be generated using teacher data including combinations of the synthesized images and corresponding product names and prices. Then, one or more product images are input to this learning model, and product names and prices corresponding to the product images can be output.

Therefore, according to the learning model generation device 1 of the embodiment, it is possible to easily confirm the product name and price.

In addition, according to the learning model generation device 1, the attribute information corresponding to each generated synthesized image is adjusted, and a plurality of synthesized images with different attribute information are further generated, thereby increasing the number of synthesized images used as teacher data. It can be faster.

[Variation]
It should be noted that the present invention is not limited to the embodiments described above, and can be implemented in various other forms without departing from the gist of the present invention. Therefore, the above-described embodiment is merely an example in all respects, and should not be construed as limiting. For example, the processing steps described above can be arbitrarily changed in order or executed in parallel as long as the content of processing is not inconsistent.

Also, the components of the learning model generation device 1 are not limited to the components in the above-described embodiments, and arbitrary components can be appropriately omitted or added as necessary. For example, the synthetic image growing unit 15 may be omitted from the functional configuration of the learning model generation device 1 .

REFERENCE SIGNS LIST 1 learning model generating device 6 terminal device 9 camera 10 control device 11 receiving unit 12 input receiving unit 13 background removing unit 14 image synthesizing unit 15 synthesized image growing unit , 16... Learning unit, 20... Communication device, 30... Storage device, 40... Input device, 61... Control device, 62... Storage device, 63... Touch panel (input device and display), 64... Communication device, 65... Camera.

Claims (6)

a receiving unit for receiving images of a product placed in front of a background of a specific color to be removed during chromakey synthesis, taken from a plurality of angles;
an input reception unit that receives input of at least the product name and price of the product corresponding to the image as product information registered in association with each of the images received by the reception unit;
a background remover that removes the background of the specific color from each of the images received by the receiver;
A composite image is generated by synthesizing the images from which the background of the specific color has been removed by the background removing unit with respect to a plurality of background images that can serve as the background of the product when the product is sold in a store. an image synthesizing unit to generate;
Learn using teacher data including combinations of the synthetic image generated by the image synthesizing unit and the product name and price included in the corresponding product information, and correspond to one or more input product images. a learning unit that generates a learning model that outputs the product name and price to be used;
A learning model generation device comprising: further comprising a synthetic image growing unit that adjusts attribute information corresponding to each of the synthetic images generated by the image synthesizing unit and generates a plurality of the synthetic images with different attribute information;
The learning unit learns using teacher data including a combination of the synthetic image generated by the image synthesizing unit and the synthetic image growing unit and the product name and price included in the corresponding product information.
The learning model generation device according to claim 1. The attribute information includes at least one of contrast, size, rotation angle and noise,
3. The learning model generation device according to claim 2. The background image includes at least an image with the inside of the shopping basket as a background,
The learning model generation device according to any one of claims 1 to 3. to the computer,
A function to display one or more products photographed by the photographing unit on the display;
The ability to input images of one or more products displayed on the display into the learning model;
A function to display the product name and price output from the learning model on the display in association with the product;
A function of displaying on the display the total amount corresponding to one or more products displayed on the display according to the user's totaling instruction;
A program for realizing
According to the learning model, the product is sold at a store by removing the background of the specific color from the images of the product placed in front of the background of the specific color to be removed during chromakey synthesis from a plurality of angles. A teacher containing a combination of a synthetic image generated by synthesizing each of a plurality of background images that can become the background of the product when the product is displayed, and the product name and price included in the product information corresponding to the image. It is learned using data and generated to output product names and prices corresponding to one or more input images of products. to the terminal device,
a process of displaying one or more products photographed by the photographing unit on the display;
A process of inputting images of one or more products displayed on the display into the learning model,
A process of displaying the product name and price output from the learning model on the display in association with the product;
A process of displaying on the display the total amount corresponding to one or more products displayed on the display according to the totaling instruction by the user;
A method of manufacturing a terminal device by distributing for installation an application program for executing
According to the learning model, the product is sold at a store by removing the background of the specific color from the images of the product placed in front of the background of the specific color to be removed during chromakey synthesis from a plurality of angles. A teacher containing a combination of a synthetic image generated by synthesizing each of a plurality of background images that can become the background of the product when the product is displayed, and the product name and price included in the product information corresponding to the image. It is learned using data and generated to output product names and prices corresponding to one or more input images of products.

Images