WO2019186974A1

WO2019186974A1 - Image identification register device, image identification register system, product information display method, and program

Info

Publication number: WO2019186974A1
Application number: PCT/JP2018/013475
Authority: WO
Inventors: 壮馬白石; 岩元　浩太; 秀雄横井; 二徳高田; 恵美北川
Original assignee: 日本電気株式会社
Priority date: 2018-03-29
Filing date: 2018-03-29
Publication date: 2019-10-03
Also published as: JPWO2019186974A1; JP6981538B2

Abstract

This image identification register system (1) is configured by including this image identification register device (10), an image capturing device (30) and a display (40). The display (40) is also used as a placement surface for a product. The image capturing device (30) is provided with: an image acquisition unit (110) which acquires an image of a product (P) placed on the display (40); a product identification unit (120) which acquires a product identification result and an identification result of a product position by analyzing the image of the product (P); a height information acquisition unit (130) which acquires height information that indicates the height of the product (P); and a display control unit (140) which outputs display data based on the product identification result at a first position on the display (40), which is determined on the basis of the identification result of the product position and the height information .

Description

Image identification register device, image identification register system, product information display method, and program

The present invention relates to a technique for supporting business in a store.

In the following Patent Document 1, a processing device identifies a product existing on a mounting table using an image generated by an imaging device directed from the upper side to the product mounting table, and the projection device identifies the identified product. And / or the technique which projects the information regarding accounting processing on the mounting surface of a mounting base is disclosed.

International Publication No. 2017/126253

The present inventor examined using a display instead of the projection device in the configuration shown in the above-mentioned Patent Document 1. In that case, the present inventor has found a problem that the position of information related to the accounting process displayed on the display is shifted depending on the height of the product to be identified.

The present invention has been made in view of the above problems. One of the objects of the present invention is to identify the product using an image of the product placed on the display, and display information corresponding to the identification result in accordance with the position of the product on the display. It is to provide a technique for reducing the displacement of the display position of the information.

The image identification cash register apparatus of the present invention is
Image acquisition means for acquiring an image of the product placed on the display;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
Is provided.

The image identification cash register system of the present invention is
A display whose display surface is used as a product placement surface;
A first imaging device that photographs a product placed on the display and generates an image of the product;
Image acquisition means for acquiring an image of the product;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
Is provided.

The product information display method of the present invention includes:
Computer
Get an image of the product placed on the display,
By analyzing the image of the product, the product identification result and the product position identification result are obtained,
Obtain height information indicating the height of the product,
Outputting display data based on the identification result of the product to a first position on the display determined based on the identification result of the position of the product and the height information;
Including that.

The program of the present invention causes a computer to execute the above-described product information display method.

According to the present invention, in the system for identifying the product using the image of the product placed on the display and displaying the information according to the identification result in accordance with the position of the product on the display, The shift of the display position can be reduced.

The above-described object and other objects, features, and advantages will be further clarified by a preferred embodiment described below and the following drawings attached thereto.

It is a figure which shows the structural example of the image identification registration system in 1st Embodiment. It is a block diagram which illustrates the hardware constitutions of an image identification registration system. It is a sequence diagram which illustrates the flow of the process performed by the image identification registration system of 1st Embodiment. It is a figure which illustrates the more specific structure of the image identification registration system in 1st Embodiment. It is a figure which illustrates the more specific structure of the image identification registration system in 1st Embodiment. It is a figure which illustrates the more specific structure of the image identification registration system in 1st Embodiment. It is a figure for demonstrating the cause which the shift | offset | difference of the output position of display data produces. It is a figure which shows concretely a mode that the output position of the display data shifted | deviated. It is a figure which shows an example of the data collected when producing | generating the function and table which derive | lead-out the magnitude | size of deviation | shift. It is a figure which shows the structural example of the image identification registration system in 2nd Embodiment. It is a figure explaining the flow from which a height information registration part acquires height information. It is a figure which shows an example of the information registered into a height information storage part. It is a figure explaining the flow from which a height information registration part acquires height information. It is a figure which illustrates the more specific structure of the image identification registration system in 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings, the same reference numerals are given to the same components, and the description will be omitted as appropriate. Also, unless otherwise specified, in each block diagram, each block represents a functional unit configuration, not a hardware unit configuration.

[First Embodiment]
〔System configuration〕
FIG. 1 is a diagram illustrating a configuration example of an image identification registration system 1 according to the first embodiment. 1 is merely an example, and the image identification registration system 1 according to the present invention is not limited to the configuration depicted in FIG. As shown in FIG. 1, the image identification registration system 1 includes an image identification registration device 10, an imaging device 30, and a display 40. The image identification registration device 10 is connected to the imaging device 30 and the display 40 by wiring or the like (not shown). A dotted line in FIG. 1 schematically indicates that the image identification registration device 10 is connected to the imaging device 30 and the display 40 so as to be communicable. The image identification register device 10 is connected to an accounting processing device (not shown) for performing accounting processing. The accounting processing device is, for example, a barcode scanner, cashier, drawer, automatic change machine, receipt printer, input device such as a keyboard or mouse, output device such as a speaker, or the like.

The display 40 displays various images used in store operations (for example, product checkout operations) on its display surface. The display 40 may be a touch panel display having a function of accepting an input by a touch operation using a finger or a touch pen. The display surface of the display 40 is also used as a placement surface on which the product P is placed as shown in the figure.

As shown in FIG. 1, the imaging device 30 is arranged so as to include the display 40 in the imaging range. The imaging device 30 captures the product P placed on the display surface of the display 40 and generates an image showing the product P. The image generated by the imaging device 30 is transmitted to the image identification register device 10.

As illustrated in FIG. 1, the image identification register device 10 of the present embodiment includes an image acquisition unit 110, a product identification unit 120, a height information acquisition unit 130, and a display control unit 140. The image acquisition unit 110 acquires an image of the product P placed on the display 40 from the imaging device 30. The product identification unit 120 acquires the identification result of the product P and the identification result of the position of the product P by analyzing the image of the product P placed on the display. For example, the product identification unit 120 uses a known object recognition technique (for example, a classifier constructed by pattern matching or machine learning) to identify a product and its position in an image generated by the imaging device 30. Can be identified. Further, even when a plurality of products are shown in the image generated by the imaging device 30, the product identification unit 120 can individually identify these products using a known object recognition algorithm. it can. The height information acquisition unit 130 acquires information indicating the height of the product P placed on the display (hereinafter referred to as “height information”). A specific method for acquiring the height information will be described later. The display control unit 140 outputs display data on the display 40 based on the identification result of the product P acquired by the product identification unit 120. At this time, the display control unit 140 determines the position on the display 40 determined based on the identification result of the position of the product P acquired by the product identification unit 120 and the height information acquired by the image acquisition unit 110 ( Display data is output to the first position.

Here, the “display data” includes information for allowing the store clerk and the customer to recognize the product identification result by the product identification unit 120. Specifically, the “display data” is information about the product (eg, product name, price, or appearance image) and information indicating the position of the product (eg: a frame-shaped display formed based on the product outline) Element). Further, the “display data” includes a user interface for proceeding with the settlement processing for the product identified by the product identification unit 120. Specifically, the “display data” includes a user interface for registering the identified product as a product to be purchased, a user interface for canceling the product, and a user interface for correcting the identification result of the product identification unit 120. Etc.

[Hardware configuration example]
The image identification registration system 1 may be realized by hardware (for example, a hard-wired electronic circuit) that implements each functional component, or a combination of hardware and software (for example, an electronic circuit and the same). Or a combination of programs for controlling the above. Hereinafter, a case where the image identification registration system 1 is realized by a combination of hardware and software will be further described.

FIG. 2 is a block diagram illustrating a hardware configuration of the image identification registration system 1.

The image identification registration device 10 includes a bus 1010, a processor 1020, a memory 1030, a storage device 1040, an input / output interface 1050, and a network interface 1060.

The bus 1010 is a data transmission path through which the processor 1020, the memory 1030, the storage device 1040, the input / output interface 1050, and the network interface 1060 exchange data with each other. However, the method of connecting the processors 1020 and the like to each other is not limited to bus connection.

The processor 1020 is a processor realized by a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like. The storage device 1040 stores program modules that implement the functions of the image identification register device 10 (such as the image acquisition unit 110, the product identification unit 120, the height information acquisition unit 130, and the display control unit 140). The processor 1020 reads each program module onto the memory 1030 and executes it, thereby realizing each function corresponding to the program module.

The input / output interface 1050 is an interface for connecting the image identification registration device 10 and various input / output devices. In FIG. 2, the image identification registration device 10 is connected to the imaging device 30 and the display 40 via an input / output interface 1050. The imaging device 30 is, for example, a camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal-Oxide Semiconductor) image sensor. As shown in the figure, the imaging device 30 is installed so as to include the display 40 (and the product P placed on the display 40) in the imaging range. The imaging device 30 captures the product P placed on the display 40 and generates an image in which the product P is captured. The display 40 is a general display device. The display 40 may be a display with a touch function (touch panel display). The display 40 is also used as a placement surface for the product P. Therefore, the display surface of the display 40 is preferably a flat display such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence). The input / output interface 1050 is connected to an accounting processing device (not shown). The accounting processing device includes, for example, a barcode scanner, cashier, drawer, automatic change machine, receipt printer, input device such as a keyboard and mouse, and output device such as a speaker. Further, as a transaction processing device, a display dedicated to a store clerk or a customer (a display different from the display 40) may be connected to the image identification register device 10 via an input / output interface 1050.

The network interface 1060 is an interface for connecting the image identification registration device 10 to a network. This network is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method of connecting the network interface 1060 to the network may be a wireless connection or a wired connection.

FIG. 2 is merely an example, and the hardware configuration of the image identification registration system 1 is not limited to the example of FIG. For example, the image identification registration device 10 may be connected to the imaging device 30 and the display 40 via the network interface 1060.

[Process flow]
The flow of processing executed by the image identification registration system 1 of this embodiment will be described with reference to FIG. FIG. 3 is a sequence diagram illustrating the flow of processing executed by the image identification registration system 1 according to the first embodiment.

First, the image acquisition unit 110 acquires a captured image generated by the imaging device 30 (S102). For example, the imaging device 30 constantly outputs the sensing result (captured image) of the image sensor to the image acquisition unit 110. Further, the imaging device 30 may be configured to generate an image in response to an instruction from the image acquisition unit 110. Then, the product identification unit 120 determines whether a product exists in the captured image acquired by the image acquisition unit 110 (S104). For example, the product identification unit 120 includes the captured image (or a feature amount extracted from the captured image) in the captured image based on an output result obtained by inputting the acquired image into the product identification engine (not shown). It can be determined whether or not a product exists. In addition, the product identification unit 120 may determine whether or not the object is a product by performing template matching after detecting the region of the object from the captured image using the edge feature amount or the like.

When the product does not exist in the captured image (S104: NO), the product identification unit 120 acquires a new captured image generated by the imaging device 30, and whether or not the product exists in the new captured image. Determine whether. On the other hand, when a product exists in the photographed image (S104: YES), the product identification unit 120 acquires the product identification result and the product position identification result in the photographed image from the product identification engine. (S106).

If there is a product in the captured image (S104: YES), the height information acquisition unit 130 acquires the height information of the product P placed on the display 40 (S108). Hereinafter, some specific examples of how the height information acquisition unit 130 acquires product height information will be described.

<Specific Example 1 of Method for Acquiring Height Information>
FIG. 4 is a diagram illustrating a more specific configuration of the image identification registration system 1 in the first embodiment. In the configuration of FIG. 4, the imaging device 30 can generate an image including depth information indicating the distance (depth) from the imaging surface to the subject. Specifically, the imaging device 30 is a depth camera, a stereo camera, or the like. In this case, the height information acquisition unit 130 can calculate the height of the product placed on the display 40 using the depth information included in the image generated by the imaging device 30. Specifically, the height information acquisition unit 130 subtracts the depth in the image area corresponding to the product P placed on the display 40 from the height from the display surface of the display 40 to the imaging surface of the imaging device 30. Thus, the height of the product P can be calculated. Further, when the imaging device 30 is a general monocular camera, the height information acquisition unit 130 acquires the focal length to the subject (the distance to the upper surface portion of the product), and the imaging device 30 from the display surface of the display 40. The height of the product can be calculated by subtracting from the height to the imaging surface. The height from the display surface of the display 40 to the imaging surface of the imaging device 30 is defined as a predetermined value in the program module of the height information acquisition unit 130, for example.

<Specific Example 2 of Method for Obtaining Height Information>
FIG. 5 is a diagram illustrating a more specific configuration of the image identification registration system 1 in the first embodiment. In the configuration of FIG. 5, an imaging device 35 different from the imaging device 30 is provided. Here, the imaging device 30 is referred to as a “first imaging device”, and the imaging device 35 is referred to as a “second imaging device”. The second imaging device 35 captures the product P placed on the display 40 from a direction parallel to the display surface of the display 40. By doing in this way, the 2nd imaging device 35 can produce | generate the image (henceforth "side image") which the side part of the goods P was reflected. In this specific example, the height information acquisition unit 130 acquires a side image of the product P generated by the second imaging device 35. Then, the height information acquisition unit 130 can calculate the height of the product P (the height from the display surface of the display 40 to the upper surface of the product P) by analyzing the side image of the product P.

<Specific example 3 of method for acquiring height information>
FIG. 6 is a diagram illustrating a more specific configuration of the image identification registration system 1 in the first embodiment. In the configuration of FIG. 6, the image identification registration device 10 further includes a height information storage unit 132. The height information storage unit 132 stores height information of each product handled in the store. In this specific example, the height information acquisition unit 130 first receives a product identification result from the product identification unit 120. Then, the height information acquisition unit 130 uses the product identification result received from the product identification unit 120 to identify the product from the product-specific height information stored in the height information storage unit 132. Height information corresponding to the result can be acquired.

3, the display control unit 140 generates display data to be output on the display 40 based on the product identification result acquired by the product identification unit 120 (S110). Here, the display control unit 140 receives a product identification unit from a table (not shown) that stores various types of product information (product name, product price, product appearance image, presence / absence of benefits such as discount), and the like. Information about the product identified by 120 can be acquired and included in the display data. Further, the display control unit 140 can include, in the display data, drawing information of various user interfaces for performing settlement processing for the product in response to the product being identified by the product identification unit 120.

Here, the display control unit 140 needs to correctly instruct the display 40 to output the display data so that the display 40 can output the display data in accordance with the position of the product. Therefore, the display control unit 140 converts the product position identification result (the product position in the image generated by the imaging device 30) into a position on the display 40. Note that the display control unit 140 converts “the position (coordinates) in the image generated by the imaging device 30” into “the position (coordinates) on the display surface of the display 40 (the mounting surface of the product P)”. Rules are held in advance. Then, the display control unit 140 can convert the position (coordinates) of the product in the image into the position (coordinates) of the display surface of the display 40 based on the conversion rule.

However, there is a possibility that the position of the product P identified based on the image is deviated from the actual position of the product P for the following reasons.

FIG. 7 is a diagram for explaining the cause of the deviation of the output position of the display data. In the figure, reference symbols P _A and P _B each indicate a product placed on the display 40. Specifically, the product P _A is located in front of the imaging device 30, and the product P _B is located in a location shifted to the right from the front of the imaging device 30. Further, in this figure, reference numeral D _A and code D _B, respectively, show the indication data issued on the basis of the identification result of the product placed on the display 40.

In the case of this figure, the part shown with the dotted line in a figure will be reflected in the image produced | generated by the imaging device 30. FIG. That is, the product identification unit 120 identifies the position of the product based on the position of the part of the product (mainly the upper surface portion of the product) reflected in the image. Here, in the image generated by the imaging device 30, a portion indicated by a dotted line in the drawing is dropped in two dimensions along a one-dot chain line extending with the imaging device 30 as a reference in the drawing. For this reason, when the product is located at a position deviated from the front surface of the imaging device 30, an error corresponding to the height of the product is included in the product position identification result based on the image. In the example of the figure, item P _A because it has positioned in front of the imaging device 30, there is no deviation between the actual position occurs. Thus, the display 40 in accordance with the identified location of goods on the basis of the image, it is possible to output the display data D _A in accordance with the actual position of the product P _A. On the other hand, the product P _B is located at a position shifted from the front of the imaging device 30 to the right side. Therefore, the position of the product P _B identified based on the image is shifted to the right side from the actual position. Thus, the display 40, when according to the identified location of the product on the basis of the image would result in outputting display data D _B at a position shifted to the right of the actual position of the product P _B .

In FIG. 7, an example of a cross section (one-dimensional) is given to simplify the description, but in actuality, the deviation as described in FIG. 7 occurs on a plane (two-dimensional) ( Example: FIG. FIG. 8 is a diagram specifically illustrating a state where the output position of the display data is shifted. In the example of FIG. 8, the display data D is output in a shifted manner depending on the positional relationship between the product P and the imaging device that is overlooked at the top and the height of the product P.

As described above, the position of the product P identified based on the image may be shifted from the actual position of the product P depending on the positional relationship between the product P and the imaging device 30 and the height of the product P. Therefore, the display control unit 140 is based on the identification result of the position of the product acquired by the product identification unit 120 and the height information acquired by the height information acquisition unit 130 in order to cancel such a shift. The display data output position (first position) is calculated (S112). As an example, the display control unit 140 operates as follows. First, the display control unit 140 calculates how much the position of the product P identified based on the image includes a shift depending on the positional relationship with respect to the imaging device 30 and the height of the product. . At this time, the display control unit 140 uses a function or a table that can derive the amount of deviation (the amount of deviation up to the actual product position) from the combination of the display position of the display data D and the height of the product. Can do. Note that this function or table can be generated based on the results of actual measurement of the amount of deviation at a plurality of sample points on the display 40 using products of various heights. In this case, for example, data as shown in FIG. 9 is collected. FIG. 9 shows an example of data collected when generating a function or a table for deriving the magnitude of deviation. Based on the data as shown in FIG. 9, a function that can derive the amount of deviation according to the combination of the position of the display data (the position of the product identified based on the image) and the height of the product, A table is generated and stored in the storage device 1040. Then, the display control unit 140 calculates the output position (first position) of the display data by adding a value in a direction for canceling the magnitude of the deviation calculated above to the result of identifying the position of the product.

The display control unit 140 transmits the display data generated in the process of S110 and the output position of the display data calculated in the process of S112 to the display 40 (S114). Then, the display 40 outputs the display data acquired from the display control unit 140 to a position corresponding to the output position acquired from the display control unit 140 (S116).

As described above, according to the present embodiment, it is possible to prevent the display data based on the product identification result from being output from the position of the product on the display 40. Moreover, the visibility of the information displayed on the display 40 is improved by correctly displaying the display data based on the product identification result according to the position of the product.

[Second Embodiment]
In the first embodiment, the height information acquisition unit 130 selects height information corresponding to the product identified by the product identification unit 120 from the height information for each product stored in the height information storage unit 132. The configuration for acquiring the above (example: FIG. 6) has been described. In the present embodiment, the image identification registration system 1 further including a configuration for constructing the height information storage unit 132 will be described.

〔System configuration〕
FIG. 10 is a diagram illustrating a configuration example of the image identification registration system 1 in the second embodiment. 10 is merely an example, and the image identification registration system 1 according to the present invention is not limited to the configuration depicted in FIG. As shown in FIG. 10, the image identification registration system 1 of the present embodiment further includes a height information registration unit 150 in addition to the configuration shown in FIG. 6.

The height information registration unit 150 acquires the height information for each product, and registers the acquired height information for each product in the height information storage unit 132. Hereinafter, some specific examples of how the height information registration unit 150 registers height information will be given.

<Specific Example 1 of Method for Registering Height Information>
FIG. 11 is a diagram illustrating a flow in which the height information registration unit 150 acquires height information. In the example of FIG. 11, the display control unit 140 outputs the display data D to a position (second position) on the display 40 that is determined based on the position of the product identified by the product identification unit 120. . In this case, the display data D has been output at a location deviated from the actual position of the product P depending on the placement position of the product P and the height of the product P.

Therefore, the user (a store clerk or the like) performs an operation (for example, a swipe operation) for moving the display data D using an instruction object such as a finger or a touch pen, and designates the actual position of the product on the display 40. That is, the user performs position designation input for rearranging the display data D displayed at the second position to the correct position (third position). The height information registration unit 150 receives the position designation input executed by the user in this way. When the display 40 has a touch panel function, the position information input generated in response to the touch operation by the height information registration unit 150 and the pointing object (for example, an input indicating the start point and the end point of the touch operation) is received. If the display 40 does not have a touch panel function, the height information registration unit 150 operates as follows, for example. First, the height information registration unit 150 analyzes the image generated by the imaging device 30 and tracks the movement of the pointing object (for example, a hand). Then, the height information registration unit 150 receives the movement of the pointing object obtained as a result of the image analysis (for example, the position change of the start point and end point of a certain reference point) as a position designation input. Further, the display control unit 140 can transmit an instruction to change the output position of the display data D to the display 40 in accordance with the position designation input executed on the display 40.

Then, the height information registration unit 150 uses a function or a table that can derive the amount of displacement (the amount of displacement up to the actual product position) from the combination of the display position of the display data D and the product height. Back up the product height. Here, the difference between the display position (second position) of the display data D and the position (third position) designated by the position designation input corresponds to “the size of deviation”. The height information registration unit 150 calculates the difference between the display position (second position) of the display data D and the position (third position) designated by the position designation input and the display position (second position) of the display data D. ) As a parameter, the product height is calculated backward from the above functions and tables. Then, the height information registration unit 150 registers the calculated product height in the height information storage unit 132. At this time, the height information registration unit 150 receives product information (product name and product identification information) corresponding to the calculated height information, and registers the information in association with the height information (example: FIG. 12). FIG. 12 is a diagram illustrating an example of information registered in the height information storage unit. According to the information as shown in FIG. 12, the height information acquisition unit 130 can acquire the height information of the product based on the product identification result of the product identification unit 120.

<Specific Example 2 of Method for Registering Height Information>
FIG. 13 is a diagram illustrating a flow in which the height information registration unit 150 acquires height information. In the example of FIG. 13, similarly to the example of FIG. 11, the display control unit 140 has a position (second position) on the display 40 determined based on the position of the product identified by the product identification unit 120. Display data D is output. In this case, the display data D has been output at a location deviated from the actual position of the product P depending on the placement position of the product P and the height of the product P.

In this specific example, the user (a store clerk or the like) moves the product P according to the output position of the display data D as shown in the figure. At this time, the height information registration unit 150 acquires the movement amount of the product P. For example, the height information registration unit 150 tracks the movement of the product P by analyzing the image generated by the imaging device 30. Then, the height information registration unit 150 acquires the movement amount of the product P based on the movement of the product P obtained as a result of the image analysis (for example, the position change of the start point and the end point of the reference point of the product P). be able to.

Then, the height information registration unit 150 uses a function or a table that can derive the amount of displacement (the amount of displacement up to the actual product position) from the combination of the display position of the display data D and the product height. Back up the product height. Here, the movement amount of the product P acquired by the height information registration unit 150 corresponds to “the size of the deviation”. The height information registration unit 150 uses the movement amount of the product P and the display position (second position) of the display data D as parameters, and calculates the height of the product from the above functions and tables. Then, the height information registration unit 150 registers the calculated product height in the height information storage unit 132. At this time, the height information registration unit 150 receives product information (product name and product identification information) corresponding to the calculated height information, and registers the information in association with the height information (example: FIG. 12). According to the information as shown in FIG. 12, the height information acquisition unit 130 can acquire the height information of the product based on the product identification result of the product identification unit 120.

<Specific Example 2 of Method for Obtaining Height Information>
FIG. 14 is a diagram illustrating a more specific configuration of the image identification registration system 1 in the second embodiment. In the configuration of FIG. 14, an imaging device 35 different from the imaging device 30 is provided. Here, the imaging device 30 is referred to as a “first imaging device”, and the imaging device 35 is referred to as a “second imaging device”. The second imaging device 35 captures the product P placed on the display 40 from a direction parallel to the display surface of the display 40. By doing in this way, the 2nd imaging device 35 can produce | generate the image (henceforth "side image") which the side part of the goods P was reflected. In this specific example, the height information registration unit 150 acquires the side image of the product P generated by the second imaging device 35. Then, the height information registration unit 150 can calculate the height of the product P (the height from the display surface of the display 40 to the top surface of the product P) by analyzing the side image of the product P. Then, the height information registration unit 150 registers the calculated product height in the height information storage unit 132. At this time, the height information registration unit 150 receives product information (product name and product identification information) corresponding to the calculated height information, and registers the information in association with the height information (example: FIG. 12). According to the information as shown in FIG. 12, the height information acquisition unit 130 can acquire the height information of the product based on the product identification result of the product identification unit 120.

As described above, the embodiments of the present invention have been described with reference to the drawings. However, these are exemplifications of the present invention, and various configurations other than the above can be adopted.

For example, the present invention can be applied to a so-called semi-self method and a self-method as well as a general method in which operations from registration of purchased products to payment of a price are performed by an apparatus for a store clerk. In the semi-self method, a cash register device for merchandise registration and a payment device for payment are provided separately. The registering device for product registration can have the function of the image identification registering device 10 described above. In the self-method, a customer operates a device from registration of a purchased product to payment of a price. The device operated by the customer can have the function of the image identification register device 10 described above.

Further, in the sequence diagram used in the above description, a plurality of steps (processes) are described in order, but the execution order of the plurality of steps described in the sequence diagram is not limited to the description order. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents. Moreover, each above-mentioned embodiment can be combined in the range in which the content does not conflict.

A part or all of the above embodiments can be described as in the following supplementary notes, but is not limited thereto.
1.
Image acquisition means for acquiring an image of the product placed on the display;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
An image identification register device.
2.
The height information acquisition means calculates the height of the product using depth information indicating the depth of the product shown in the image.
1. The image identification cash register apparatus described in 1.
3.
The height information acquisition means includes
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
1. The image identification cash register apparatus described in 1.
4).
The height information acquisition means refers to a storage device storing height information for each product, and acquires height information corresponding to the product identification result from the height information for each product.
1. The image identification cash register apparatus described in 1.
5.
Further comprising height information registration means for registering the product-specific height information in the storage device;
4). The image identification cash register apparatus described in 1.
6).
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Receiving an input designating a third position on the display different from the second position;
Calculating the height of the product based on the difference between the second position and the third position and the second position;
5. The image identification cash register apparatus described in 1.
7.
The third position is an actual position of a product placed on the display.
6). The image identification cash register apparatus described in 1.
8).
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Obtaining the amount of movement when the product is moved to the second position,
Calculating the height of the product based on the acquired movement amount and the second position;
5. The image identification cash register apparatus described in 1.
9.
The height information registration means includes:
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
5. The image identification cash register apparatus described in 1.
10.
A display whose display surface is used as a product placement surface;
A first imaging device that photographs a product placed on the display and generates an image of the product;
Image acquisition means for acquiring an image of the product;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
An image identification cash register system.
11.
Further comprising a second imaging device that captures a product placed on the display from a direction parallel to a display surface of the display and generates a side image of the product;
The height information acquisition means calculates the height of the product by analyzing a side image of the product.
10. The image identification cash register system described in 1.
12
The height information acquisition means calculates the height of the product using depth information indicating the depth of the product shown in the image.
10. The image identification cash register system described in 1.
13.
The height information acquisition means refers to a storage device storing height information for each product, and acquires height information corresponding to the product identification result from the height information for each product.
10. The image identification cash register system described in 1.
14
Further comprising height information registration means for registering the product-specific height information in the storage device;
13. The image identification cash register system described in 1.
15.
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Receiving an input designating a third position on the display different from the second position;
Calculating the height of the product based on the difference between the second position and the third position and the second position;
14 The image identification cash register system described in 1.
16.
The third position is an actual position of a product placed on the display.
15. The image identification cash register system described in 1.
17.
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Obtaining the amount of movement when the product is moved to the second position,
Calculating the height of the product based on the acquired movement amount and the second position;
14 The image identification cash register system described in 1.
18.
The height information registration means includes:
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
14 The image identification cash register system described in 1.
19.
Computer
Get an image of the product placed on the display,
By analyzing the image of the product, the product identification result and the product position identification result are obtained,
Obtain height information indicating the height of the product,
Outputting display data based on the identification result of the product to a first position on the display determined based on the identification result of the position of the product and the height information;
Product information display method.
20.
The computer is
Calculate the height of the product using depth information indicating the depth of the product in the image,
Including. Product information display method described in.
21.
The computer is
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
Including. Product information display method described in.
22.
The computer is
With reference to a storage device that stores the height information for each product, the height information corresponding to the identification result of the product is acquired from the height information for each product.
Including. Product information display method described in.
23.
The computer is
Register height information for each product in the storage device;
Including. Product information display method described in.
24.
The computer is
Outputting the display data to a second position on the display, determined based on the identification result of the position of the product,
Receiving an input designating a third position on the display different from the second position;
Calculating the height of the product based on the difference between the second position and the third position and the second position;
23. Product information display method described in.
25.
The third position is an actual position of a product placed on the display.
24. Product information display method described in.
26.
The computer is
Outputting the display data to a second position on the display, determined based on the identification result of the position of the product,
Obtaining the amount of movement when the product is moved to the second position,
Calculating the height of the product based on the acquired movement amount and the second position;
23. Product information display method described in.
27.
The computer is
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
23. Product information display method described in.
28.
In the computer, 19. To 27. The program which performs the merchandise information display method as described in any one of these.

Claims

Image acquisition means for acquiring an image of the product placed on the display;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
An image identification register device.
The height information acquisition means calculates the height of the product using depth information indicating the depth of the product shown in the image.
The image identification registration device according to claim 1.
The height information acquisition means includes
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
The image identification registration device according to claim 1.
The height information acquisition means refers to a storage device storing height information for each product, and acquires height information corresponding to the product identification result from the height information for each product.
The image identification registration device according to claim 1.
Further comprising height information registration means for registering the product-specific height information in the storage device;
The image identification registration device according to claim 4.
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Receiving an input designating a third position on the display different from the second position;
Calculating the height of the product based on the difference between the second position and the third position and the second position;
The image identification register device according to claim 5.
The third position is an actual position of a product placed on the display.
The image identification register device according to claim 6.
The display control means outputs the display data to a second position on the display, which is determined based on the identification result of the position of the product,
The height information registration means includes:
Obtaining the amount of movement when the product is moved to the second position,
Calculating the height of the product based on the acquired movement amount and the second position;
The image identification register device according to claim 5.
The height information registration means includes:
Obtaining a side image of the product obtained by photographing the product from a direction parallel to a display surface of the display;
Calculating the height of the product by analyzing a side image of the product;
The image identification register device according to claim 5.
A display whose display surface is used as a product placement surface;
A first imaging device that photographs a product placed on the display and generates an image of the product;
Image acquisition means for acquiring an image of the product;
Product identification means for obtaining an identification result of the product and an identification result of the position of the product by analyzing the image of the product;
Height information acquisition means for acquiring height information indicating the height of the product;
Display control means for outputting display data based on the product identification result to a first position on the display determined based on the product location identification result and the height information;
An image identification cash register system.
Further comprising a second imaging device that captures a product placed on the display from a direction parallel to a display surface of the display and generates a side image of the product;
The height information acquisition means calculates the height of the product by analyzing a side image of the product.
The image identification cash register system according to claim 10.
Computer
Get an image of the product placed on the display,
By analyzing the image of the product, the product identification result and the product position identification result are obtained,
Obtain height information indicating the height of the product,
Outputting display data based on the identification result of the product to a first position on the display determined based on the identification result of the position of the product and the height information;
Product information display method.
A program for causing a computer to execute the product information display method according to claim 12.