JP7211447B2 - Image recognition device, system, method and program - Google Patents

Description

The present invention relates to an apparatus, system, method, and program for image recognition.

Conventionally, when a customer purchases a product at a retail store, it is common for a POS terminal to read the bar code attached to the product to identify the product and acquire product information such as name and price. there were. In recent years, technology has been developed to identify products based on the appearance characteristics of products using image recognition technology, without requiring the products to be bar-coded.

Japanese Patent Application Laid-Open No. 2002-200002 describes a technique of capturing images of a plurality of products arranged in a predetermined recognition area and collectively recognizing the plurality of products from the feature values of the captured images. As a result, it is possible to simultaneously identify a plurality of products and efficiently acquire product information.

JP 2013-54673 A

Products such as beverage cans that have common logos and are sold in various sizes may be difficult to distinguish by image recognition processing. In addition, there are products such as perishables that come in a variety of shapes and colors, making it difficult to apply image recognition processing that uses shapes and colors. Therefore, it is considered that the accuracy of image recognition can be improved by performing different processing according to the type and condition of the product. However, with the technique described in Patent Document 1, multiple products arranged in the recognition area are treated equally, so different recognition processing cannot be applied to each product.

The present invention has been made in view of the above problems, and provides an image recognition apparatus, system, method, and program that can apply different recognition processes to a plurality of products to be subjected to image recognition. for the purpose.

A first aspect of the present invention is an image recognition device comprising an image acquisition unit for acquiring an image of a product, and an image recognition unit for identifying the product in the acquired image, wherein the image recognition unit , based on the location of the item, identify the item using different image recognition processes.

A second aspect of the present invention is an image recognition method, comprising the steps of acquiring an image of a product, and identifying the product in the acquired image, wherein the step of identifying the product comprises: Identifying the item using different image recognition processes based on the location of the item.

A third aspect of the present invention is a program, which is an image recognition method comprising the steps of acquiring an image of a product in a computer, and identifying the product in the acquired image, wherein the product comprises identifying the item using different image recognition processes based on the location of the item.

A fourth aspect of the present invention is an image recognition system comprising a placing table for placing a product, and an image recognition device for performing image recognition on an image of the product acquired by an imaging device. wherein the image recognition device comprises an image acquisition unit that acquires the image of the product placed on the mounting table, and an image recognition unit that identifies the product in the acquired image, A recognition unit identifies the item using different image recognition processes based on the location of the item.

According to the present invention, different image recognition processes can be performed on a plurality of products arranged in a recognition area based on the position of each product.

1 is a schematic diagram of an image recognition system according to a first embodiment; FIG. 1 is a block diagram of a POS terminal according to the first embodiment; FIG. It is a perspective view of the support part which concerns on 1st Embodiment. FIG. 2 is a schematic diagram of a product to be recognized in the first embodiment; FIG. It is a top view of the support part in the state where the goods which concern on 1st Embodiment are arrange|positioned. 1 is a schematic configuration diagram of a POS terminal according to a first embodiment; FIG. It is a figure which shows the flowchart of the image recognition method which concerns on 1st Embodiment. FIG. 11 is a top view of a recognition area on a mounting table according to the second embodiment; FIG. 10 is a diagram showing a flowchart of an image recognition method according to the second embodiment; FIG. 11 is a top view of a recognition area on a mounting table according to the third embodiment; FIG. 13 is a diagram showing a flowchart of an image recognition method according to the third embodiment; 1 is a schematic configuration diagram of a POS terminal according to each embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings, but the present invention is not limited to these embodiments. In the drawings described below, elements having the same function are denoted by the same reference numerals, and repeated description thereof may be omitted.

(First embodiment)
FIG. 1 is a schematic diagram of an image recognition system 10 according to this embodiment. The image recognition system 10 includes a POS terminal (point-of-sale information management terminal) 100 as an image recognition device, a mounting table 14 for placing a product to be subjected to image recognition, and a table for supporting the product on the mounting table 14. is provided with a support portion 200 of The POS terminal 100 includes an imaging device 11 provided at a position capable of capturing an image of the product on the mounting table 14, a projection device 12 capable of projecting a predetermined image on the mounting table 14, and a display for displaying information related to the product. A device 13 is connected. In the present embodiment, the image recognition apparatus is configured integrally with the POS terminal 100, but is provided separately from the POS terminal 100, and exchanges necessary information with the POS terminal 100 to obtain the image recognition method according to the present embodiment. may be configured to implement

The POS terminal 100 displays, on the display device 13, product information during checkout, the checkout amount, a predetermined message, and the like. Merchandise information is acquired by an image recognition method, which will be described later. In addition, the POS terminal 100 may perform arbitrary processing related to accounting, such as depositing and withdrawing money, printing receipts, and the like.

The mounting table 14 has a mounting surface extending in the horizontal direction (that is, the direction perpendicular to the direction of gravity). One or more products B are placed on the placing surface of the placing table 14 .

The imaging device 11 is provided above the mounting table 14 along the direction of gravity, and fixed to a pillar, an arm, a ceiling, or the like (not shown). The imaging device 11 is an arbitrary imaging device such as a camera or a scanner that acquires an image using an image sensor such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging device 11 can capture an image of at least an area including the mounting surface of the mounting table 14 and transmits a signal indicating the captured image to the POS terminal 100 . A signal indicating an image captured by the imaging device 11 may be directly transmitted to the POS terminal 100 or may be read by the POS terminal 100 after being temporarily recorded in a storage device. It is desirable that the optical axis of the imaging device 11 is aligned with the normal line of the mounting surface of the mounting table 14 in order to uniformly image the products on the mounting table 14 . The imaging device 11 may take images at predetermined time intervals, or may take images according to instructions from the POS terminal 100 .

The projection device 12 is provided above the mounting table 14 along the direction of gravity, and fixed to a pillar, an arm, a ceiling, or the like (not shown). The projection device 12 is an arbitrary projection device such as a projector that projects a desired image using a light source such as a lamp or an LED (Light Emitting Diode). The projection device 12 projects an image onto the mounting surface of the mounting table 14 according to the signal from the POS terminal 100 . The projection device 12 is installed at a predetermined position where such projection can be realized. The installation position of the projection device 12 is a matter of design. For example, the projection device 12 may be installed directly above or obliquely above the placement surface and project an image from that direction. Alternatively, the projection device 12 may be installed on the mounting surface and project the image from the lateral direction. Alternatively, the mounting table 14 may be made of a transparent member, and the projector 12 may be installed below the mounting table 14 . In this case, the projection device 12 projects an image onto the mounting surface of the mounting table 14 from below the mounting table 14 .

In this embodiment, the projection device 12 projects an image (for example, a frame surrounding the recognition region A) indicating the recognition region A imaged by the imaging device 11 and subjected to image recognition onto the mounting surface of the mounting table 14 . The recognition area A may match the imaging area of the imaging device 11 or may be a part of the imaging area of the imaging device 11 . The recognition area A may be preset in the POS terminal 100, or may be set by the user.

The support part 200 is a support structure for supporting the product B on the mounting surface of the mounting table 14 at a predetermined angle with respect to the direction of gravity. In the recognition area A, the support part 200 may be configured integrally with the mounting table 14 (that is, as part of the mounting table 14 ), or may be configured as a member mounted on the mounting table 14 . A detailed configuration of the support portion 200 will be described later with reference to FIG. 3 .

The display device 13 is a display device that displays arbitrary information such as product information to a user (that is, a store clerk or a customer). As the display device 13, any display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like may be used.

When executing the product recognition method, each product B to be recognized is placed directly within the recognition area A on the mounting table 14 or placed on the support section 200 . Specifically, when the product B is placed, if the external features of the product B are naturally directed upward along the direction of gravity (that is, directed toward the imaging device 11), the product B is placed on the placing table. 14 directly within the recognition area A. On the other hand, when the product B is placed, if the external features of the product B face the direction of gravity (that is, do not face the imaging device 11), the product B is placed on the support section 200. be done. Commodity B, whose external features are oriented laterally with respect to the direction of gravity, is laid down, for example, like a drink can, which has a substantially cylindrical shape, or a soft confectionery in a transparent container. It is an article that will be damaged by

FIG. 2 is a block diagram of the POS terminal 100 (image recognition device) according to this embodiment. In FIG. 2, arrows indicate main data flows, and there may be data flows other than those shown in FIG. In FIG. 2, each block does not show the configuration in units of hardware (apparatus), but the configuration in units of functions. As such, the blocks shown in FIG. 2 may be implemented within a single device, or may be implemented separately within multiple devices. Data exchange between blocks may be performed via any means such as a data bus, network, or portable storage medium.

The POS terminal 100 includes an image acquisition unit 110, an image recognition unit 120, a product information acquisition unit 130, and a display control unit 140 as processing units. The POS terminal 100 also includes an identification information storage unit 151 and a product information storage unit 152 as storage units.

The image acquisition unit 110 receives a signal indicating an image captured by the imaging device 11 and inputs the signal to the POS terminal 100 as image data. The image data may be obtained directly from the imaging device 11, or may be obtained by reading out data recorded in a storage device.

The image recognition section 120 extracts each product included in the image data acquired by the image acquisition section 110 . The image recognition unit 120 performs normal image recognition processing, for example, in the identification information storage unit 151, feature amounts calculated in advance from reference images of various products are recorded in association with product IDs that identify each product. be done. Any information indicating the appearance of a product that can be calculated from image data, such as shape, color, and color distribution, may be used as the feature amount. The image recognition unit 120 calculates a feature amount from the image data acquired by the image acquisition unit 110 and compares it with the feature amount recorded in the identification information storage unit 151 . Then, the image recognition unit 120 identifies the product having the most similar feature amount (that is, satisfying a predetermined criterion) in the identification information storage unit 151 as the product in the image data. The image recognition unit 120 acquires the product ID of the identified product from the identification information storage unit 151 . The product recognition method executed by the image recognition unit 120 is not limited to the specific methods shown here, and any image recognition technology that can identify products from image data may be used. Furthermore, in this embodiment, in addition to or in place of normal image recognition processing, different image recognition processing is performed for each position (region) where products are arranged as described later.

Furthermore, the image recognition unit 120 applies different processing to products supported by the support unit 200 and products placed directly on the mounting table 14 (that is, not supported by the support unit 200). The angle C1 (see FIG. 3) at which the support section 200 supports the product and the positional relationship between the imaging device 11 and the support section 200 are constant and known. Therefore, the image recognition section 120 corrects the area corresponding to the support section 200 in the image data using the angle C<b>1 and the positional relationship between the imaging device 11 and the support section 200 . That is, the image recognition unit 120 expands the image of the product imaged by the imaging device 11 using the angle C1 at which the product is supported and the positional relationship between the imaging device 11 and the support unit 200, thereby recognizing the image of the product. To generate an image similar to the case of imaging a side face from the front. Since the image corrected in this way is generally in a state similar to the reference image obtained by photographing the product from the front, the accuracy of the image recognition described above can be improved.

In the product information storage unit 152, arbitrary product information related to products such as product names and prices is recorded in advance in association with product IDs. The product information acquisition unit 130 acquires product information recorded in the product information storage unit 152 based on the product ID acquired by the image recognition unit 120 .

The display control unit 140 controls display of product information of each product acquired by the product information acquisition unit 130 . In this embodiment, the display control unit 140 controls not only the display by the display device 13 but also the processing visually presented to the user, such as receipt printing by a printer.

FIG. 3 is a perspective view of a support portion 200 as a product support structure according to this embodiment. In the present embodiment, the support portion 200 is configured as a member separate from the mounting table 14, but may be configured by forming part of the mounting surface of the mounting table 14 into the commodity support structure shown in FIG. Support 200 comprises side supports 210 and bottom supports 220 . Side supports 210 have side support surfaces 211 for supporting the sides of the item, and bottom supports 220 have bottom support surfaces 221 for supporting the bottom of the item. The side support surface 211 and the bottom support surface 221 are perpendicular to each other (that is, the angle formed by them is 90 degrees), and the side support surface 211 is at a predetermined angle with respect to the horizontal direction (that is, the direction perpendicular to the direction of gravity). A side support 210 and a bottom support 220 are provided to form C1. With such a configuration, the support section 200 supports the product at the angle C1, and the imaging device 11 can easily image the side surface of the product.

The angle C1 is preferably an angle larger than 0 degrees and smaller than 90 degrees, more preferably an angle of 30 degrees or more and 60 degrees or less, so that the imaging device 11 can stably image the side surface of the product. In form it is 45 degrees. If the angle C1 is greater than 60 degrees with respect to the horizontal direction, the side surface of the product will be imaged at a steep angle from the imaging device 11, resulting in deterioration in image recognition accuracy. If the angle C1 is less than 30 degrees with respect to the horizontal direction, there is a risk that the substantially cylindrical product will roll or the soft product will be damaged.

The side support surface 211 is provided with a region dividing portion 212 that divides the side support surface 211 into two regions. The area dividing portion 212 is a projection extending in a direction perpendicular to the bottom support surface 221 on the side support surface 211 . The area segmentation part 212 is formed by a symbol or color displayed on the side support surface 211, light projected by the projection device 12, or provided on the side support surface 211, as long as the area can be visually indicated to the user. It is good to have a structure that is

FIG. 4 is a schematic diagram of products D1 and D2 to be recognized in this embodiment. Products D1 and D2 are articles having a substantially cylindrical shape, such as beverage cans. Products D1 and D2 have different sizes. For example, product D1 contains 350 ml and product D2 contains 500 ml. A common logo mark E is displayed on the sides of the products D1 and D2. That is, the products D1 and D2 are products of variations with different sizes. General image recognition technology performs image recognition based on feature amounts such as shape, color, and color distribution in image data. Therefore, the products D1 and D2 having the common logo mark E are difficult to distinguish by image recognition, and are likely to be misidentified as the same product.

In this embodiment, the accuracy of image recognition is improved by using information on the positions (areas) where the products D1 and D2 having similar appearances are arranged.

FIGS. 5(a) and 5(b) are top views of the support portion 200 in which the product is placed. 5(a) and 5(b) are views of the support portion 200 supporting the products D1 and D2 from a direction perpendicular to the side support surface 211, respectively. As shown in FIGS. 5A and 5B, the side support surfaces 211 support the side surfaces of the products D1 and D2, and the bottom surface support surfaces 221 support the bottom surfaces of the products D1 and D2. , D2 can be stably supported, and imaging of the sides of the products D1 and D2 by the imaging device 11 can be facilitated.

The side support surface 211 is divided into a first area F1 and a second area F2 by an area dividing portion 212 . The image recognition unit 120 performs different recognition processes depending on which of the areas F1 and F2 the product is placed (that is, the position where the product is placed).

In order to discriminate between the areas F1 and F2, the image recognition unit 120 presets information indicating to which of the areas F1 and F2 each pixel in the image acquired by the imaging device 11 belongs, and identifies the area corresponding to the position of the product. In which of the regions F1 and F2 the product is located may be determined based on the pixels that are displayed. Alternatively, the image recognition unit 120 recognizes the position of the area dividing unit 212, and determines in which of the areas F1 and F2 the product is placed based on the positional relationship between the position of the area dividing unit 212 and the product. may

For example, when the products D1 and D2 are arranged in the first region F1, the image recognition unit 120 determines the products D1 and D2 as small products. That is, in the example of FIG. 5A, both the small product D1 and the large product D2 are determined as small products. Also, when the products D1 and D2 are arranged in the second area F2, the image recognition unit 120 determines the products D1 and D2 as large products. That is, in the example of FIG. 5B, both the small product D1 and the large product D2 are determined as large products. In actual operation, when the user arranges the small product D1 in the first area F1 and the large product D2 in the second area F2, the image recognition unit 120 recognizes similar products D1 of different sizes. , D2 can be correctly identified.

In other words, the image recognition unit 120 identifies the product based on the information on the appearance of the product and the information on the position where the product is arranged. As a result, it is possible to improve the accuracy of identification of products that are difficult to identify only from the appearance of the product.

The number of regions used by the image recognition unit 120 is not limited to two, and may be any number of at least two or more according to the number of sizes to be distinguished. In addition, the image recognition unit 120 may use positional information to distinguish other characteristics (variations in taste, aroma, etc.) of products that are difficult to distinguish only from the appearance, without being limited to the size of the product.

FIG. 6 is a schematic configuration diagram showing an exemplary configuration of the POS terminal 100 (image recognition device) according to this embodiment. The POS terminal 100 includes a CPU (Central Processing Unit) 101 , a memory 102 , a storage device 103 and an interface 104 . POS terminal 100 may be a separate device or may be integrated with other devices.

The interface 104 is a connection unit that is connected to another device to transmit and receive signals. The interface 104 includes a processor, an electric circuit, an antenna, connection terminals, etc. necessary for transmitting and receiving signals. The interface 104 transmits and receives signals to and from connected devices in accordance with signals from the CPU 101 . The interface 104 is connected to, for example, the imaging device 11, the projection device 12, the display device 13, and the input device 15, and transmits and receives signals to and from them. The interface 104 may also be connected to networks and other devices.

The storage device 103 stores programs executed by the POS terminal 100, data of processing results of the programs, and the like. The storage device 103 includes a read-only ROM (Read Only Memory), a readable/writable hard disk drive, a flash memory, or the like. The storage device 103 may also include a computer-readable portable storage medium such as a CD-ROM. The memory 102 includes a RAM (Random Access Memory) for temporarily storing data being processed by the CPU 101 and programs and data read from the storage device 103 .

The CPU 101 temporarily records temporary data used for processing in the memory 102, reads a program recorded in the storage device 103, and performs various calculations, controls, determinations, etc. on the temporary data according to the program. is a processor that executes the processing operations of The CPU 101 also records data of the processing result in the storage device 103 and transmits data and control signals to the outside via the interface 104 .

In this embodiment, the CPU 101 functions as the image acquisition unit 110, the image recognition unit 120, the product information acquisition unit 130, and the display control unit 140 shown in FIG. 2 by executing programs recorded in the storage device 103. FIG. Further, in this embodiment, the storage device 103 functions as the identification information storage unit 151 and the product information storage unit 152 shown in FIG.

The POS terminal 100 is not limited to the specific configuration shown in FIG. The POS terminal 100 is not limited to one device, and may be configured by connecting two or more physically separated devices by wire or wirelessly. Each unit included in the POS terminal 100 may be realized by an electric circuit configuration. Here, an electrical circuit configuration is a term that conceptually includes a single device, multiple devices, a chipset, or a cloud.

Also, at least part of the POS terminal 100 may be provided in a SaaS (Software as a Service) format. That is, at least part of the functions for realizing the POS terminal 100 may be executed by software executed via a network.

FIG. 7 is a diagram showing a flowchart of the image recognition method according to this embodiment. The flowchart of FIG. 7 is started when the user performs a predetermined operation for executing image recognition on the image recognition system 10 . For example, when the user presses a button on the input device 15 (keyboard or touch panel) connected to the POS terminal 100, or at predetermined intervals, the flowchart of FIG. 7 is started.

First, the image acquisition unit 110 receives a signal indicating an image captured by the imaging device 11, and inputs the signal to the POS terminal 100 as image data (step S101).

Next, the image recognizing unit 120 determines the angle C1 of the supporting unit 200 and the angle C1 of the imaging device 11 and the supporting unit 200 with respect to the area corresponding to the supporting unit 200 (supporting area) in the image acquired in step S101. are corrected using the positional relationship between and and output as one piece of image data (step S102). The support region is preset as a position (for example, pixel coordinates) in the image. Alternatively, the position of the support portion 200 may be determined from the image by image recognition, and the determined position may be used as the support portion region.

The image recognition unit 120 performs image recognition processing of the product using the image data corrected in step S102, and identifies the product based on the identification information of the identification information storage unit 151 (step S103). The image recognition unit 120 then determines the product identified in step S103 as the product in the image (step S104). When a plurality of products are identified in one image data, the subsequent processing is performed for each product.

When the coordinate determination flag is set for the product determined in step S104 (YES in step S105), the image recognition unit 120 acquires the coordinates (position) of the product in the image (step S106). A coordinate determination flag is set in advance for a product to be subjected to identification processing using the coordinates at which the product is arranged. The coordinate determination flag is recorded in the identification information storage unit 151, for example. The coordinates of the product are represented, for example, as a set of coordinates indicating the area of the product in the image. Acquisition of the coordinates of the product in step S106 may be performed after determination of the coordinate determination flag in step S105 as shown in FIG. 7, or may be performed together with image recognition in step S103.

The image recognition unit 120 determines whether or not the product determined in step S104 is correct based on the coordinates of the product acquired in step S106 (step S107). Specifically, if the coordinates of the product acquired in step S106 are within the area corresponding to the product determined in step S104, the image recognition unit 120 determines that the correct product has been determined. On the other hand, if the coordinates of the product acquired in step S106 are not within the area corresponding to the product determined in step S104, the image recognition unit 120 determines that the wrong product has been determined.

For example, the following processing is performed. Here, for the sake of simplification, it is assumed that there are two types of products, a small product and a large product, as shown in FIG. The image recognition unit 120 acquires the correct position (first area F1 or second area F2) corresponding to the product determined in step S104. The correct position of the goods is preset. In the example of FIG. 5, the correct position for small items is the first area F1 and the correct position for large items is the second area F2. Then, the image recognition unit 120 determines whether the coordinates of the product acquired in step S106 are in the correct position (first area F1 or second area F2) corresponding to the product. This determination may be made by any method. For example, when more than half of the pixel group corresponding to the product is within the area in the image corresponding to the first area F1, the position of the product is the first area. It may be determined to be on region F1, otherwise it may be determined to be on second region F2. Alternatively, if the center (center of gravity) of the product area is in the first area F1, the position of the product is determined to be on the first area F1, otherwise it is determined to be on the second area F2. may be The areas in the image corresponding to the first area F1 and the second area F2 may be set in advance by the user, or the areas defined by the image recognition unit 120 to define the first area F1 and the second area F2. It may be set by recognizing the segmented portion 212 as an image.

The number of product types and the number of areas are not limited to two, and may be any number of at least two or more.

If the product is not in the correct position (NO in step S108), the product determined in step S104 is changed to the correct product (step S109). The correct product is the product corresponding to the area containing the product coordinates obtained in step S106. In the example of FIG. 5, if the coordinates of the product acquired in step S106 are within the first region F1, the correct product is the small product, and if they are within the second region F2, the correct product is the large product. It is a commodity.

If the coordinate determination flag is not set for the product determined in step S104 (NO in step S105) or if the product is in the correct position (YES in step S108), the product determined in step S104 is left as it is. use.

Finally, the product information acquisition unit 130 acquires product information from the product information storage unit 152 based on the product ID of the product determined in step S104 or changed in step S109. 13 is used to control the display of product information (step S110).

In particular, when the flowchart of FIG. 7 is executed at predetermined time intervals, product information is repeatedly acquired for the same product placed on the table 14, and the price of the same product is totaled multiple times in accounting. There is a risk that Therefore, when the image recognition unit 120 identifies the same product at an approximate position (for example, the difference in the position of the center of gravity is less than a predetermined value) within a predetermined time or within a predetermined number of frames, the product information acquisition unit 130 In step S110, product information about the product is not acquired (not shown in the flowchart of FIG. 7).

In the flowchart of FIG. 7, a product is provisionally determined in step S104, and if the coordinates of the product are in an incorrect area, the product is changed to a correct product in step S109, but the present invention is not limited to this specific form. For example, in step S104, a plurality of candidates for products similar to the product in the image may be extracted, and a candidate whose coordinates are correct among the plurality of candidates may be determined as the product in the image.

The CPU 101 of the POS terminal 100 is the subject of each step (process) included in the image recognition method shown in FIG. That is, the CPU 101 reads a program for executing the image recognition method shown in FIG. carry out the method.

The image recognition system 10 according to the present embodiment performs product image recognition using the method of the position where the product is arranged in addition to the appearance of the product. As a result, for products that are difficult to distinguish by image recognition only from the appearance of the product (in this embodiment, beverage cans with similar appearance but different sizes), image recognition can be performed based on the position where the product is placed. Since the results can be narrowed down, the accuracy of image recognition can be improved.

Also, since the angle at which the support portion 200 supports the product is fixed and known, the angle can be used to correct the image of the product. Thereby, the accuracy of image recognition can be further improved.

(Second embodiment)
In the first embodiment, positions are used to facilitate identification of products of different sizes, whereas in this embodiment, positions are used to facilitate identification of products of different types. This embodiment uses an image recognition system 10 having a configuration similar to that of the first embodiment. This embodiment may be used instead of the first embodiment, or may be used in combination with the first embodiment.

FIG. 8 is a top view of the recognition area A on the mounting table 14 according to this embodiment. In FIG. 8, the position of the imaging device 11 projected onto the mounting table 14 is indicated by a dashed line.

The recognition area A is divided into a first area G1 and a second area G2 by area dividing lines G. FIG. A region dividing line G is a line drawn on the recognition region A. FIG. The area dividing line G may be a symbol or color displayed on the recognition area A, light projected by the projection device 12, or a structure is fine. The image recognition unit 120 performs different recognition processes depending on which of the regions G1 and G2 the product is placed (that is, the position where the product is placed).

In order to discriminate between the areas G1 and G2, the image recognition unit 120 presets information indicating to which of the areas G1 and G2 each pixel in the image acquired by the imaging device 11 belongs, and identifies the area corresponding to the position of the product. In which of the regions G1 and G2 the product is located may be determined based on the pixels that are displayed. Alternatively, the image recognition unit 120 recognizes the position of the area dividing line G, and determines in which of the areas G1 and G2 the product is placed based on the positional relationship between the position of the area dividing line G and the product. may

The first type of product H1 and the second type of product H2 to be recognized in the present embodiment are products of different types. Specifically, the product H1 is a general product that is wrapped in a package representing the content of the product and has a fixed appearance. Since general articles have stable appearances, they can be easily identified by image recognition processing using appearances. On the other hand, the product H2 is a perishable food that is not wrapped in a package or is wrapped in a transparent package so that the product itself can be seen. Perishable foods have various appearances such as shapes and colors, and are difficult to identify by image recognition processing that uses appearances. Perishable foods are often labeled with the product name, price and bar code.

For example, when a product is placed in the first region G1, the image recognition unit 120 performs image recognition processing assuming that the product is a general product. That is, in the example of FIG. 8, image recognition processing for general goods is performed on the goods H1. When the product is placed in the second region G2, the image recognition unit 120 performs image recognition processing assuming that the product is fresh food. That is, in the example of FIG. 8, image recognition processing for perishables is performed on the product H2.

Image recognition processing for general goods performs identification using feature amounts (shape, color, color distribution, etc.) calculated from the entire product. Image recognition processing for perishables performs identification using a feature amount calculated from an area within a label attached to a product rather than the entire product. As a result, the accuracy of image recognition is improved because normal image recognition is applied to general goods, while image recognition is applied using label information with stable appearance for perishable foods with inconsistent appearance. can be made Specific image recognition processing for each type of product is not limited to those shown here, and any method suitable for the type may be used.

In actual operation, the image recognition unit 120 recognizes the type of product H1, which is a general product, in the first area G1 and the product H2, which is fresh food, in the second area G2. Appropriate image recognition processing can be applied to different products H1 and H2.

In other words, the image recognition unit 120 identifies the product based on the information on the appearance of the product and the information on the position where the product is arranged. Accordingly, an appropriate image recognition process can be selected depending on the position where the product is arranged, and the accuracy of identification can be improved.

The number of regions used by the image recognition unit 120 is not limited to two, and may be any number of at least two or more according to the types of products to be distinguished.

FIG. 9 is a diagram showing a flowchart of the image recognition method according to this embodiment. The flowchart of FIG. 9 is started when the user performs a predetermined operation for executing image recognition on the image recognition system 10 . For example, when the user presses a button on the input device 15 (keyboard or touch panel) connected to the POS terminal 100, or at predetermined time intervals, the flowchart of FIG. 9 is started.

First, the image acquiring unit 110 receives a signal indicating an image captured by the imaging device 11, and inputs the signal to the POS terminal 100 as image data (step S201).

Next, the image recognition unit 120 acquires the position of the product in the image acquired in step S201 (step S202). The following steps S203 to S207 are performed for each product detected in the image.

The image recognition unit 120 determines in which of the first area G1 and the second area G2 on the recognition area A the product is located. This determination may be made by any method, but for example, if more than half of the pixel group corresponding to the product is within the region in the image corresponding to the first region G1, the position of the product is the first region. It may be determined to be on region G1, otherwise it may be determined to be on second region G2. The areas in the image corresponding to the first area G1 and the second area G2 may be set in advance by the user, or the areas defined by the image recognition unit 120 to divide the first area G1 and the second area G2. It may be set by recognizing the division line G as an image.

If the position of the product acquired in step S202 is within the first region G1 (YES in step S203), the image recognition unit 120 performs general product image recognition processing on the product to identify the product. The product is identified based on the identification information in the information storage unit 151 (step S204).

When the position of the product acquired in step S202 is within the second region G2 (NO in step S203), the image recognition unit 120 performs image recognition processing for perishables on the product to identify the product. The product is identified based on the identification information in the information storage unit 151 (step S205).

The image recognition unit 120 then determines the product identified in step S204 or S205 as the product in the image (step S206).

Finally, the product information acquisition unit 130 acquires product information from the product information storage unit 152 based on the product ID of the product determined in step S206, and the display control unit 140 displays the product information using the display device 13. control is performed (step S207).

The CPU 101 of the POS terminal 100 is the subject of each step (process) included in the image recognition method shown in FIG. That is, the CPU 101 reads a program for executing the image recognition method shown in FIG. carry out the method.

As in the first embodiment, the image recognition system 10 according to the present embodiment performs product image recognition using the method of the position where the product is arranged in addition to the appearance of the product. As a result, it is possible to perform appropriate image recognition processing for each type of product (general goods and perishables in this embodiment) based on the position where the product is placed, thereby improving the accuracy of image recognition.

(Third embodiment)
In the first embodiment, the position is used to make it easier to identify products of different sizes, but in this embodiment, the accuracy of image recognition is improved even when the environment differs depending on the position in the recognition area. position is used to This embodiment uses an image recognition system 10 having a configuration similar to that of the first embodiment. This embodiment may be used in place of the first or second embodiment, or may be used in combination with the first or second embodiment.

FIG. 10 is a top view of the recognition area A on the mounting table 14 according to this embodiment. In FIG. 10, the position of the imaging device 11 projected onto the mounting table 14 is indicated by a dashed line.

The recognition area A is divided into a first area J1 and a second area J2 by applying different colors on each area. In FIG. 10, a different color than the first region J1 is indicated by representing a number of dots on the second region J2. If each area can be visually shown to the user, each area can be identified by a symbol or color displayed on the recognition area A, light projected by the projection device 12, or a structure provided on the recognition area A. may be represented. The image recognition unit 120 performs different recognition processes depending on which of the areas J1 and J2 the product is placed (that is, the position where the product is placed).

In order to discriminate between the areas J1 and J2, the image recognition unit 120 presets information indicating to which of the areas J1 and J2 each pixel in the image acquired by the imaging device 11 belongs, and identifies the area corresponding to the position of the product. In which of the regions J1 and J2 the product is located may be determined based on the pixels that are displayed. Alternatively, the image recognition unit 120 may specify the range based on the color of each area, and determine in which of the areas J1 and J2 the product is placed.

In this embodiment, the brightness of ambient light differs between the first region J1 and the second region J2. If there is a shielding object between the light source and the recognition area A, a shadow is formed on the recognition area A, so the brightness changes depending on the position on the recognition area A. FIG. As a result, the brightness of the image of the product acquired by the imaging device 11 changes depending on the position where the product is placed. can. In the example of FIG. 10, the first region J1 is a relatively bright bright region and the second region J2 is a relatively dark dark region.

For example, when the product is placed in the first area J1, the image recognition unit 120 performs image recognition processing suitable for bright conditions. That is, in the example of FIG. 10, image recognition processing for a bright region is performed on product K1. When the product is placed in the second area J2, the image recognition unit 120 performs image recognition processing suitable for dark conditions. That is, in the example of FIG. 10, image recognition processing for a dark area is performed on product K2.

For example, the image recognition process for the bright area performs image recognition after correcting the product image data to lower the brightness, and the image recognition process for the dark area performs the correction to increase the brightness of the product image data. After that, image recognition is performed. As a result, since image recognition can be performed with the brightness approaching a constant value, similar image recognition conditions can be set regardless of the position where the product is placed, and the accuracy of image recognition can be improved. Specific image recognition processing for products in each brightness area is not limited to the one shown here, and any method suitable for the brightness may be used. Here, the brightness of the image is simply corrected. , the processing may vary significantly from region to region.

In other words, the image recognition unit 120 identifies the product based on the information on the appearance of the product and the information on the position where the product is arranged. This makes it possible to apply appropriate image recognition processing to the environment of the position where the product is placed, and improve the accuracy of identification.

The number of regions used by the image recognition unit 120 is not limited to two, and may be any number according to the types of products to be distinguished. If the brightness of the recognition area A changes due to changes in the state of the light source such as the sun or fluorescent lamp, the range of each area and the processing performed on each area may be changed depending on the time and date. good.

FIG. 11 is a diagram showing a flowchart of the image recognition method according to this embodiment. The flowchart of FIG. 11 is started when the user performs a predetermined operation for executing image recognition on the image recognition system 10 . For example, when the user presses a button on the input device 15 (keyboard or touch panel) connected to the POS terminal 100, or at predetermined time intervals, the flowchart of FIG. 11 is started.

First, the image acquiring unit 110 receives a signal indicating an image captured by the imaging device 11, and inputs the signal to the POS terminal 100 as image data (step S301).

Next, the image recognition unit 120 acquires the position of the product in the image acquired in step S301 (step S302). The following steps S303 to S307 are performed for each product detected in the image.

The image recognition unit 120 determines in which of the first area J1 and the second area J2 on the recognition area A the product is located. This determination may be made by any method. For example, when more than half of the pixels corresponding to the product are within the area in the image corresponding to the first area J1, the position of the product is the first area. It may be determined to be on the region J1, otherwise it may be determined to be on the second region J2. The areas in the image corresponding to the first area J1 and the second area J2 may be set in advance by the user, or the image recognition unit 120 may determine the colors of the first area J1 and the second area J2 from the image. It may be set by recognizing

If the position of the product acquired in step S302 is within the first region J1 (YES in step S303), the image recognition unit 120 performs image recognition processing for a bright region on the product to identify the product. The product is identified based on the identification information in the information storage unit 151 (step S304).

If the position of the product acquired in step S302 is within the second region J2 (NO in step S303), the image recognition unit 120 performs dark region image recognition processing on the product to identify the product. The product is identified based on the identification information in the information storage unit 151 (step S305).

The image recognition unit 120 then determines the product identified in step S304 or S305 as the product in the image (step S306).

Finally, the product information acquisition unit 130 acquires product information from the product information storage unit 152 based on the product ID of the product determined in step S306, and the display control unit 140 displays the product information using the display device 13. control is performed (step S307).

The CPU 101 of the POS terminal 100 is the subject of each step (process) included in the image recognition method shown in FIG. That is, the CPU 101 reads a program for executing the image recognition method shown in FIG. carry out the method.

As in the first embodiment, the image recognition system 10 according to the present embodiment performs product image recognition using the method of the position where the product is arranged in addition to the appearance of the product. As a result, it is possible to perform image recognition processing suitable for the environment of the position where the product is placed, so that it is possible to improve the accuracy of image recognition.

(Other embodiments)
FIG. 12 is a schematic configuration diagram of the POS terminal 100 according to each of the above-described embodiments. FIG. 12 shows a configuration example for the POS terminal 100 to function as an image recognition device that performs image recognition using product position information. The POS terminal 100 acquires an image of a product placed in a recognition area, and identifies the product based on the appearance of the product in the acquired image and the position of the product in the recognition area. and an image recognition unit 120 for identifying.

The present invention is not limited to the above-described embodiments, and can be modified as appropriate without departing from the gist of the present invention.

A program for operating the configuration of each embodiment described above (for example, a program for causing the POS terminal 100 to execute the processes shown in FIGS. 7, 9, and 11) to realize the functions of each embodiment described above is recorded on a recording medium, and A processing method of reading a program recorded on a recording medium as a code and executing it on a computer is also included in the category of each embodiment. That is, a computer-readable recording medium is also included in the scope of each embodiment. In addition to recording media on which the computer programs described above are recorded, the computer programs themselves are also included in each embodiment.

For example, a floppy (registered trademark) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, nonvolatile memory card, and ROM can be used as the recording medium. Further, not only the program recorded on the recording medium alone executes the process, but also the one that operates on the OS and executes the process in cooperation with other software and functions of the expansion board. included in the category of

Some or all of the above-described embodiments can also be described in the following supplementary remarks, but are not limited to the following.

(Appendix 1)
an image acquisition unit that acquires an image of a product placed in the recognition area;
an image recognition unit that identifies the product based on the appearance of the product in the acquired image and the position of the product in the recognition area;
An image recognition device comprising:

(Appendix 2)
The recognition region is divided into at least two regions,
1. The image recognition device according to supplementary note 1, wherein the image recognition unit identifies the product using different processing for the at least two areas.

(Appendix 3)
Supplementary note 2, wherein the at least two areas are indicated by any of symbols or colors represented in the recognition area, light projected onto the recognition area, and structures provided in the recognition area. The image recognition device according to .

(Appendix 4)
The image acquisition unit acquires the image of the product supported by a support structure provided in the recognition area,
4. The image recognition unit according to any one of Appendices 1 to 3, wherein the product is identified based on the appearance of the product in the acquired image and the position of the product on the support structure. The described image recognition device.

(Appendix 5)
The image recognition unit according to appendix 4, wherein the acquired image is corrected based on the angle at which the support structure supports the product, and the product is identified using the corrected image. Image recognition device.

(Appendix 6)
The image recognition unit determines the size of the product based on the position of the product in the recognition area, and identifies the product using the size. The image recognition device according to any one of the items.

(Appendix 7)
The image recognition unit, when identifying the product, determines the type of the product based on the position of the product in the recognition area, and identifies the product using the type. , the image recognition device according to any one of Appendices 1 to 6.

(Appendix 8)
wherein, when identifying the product, the image recognition unit determines the brightness of the position in the recognition area of the product in the acquired image based on the position of the product in the recognition area; 8. The image recognition device according to any one of appendices 1 to 7, wherein the brightness is used to identify the product.

(Appendix 9)
obtaining an image of the product placed in the recognition area;
identifying the item based on the appearance of the item in the acquired image and the location of the item in the recognition area;
An image recognition method comprising:

(Appendix 10)
to the computer,
obtaining an image of the product placed in the recognition area;
identifying the item based on the appearance of the item in the acquired image and the location of the item in the recognition area;
program to run.

(Appendix 11)
a mounting table for mounting the product;
an image recognition device that performs image recognition on an image of the product acquired by an imaging device;
with
The image recognition device is
an image acquiring unit that acquires the image of the product placed in the recognition area on the placing table;
an image recognition unit that identifies the product based on the appearance of the product in the acquired image and the position of the product in the recognition area.

REFERENCE SIGNS LIST 10 image recognition system 11 imaging device 12 projection device 13 display device 14 mounting table 100 POS terminal 101 CPU
102 Memory 103 Storage Device 104 Interface 110 Image Acquisition Unit 120 Image Recognition Unit 130 Merchandise Information Acquisition Unit 140 Display Control Unit 200 Support Unit 211 Side Support Surface 212 Bottom Support Surface A Recognition Area

Claims (10)

an image acquisition unit that acquires an image of a product placed in a recognition area divided into at least two areas ;
an image recognition unit that identifies a product in the acquired image;
with
the at least two regions are separated by any of a symbol or color represented in the recognition region, light projected onto the recognition region, and a structure provided in the recognition region;
The image recognition device, wherein the image recognition unit identifies the product using image recognition processing selected based on the position of the product. 2. The image recognition apparatus according to claim 1, wherein said image recognition processing includes brightness correction processing for image data . The image acquisition unit acquires the image of the product supported by a support structure provided in the recognition area,
3. The image recognition according to claim 1, wherein the image recognition unit identifies the product based on the appearance of the product and the position of the product on the support structure in the acquired image. Device. 4. The product according to claim 3, wherein the image recognition unit corrects the acquired image based on an angle at which the support structure supports the product, and identifies the product using the corrected image. image recognition device. 4. The image recognition unit determines the size of the product based on the position of the product in the recognition area, and identifies the product using the size. The image recognition device according to any one of . The image recognition unit, when identifying the product, determines the type of the product based on the position of the product in the recognition area, and identifies the product using the type. The image recognition device according to any one of claims 1 to 5. wherein, when identifying the product, the image recognition unit determines the brightness of the position in the recognition area of the product in the acquired image based on the position of the product in the recognition area; The image recognition device according to any one of claims 1 to 6, wherein the brightness is used to identify the product. obtaining an image of a product placed in a recognition area segmented into at least two areas ;
identifying a product in the image obtained;
has
the at least two regions are separated by any of a symbol or color represented in the recognition region, light projected onto the recognition region, and a structure provided in the recognition region;
The image recognition method, wherein identifying the product includes identifying the product using an image recognition process selected based on the location of the product. to the computer,
obtaining an image of a product placed in a recognition area segmented into at least two areas ;
identifying a product in the image obtained;
An image recognition method comprising
the at least two regions are separated by any of a symbol or color represented in the recognition region, light projected onto the recognition region, and a structure provided in the recognition region;
identifying the item includes identifying the item using an image recognition process selected based on the location of the item;
A program that executes an image identification method. a mounting table for mounting the product;
an image recognition device that performs image recognition on an image of the product acquired by an imaging device;
with
The image recognition device is
an image acquiring unit that acquires the image of the product placed on the placing table in a recognition area divided into at least two areas ;
an image recognition unit that identifies a product in the acquired image;
with
the at least two regions are separated by any of a symbol or color represented in the recognition region, light projected onto the recognition region, and a structure provided in the recognition region;
The image recognition system, wherein the image recognition unit identifies the product using an image recognition process selected based on the position of the product.

Images