JP2022117990A - Information processor and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for accurately determining the validity of a commodity recognized by a device that registers the commodity to be settled.

SOLUTION: An information processor having a function of registering a commodity to be settled includes: first acquisition means for acquiring commodity information relating to the commodity to be settled; second acquisition means for acquiring image information including the commodity; and determination means for determining, based on the output of the first and second acquisition means, whether or not there is any irregularity in commodity registration. The image information includes at least one of an image of the commodity when the commodity is placed on a registered commodity placement site or an image of the commodity placed on the registered commodity placement site.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to technology for determining whether or not a product recognized by a device is a correct product.

In a device for registering products for payment, when the product for payment is recognized using a code for identifying the product, such as a so-called bar code, a weight sensor is provided at a place where the recognized product is temporarily placed. , there are techniques for judging the validity of goods recognized by the device. The term "appropriate" as used herein means that the device for registering the products to be settled recognizes the correct product as the product to be purchased by the customer. In this technology, it is determined whether or not the two match based on the weight measured by the weight sensor and the weight of the product linked to the product identification code.

In addition, for example, Patent Literature 1 describes an example of technology for appropriately determining the validity of products recognized by a device that registers products to be settled. In Patent Document 1, in response to reading a bar code for identifying a product, an image of an object (product) to which the bar code is attached and a basic figure of the product associated with the bar code are obtained. A technique is disclosed for determining the validity of a recognized product based on a correlation value calculated using the shape of an object recognized from an image and a basic figure associated with a read barcode.

Patent Literature 2 and Patent Literature 3 disclose techniques for identifying products by recognizing objects from images. Specifically, in Patent Documents 2 and 3, a product image is associated with a product ID, and the product image and an object included in the imaged image are identified by the color of the surface, the pattern, and the unevenness of the surface. A technique is disclosed in which matching is performed based on feature amounts such as surface conditions, shapes, etc., and product candidates corresponding to objects in an image are specified based on the degree of similarity. In addition, Patent Document 3 discloses a technique of detecting a region corresponding to a person's hand from an image using a thermo camera using far infrared rays, and performing the above-described feature amount matching by excluding the region. ing.

JP 2014-132501 A JP 2013-182323 A JP 2014-052806 A

It is preferable that the accuracy of the technology for judging the validity of the products recognized by the device for registering the products to be settled as described above is as high as possible.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a technique for accurately determining the validity of products recognized by a device for registering products to be settled.

According to the invention,
a first acquisition means for acquiring product information related to products to be settled;
a second acquiring means for acquiring image information including the product;
determination means for determining whether or not there is fraud related to product registration based on the outputs of the first and second acquisition means;
with
The image information includes at least one of an image when the product is placed on the registered product placement location or an image of the product placed on the registered product placement location,
An information processing device is provided.

According to the invention,
the computer
Acquire product information related to products for payment,
Acquiring image information including the product,
Determining whether there is fraud related to product registration based on the product information and the image information,
The image information includes at least one of an image when the product is placed on the registered product placement location or an image of the product placed on the registered product placement location,
A method of processing information is provided.

According to one aspect of the invention,
the computer,
a first acquisition means for acquiring product information related to products to be settled;
a second acquisition means for acquiring image information including the product;
Determining means for determining whether or not there is fraud related to product registration based on the outputs of the first and second obtaining means;
function as
The image information includes at least one of an image when the product is placed on the registered product placement location or an image of the product placed on the registered product placement location,
A program is provided.

ADVANTAGE OF THE INVENTION According to this invention, the validity of the goods recognized by the apparatus which registers the goods for payment can be judged accurately.

2 is a diagram conceptually showing the processing configuration of the information processing apparatus in the first embodiment; FIG. It is a figure which shows an example of the information which a size/shape information storage part memorize|stores. 1 is a diagram conceptually showing the hardware configuration of an information processing apparatus; FIG. 4 is a flow chart showing the flow of processing of the information processing apparatus of the first embodiment; It is a figure which shows the example of attachment of the information processing apparatus in 2nd Embodiment. FIG. 11 is a diagram conceptually showing the processing configuration of an information processing apparatus according to a third embodiment; 10 is a flow chart showing the flow of processing of the information processing apparatus of the third embodiment; FIG. 12 is a diagram conceptually showing a processing configuration of an information processing apparatus according to a fourth embodiment; It is a figure which shows an example of the information which a weight information storage part memorize|stores. FIG. 14 is a flow chart showing the flow of processing of the information processing apparatus of the fourth embodiment; FIG. FIG. 12 is a diagram conceptually showing a processing configuration of an information processing apparatus according to a fifth embodiment; 14 is a flow chart showing the flow of processing of the information processing apparatus of the fifth embodiment; It is a figure which shows notionally the system configuration|structure of the settlement system in 6th Embodiment. It is a sequence diagram which shows the flow of processing of the settlement system of the sixth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in all the drawings, the same constituent elements are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

[First embodiment]
[Processing configuration]
FIG. 1 is a diagram conceptually showing the processing configuration of an information processing apparatus 10 according to the first embodiment. The information processing device 10 is a product registration device such as a so-called POS (Point of Sales) terminal that has at least a function of registering products to be settled. The information processing apparatus 10 may further have a function of adjusting the price of the registered product. The information processing apparatus 10 of the present embodiment includes an image recognition unit 110, a product recognition unit 120 (a code acquisition unit 122, an imaging unit 124, and a thermography information acquisition unit 126), a size/shape information acquisition unit 130, a size/shape An information storage unit 132 and a determination unit 140 are provided.

<Registration of products for payment>
In the present embodiment, products to be settled are registered as follows.

Items to be settled are moved as indicated by the dotted line in FIG. That is, the product before it is registered as the product to be settled (hereinafter, such product is referred to as an “unregistered product”) is taken out from the place where the unregistered product is temporarily placed, and the product recognition unit 120 position. Here, as will be described later in detail, the unregistered product is registered as the product for payment. Then, the product registered as the object of payment (such product is hereinafter referred to as "registered product") is moved to a place where the registered product is temporarily placed.

Unregistered commodities are registered as commodities for settlement in the following manner, for example. First, the code acquisition unit 122 of the product recognition unit 120 acquires the product identification code of the unregistered product held over the product recognition unit 120 as the product identification code of the product to be settled. Here, the product identification code is data formed from letters, numbers, symbols, etc., which is predetermined for specifying each product. A PLU (Price Look Up) code (for example, a JAN (Japan Article Number) code) or a Non-PLU code can be used as the product identification code.

The product identification code is obtained by detecting and analyzing the product information symbol from the product to be settled. A product information symbol is a symbol specifying information about a product. Here, the symbol is a bar code, a two-dimensional code (QR code (registered trademark), etc.), a character string symbol, or the like. Note that the character string here also includes a numeric string. The product information symbol is a barcode or the like in which information specifying product information (such as product information ID) is encoded, or a character string symbol representing information specifying product information. In this case, for example, by using a code reader having a light source and a light receiving element such as a photodiode, or a code reader having a lens and an imaging element such as a CCD (Charge Coupled Device) image sensor, the product information symbol can be converted to the product identification code. is obtained. In the case of a code reader equipped with a lens and an image pickup device, the feature amount of the product (for example, color, shape, packaging, or a specific mark or character string that identifies the product) is extracted from the image containing the product to be paid. It is also possible to acquire the product identification code by performing matching processing with reference to a storage unit (not shown) that stores the product identification code and the feature amount of the product in association with each other.

Then, the product recognition unit 120 uses the product identification code acquired by the code acquisition unit 122 to associate the product identification code with the product information (for example, product name, product unit price, discount, discount, etc.). By referring to a storage unit (not shown) that is attached and stored, the product information associated with the acquired product identification code is read. Then, the product recognition unit 120 adds the read product information as information to be used for settlement processing executed by the information processing device 10 . As a result, the pre-registration product is recognized on the information processing apparatus 10 and registered as a product for payment.

<Determination of validity of products recognized by device>
Further, in the present embodiment, the validity of the product registered as the settlement object as described above is determined by each processing unit described below. It should be noted that "appropriate" in the present embodiment means that the information processing apparatus 10 recognizes the correct product as the product to be purchased by the customer.

The imaging unit 124 is a so-called 3D camera or the like, and is configured to generate an image by associating depth information indicating the distance in the depth direction of the product (unregistered product) included in the imaging range. The imaging unit 124 generates the image and the depth information in association with each other by a known method such as a method using parallax in a stereo camera. The imaging unit 124 of this embodiment includes the vicinity of the product recognition unit 120 in its imaging range, and its orientation and position are adjusted so that the unregistered product that has been moved to the product recognition unit 120 can be imaged. Note that the imaging unit 124 may be provided integrally with the information processing apparatus 10, such as a camera used when recognizing a product as an object, for example, or may be provided as an external camera, such as an external camera. It may be installed around the processing device 10 .

Also, the imaging unit 124 preferably captures an image at the timing when the code acquisition unit 122 reads the product identification code. For example, when the code acquisition unit 122 detects the existence of the product information symbol of the product, the code acquisition unit 122 sends a notification to that effect to the imaging unit 124 . Upon receiving this notification, the imaging unit 124 can know the timing at which the code acquisition unit 122 reads the code. In this way, in the case of a product having the product information symbol described above, if the imaging unit 124 captures an image at the timing when the code acquisition unit 122 reads the product identification code, the posture of the product included in the image is stabilized. , the change in pose for each acquired image is small. As a result, the effect of reducing the processing load by reducing the number of templates used in image matching and the effect of improving recognition accuracy by image matching can be expected.

The thermography information acquisition unit 126 is a so-called thermography camera that uses infrared rays or the like, captures infrared radiant energy emitted by an object included in an imaging range, and generates thermography information indicating the temperature distribution of the object. The thermography information acquisition unit 126 is configured to acquire thermography information in synchronization with the imaging timing of the imaging unit 124 . Further, the position and orientation of the thermography information acquisition unit 126 are adjusted so that the thermography information of the same imaging range as the imaging range of the imaging unit 124 or the imaging range including the imaging range of 124 can be acquired. Then, the thermography information acquisition unit 126 aligns the positional relationship between the image generated by the imaging unit 124 and the thermography information by, for example, image space coordinate conversion processing. In this manner, the thermography information acquisition unit 126 acquires thermography information associated with the image generated by the imaging unit 124 .

The image recognition unit 110 performs the following processing using the image generated by the imaging unit 124 and the thermography information acquired by the thermography information acquisition unit 126 .

The image recognition unit 110 performs known image recognition processing on the image generated by the imaging unit 124 to detect, for example, edges of products in the image. Then, the image recognition unit 110, for example, matches the shape indicated by the detected edge with preset two-dimensional or three-dimensional template data, and recognizes the shape with the highest degree of similarity as the product shape. do. Furthermore, the image recognition unit 110 uses the depth information associated with the image generated by the imaging unit 124 and the size of the area occupied by the product within the image to determine, for example, the actual size of the product within the image. (For example, height, width, depth width, area of each surface, etc.) are calculated. Since the actual distance from the imaging device to the product can be determined based on the depth information, the image recognition unit 110 converts the size of the area occupied by the product in the image into the actual size of the product based on the distance. be able to.

In addition, the image recognition unit 110 uses the thermography information acquired by the thermography information acquisition unit 126 to exclude a region estimated to be a person from the product shape recognized from the image generated by the imaging unit 124 . For example, the image recognition unit 110 is preset with information indicating, for example, the temperature range of a person (for example, 35 to 40° C.), and the image recognition unit 110 identifies the area indicating the set temperature in the thermography information. Extract from Based on the positional relationship between the thermography information acquired by the thermography information acquisition unit 126 and the image generated by the imaging unit 124, the image recognition unit 110 converts the extracted region from the target region for edge detection processing. The corresponding area is excluded, or the area corresponding to the extracted area is excluded from the recognized shape based on the detected edge. As a result, the image recognition unit 110 can recognize a more accurate area as the area corresponding to the product from the image generated by the imaging unit 124, and as a result, the actual size of the product converted from the area can be obtained. can be calculated more accurately.

The size/shape information acquisition unit 130 refers to the size/shape information storage unit 132 using the product identification code acquired by the code acquisition unit 122, and obtains the size/shape associated with the product identification code. Get information. The size/shape information is information indicating the size of each product (for example, the height, width, depth of the product, and the area of each surface calculated from these) and the external shape of each product. The size/shape information storage unit 132 stores the product identification code and the size/shape information in association with each other, as shown in FIG. 2, for example. FIG. 2 is a diagram showing an example of information stored in the size/shape information storage unit 132. As shown in FIG. Information stored in the size/shape information storage unit 132 is not limited to the example in FIG.

The size/shape information acquisition unit 130 searches the size/shape information storage unit 132 to detect the same product identification code as the product identification code acquired by the code acquisition unit 122, and assigns the detected product identification code Get the associated size/shape information. 1 shows an example in which the size/shape information storage unit 132 is provided in the information processing apparatus 10, the size/shape information storage unit 132 communicates with the information processing apparatus 10. It may be provided in other possible devices. In this case, the size/shape information acquisition unit 130 acquires the size/shape information associated with the product identification code acquired by the code acquisition unit 122 by communicating with the other device.

The determination unit 140 determines whether the size or shape recognized from the image generated by the imaging unit 124 and the size/shape information obtained using the product identification code satisfy a predetermined standard. do. Here, the predetermined standard is a standard relating to the size or shape of the product, and enables confirmation that the product recognized from the image and the product identified from the product identification code indicate the same product. It is the standard. For example, the determination unit 140 calculates the degree of similarity between the two according to a predetermined similarity calculation algorithm related to size and shape, and the degree of similarity is set in advance as a predetermined reference value (a value considered to be the same product). threshold) or more.

As a result of determination, if the degree of matching between the two is equal to or greater than a predetermined reference value, there is a possibility that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are the same. is high, and the determination unit 140 can determine that the recognition result by the information processing apparatus 10 is valid. On the other hand, if the degree of matching between the two is less than the predetermined reference value, there is a high possibility that the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are different. The unit 140 can determine that the recognition result by the information processing device 10 is invalid. Also, in this case, it is conceivable that some kind of defect has occurred, such as a code reading mistake or an incorrect code being pasted. Therefore, when the degree of matching between the two is less than a predetermined reference value, the determination unit 140 displays a message prompting reloading of the product, a predetermined warning message, or the like on the display unit of the information processing device 10, or audibly It may be configured to output

In addition, in this embodiment, the size/shape information storage unit 132 may store a plurality of size/shape information in association with each product identification code. Specifically, the size/shape information storage unit 132 stores, for example, size/shape information when viewed from the front, size/shape information when viewed from above, and size/shape information when viewed obliquely from above. Size/shape information, such as size/shape information, indicating the posture of the product viewed from a plurality of directions may be stored in association with each product identification code. In this case, the determination unit 140 calculates the degree of similarity between the size or shape recognized by the image recognition unit 110 and the size or shape of each of the plurality of size/shape information linked to each product identification code. is equal to or greater than a predetermined reference value. By doing so, it is possible to prevent the accuracy of the determination by the determination unit 140 from being lowered due to the posture of the product included in the image generated by the imaging unit 124 .

[Hardware configuration]
FIG. 3 is a diagram conceptually showing the hardware configuration of the information processing apparatus 10. As shown in FIG. As shown in FIG. 3, the information processing apparatus 10 includes a CPU 101, a memory 102, a storage 103, an input/output interface (input/output I/F) 104, a communication module 105, and the like.
The CPU 101, the memory 102, the storage 103, the input/output interface 104, and the communication module 105 are connected by a data transmission line for mutual data transmission/reception.

The memory 102 is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage 103 is a storage device such as a hard disk, SSD (Solid State Drive), or memory card. The storage 103 includes the image recognition unit 110, the product recognition unit 120, the code acquisition unit 122, the imaging unit 124, the thermography information acquisition unit 126, the size/shape information acquisition unit 130, and the determination unit 140 of the information processing apparatus 10. It stores program modules for realizing the functions of the processing unit. The CPU 101 realizes the function of each processing unit by executing each program module. When executing each module, the CPU 101 may execute these modules after reading them onto the memory 102 , or may execute them without reading them onto the memory 102 . The storage 103 also functions as a size/shape information storage unit 132 .

The input/output interface 104 is connected to a display device 1041, an input device 1042, an imaging device 1043, a reading device 1044, and the like. The display device 1041 displays a screen, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display, corresponding to drawing data processed by the CPU 101 or a GPU (Graphics Processing Unit) (not shown). It is a device. One or more display devices 1041 may be connected to the input/output interface 104 . The input device 1042 is a device that receives input by an operator's operation, and is implemented as, for example, a keyboard, mouse, touch sensor, or the like. The display device 1041 and the input device 1042 may be integrated and implemented as a touch panel. The imaging device 1043 is a so-called 3D camera or the like, and includes a monocular imaging module or a binocular imaging module (not shown). In this case, the imaging device 1043 can implement the functions of the imaging unit 124 described above. The imaging device 1043 is a so-called thermography camera or the like, and includes an imaging module (not shown) that detects infrared energy emitted from an object and generates an image showing the temperature distribution of the object based on the amount of infrared energy. In this case, the imaging device 1043 can implement the function of the thermography information acquisition unit 126 described above. The reading device 1044 is a device such as a code reader including a light source and a light receiving element such as a photodiode, or a code reader including a lens and an imaging element such as a CCD (Charge Coupled Device) image sensor. The reading device 1044 can implement the functions of the code acquisition unit 122 described above.

A communication module 105 is used to transmit and receive data to and from an external device or the like. There are various methods for connecting the information processing device 10 and an external device via the communication module 105 . For example, this connection is a bus connection via a bus line (for example, a USB (Universal Serial Bus) line) or a network connection via a network line. Note that the network line may be a wireless line or a wired line.

Note that the hardware configuration of the information processing device 10 is not limited to the configuration shown in FIG. For example, the input/output interface 104 may be connected to various devices (money insertion device, change dispensing device, etc.) necessary for product settlement processing.

[Example of operation]
An operation example of the information processing apparatus 10 of this embodiment will be described with reference to FIG. FIG. 4 is a flow chart showing the flow of processing of the information processing apparatus 10 of the first embodiment.

When the product is moved to the position of the product recognition unit 120, the code acquisition unit 122 acquires the product identification code of the product (S101). Further, in response to the acquisition of the product identification code, the imaging unit 124 generates an image of the product in association with the depth information indicating the distance in the depth direction (S102), and the thermography information acquisition unit 126 causes the imaging unit 124 to generate Thermography information is acquired in association with the image obtained (S103).

Then, the image recognition unit 110 recognizes the size or shape of the product using the generated image and the depth information and thermography information associated with the image (S104). For example, the image recognition unit 110 uses the actual distance to the product obtained from the depth information to convert the size of the area occupied by the product in the image into the actual size of the product. Further, for example, the image recognition unit 110 extracts the product area from the image using known image recognition processing such as edge processing, and obtains the shape of the product as the shape of the product.

In addition, the size/shape information acquisition unit 130 refers to the size/shape information storage unit 132 using the product identification code acquired by the code acquisition unit 122, and determines the size/shape information associated with the product identification code. The shape information is read from the size/shape information storage unit 132 (S105).

The determination unit 140 compares the size or shape of the product recognized by the image recognition unit 110 with the size/shape information linked to the product identification code, and determines whether or not it satisfies a predetermined criterion. (S106). For example, the determining unit 140 may calculate the degree of similarity between the two according to a predetermined degree of similarity calculation algorithm, and determine whether or not the two satisfy a predetermined criterion based on whether the degree of similarity is equal to or greater than a predetermined threshold. can. When determining that the predetermined criteria are satisfied (S106: YES), the determination unit 140 determines that the information processing apparatus 10 has normally recognized the product, and does not execute the process of S107 described later. On the other hand, if the determination unit 140 determines that the predetermined criteria are not satisfied (S106: NO), the determination unit 140 outputs a predetermined output (for example, a message prompting to reload the product, a warning message, a warning sound, etc.) to a display, speaker, or the like (not shown). (S107).

[Action and effect of the first embodiment]
As described above, according to the present embodiment, for a product read by the product recognition unit 120, the actual size or shape of the product calculated from the image and the product identification code associated with the product A degree of similarity with size/shape information indicating the size or shape of is calculated. Then, it is determined whether or not both satisfy a predetermined standard depending on whether or not the degree of similarity between the two is equal to or greater than a predetermined threshold (threshold set in advance as a value for regarding the same product). If the sizes or shapes of both substantially match and satisfy a predetermined standard, the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are the same. There is a high possibility that there is, and the determination unit 140 determines that the recognition result by the information processing apparatus 10 is valid. On the other hand, if the sizes or shapes of the two do not match and the predetermined criteria are not met, the product moved to the position of the product recognition unit 120 and the product recognized by the information processing device 10 are different. , and the determination unit 140 determines that the recognition result by the information processing apparatus 10 is invalid. In this way, by comparing the actual size or shape of the product recognized from the image with the theoretical size or shape of the product linked to the product identification code, the information processing device 10 recognizes the product. It is possible to accurately judge the validity of the purchased product.

[Second embodiment]
There is also an operation method in which a device that reads the products to be purchased is attached to the basket that the customer carries with the products to be purchased, and when the product is put in the basket, it is registered as the product to be paid for, and only the payment is made at the cash register terminal. . In such a case, by equipping the basket with a device having the same configuration as the information processing device 10 of the first embodiment, the same effects as those of the first embodiment can be obtained.

[Processing configuration/Hardware configuration]
Similar to the configuration shown in FIG. An information acquisition unit 130 , a size/shape information storage unit 132 and a determination unit 140 are provided. The information processing apparatus 10 of this embodiment also has the same hardware configuration as that of the first embodiment, and implements each processing unit in the same manner as in the first embodiment. However, each hardware component of the information processing apparatus 10 shown in FIG. 4 is made smaller than that of the first embodiment in order to be attached to the basket 20 .

FIG. 5 is a diagram showing an example of attachment of the information processing device 10 according to the second embodiment. The information processing device 10 is attached to the basket 20 as shown in FIG. As shown in FIG. 5, the unregistered product is placed in the basket 20 after being registered as a product for settlement in the information processing device 10 .

[Example of operation]
The information processing apparatus 10 of this embodiment operates in the same manner as the information processing apparatus 10 of the first embodiment, as shown in FIG.

[Action and effect of the second embodiment]
As described above, according to the present embodiment, it is possible to obtain the same actions and effects as those of the first embodiment.

[Third Embodiment]
This embodiment is the same as the first and second embodiments except that the information processing apparatus 10 further includes a number acquisition unit 150 .

[Processing configuration]
FIG. 6 is a diagram conceptually showing the processing configuration of the information processing apparatus 10 according to the third embodiment. As shown in FIG. 6 , the information processing apparatus 10 of this embodiment further includes a number acquisition unit 150 .

The number acquisition unit 150 acquires the number of products included in the imaging range of the image generated by the imaging unit 124 . Specifically, the number acquisition unit 150 acquires the number of product shapes recognized by the image recognition unit 110 as the number of products. Alternatively, the number acquiring unit 150 acquires the number of objects detected from the image by pattern matching of feature amounts of products stored in advance as the number of products.

The determination unit 140 of the present embodiment determines whether or not the number of product identification codes acquired by the code acquisition unit 122 and the number of products acquired by the number acquisition unit 150 match. Note that "match" here means perfect match. In this embodiment, when a plurality of product information symbols are present in an image, the code acquisition unit 122 detects each product information symbol by performing image recognition processing, for example, and obtains a plurality of product identification codes at substantially the same timing. get. The code acquisition unit 122 of this embodiment further notifies the determination unit 140 of the number of product identification codes acquired here.

[Hardware configuration]
The information processing apparatus 10 of this embodiment also has a hardware configuration as shown in FIG. 4, like the first and second embodiments. The storage 103 further stores a program module that implements the function of the number acquisition unit 150, and the number acquisition unit 150 is implemented by the CPU 101 executing the program module.

[Example of operation]
An operation example of the information processing apparatus 10 of the present embodiment will be described with reference to FIG. FIG. 7 is a flow chart showing the processing flow of the information processing apparatus 10 of the third embodiment. Note that the steps (S201 to S203) different from those of the first and second embodiments will be mainly described below.

When acquiring the product identification code, the code acquisition unit 122 totalizes the number of acquired product identification codes (S201). Then, the number acquisition unit 150 acquires the number of product shapes recognized by the image recognition unit 110 as the number of products included in the image generated by the imaging unit 124 (S202). In addition, the determination unit 140 compares the number of product identification codes counted in S201 and the number of products acquired in S202, and confirms whether or not they match (S203). If the number of products and the number of product identification codes do not match (S203: NO), the determination unit 140 determines that an error such as misreading of the product has occurred, and prompts, for example, to reread the product. A predetermined message such as a message or warning message is displayed on the display or output as audio information (S107). On the other hand, if the number of products and the number of product identification codes match (S203: YES), the determination unit 140 determines that an error such as a product reading error has not occurred, and the details are checked in the first embodiment. As described above, it is checked whether or not the size or shape obtained from the image and the size/shape information obtained from the product identification code satisfy predetermined criteria (S106).

[Action and effect of the third embodiment]
As described above, in the present embodiment, it is further determined whether or not the number of products recognized from the image matches the number of product identification codes actually recognized by the information processing apparatus 10 . Thus, according to the present embodiment, it is possible to determine whether or not an error such as an error in reading a product has occurred. can be encouraged to accurately recognize

[Fourth Embodiment]
In this embodiment, a technique using a weight sensor is combined with the information processing apparatus 10 described in each of the above embodiments.

If the validity of the product recognized by the information processing device 10 is determined based on the weight measured by the weight sensor and the weight of the product linked to the product identification code, the weight of the product is extremely light. Also, if there is variation in the weight of the same product, correct judgment results may not be obtained. When targeting such products, the information processing apparatus 10 of the present embodiment acquires from the size or shape of the product recognized from the image and the product identification code, as described in each of the above embodiments. Recognition validity is determined based on the received size/shape information.

[Processing configuration]
FIG. 8 is a diagram conceptually showing the processing configuration of the information processing apparatus 10 in the fourth embodiment. Note that the information processing apparatus 10 of the present embodiment includes weight sensors (160 and 162) at the placement location of the unregistered product and the placement location of the registered product, respectively. The validity of the product recognized by the information processing apparatus 10 is determined based on the amount of change in weight.

As shown in FIG. 8 , the information processing apparatus 10 of this embodiment further includes a weight information storage section 142 . The weight information storage unit 142 stores, in association with the product identification code, information indicating the weight per product of the product identification code (hereinafter referred to as weight information) or information indicating the weight deviation of the product (hereinafter referred to as weight deviation). , weight deviation information). An example of information stored in the weight information storage unit 142 is shown in FIG. FIG. 9 is a diagram showing an example of information stored in the weight information storage unit 142. As shown in FIG. In the example of FIG. 9, the weight information storage unit 142 stores weight information and weight deviation information in association with product identification codes.

The determination unit 140 of the present embodiment acquires the product identification code from the code acquisition unit 122, and uses the product identification code from the weight information storage unit 142 to associate with the product identification code acquired by the code acquisition unit 122. Get weight information or weight deviation information. When the weight information storage unit 142 stores information as shown in FIG. 9, the determination unit 140 can acquire weight information and weight deviation information. Based on the acquired weight information or weight deviation information, the determining unit 140 determines whether (1) the weight associated with the acquired product identification code is equal to or greater than a reference value, or (2) the acquired product identification code and (3) the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold. judge.

These conditions are conditions for determining whether or not the product with the product identification code acquired by the code acquisition unit 122 is not suitable for processing by the weight sensor. In the case of (1) above, for example, if the product is extremely light, the amount of change in weight that can be detected by the weight sensor is small. It is a condition to The above (2) is a condition for judging whether or not the product cannot be processed by the weight sensor in the first place when the weight is not set. Further, the above (3) is a condition for judging whether or not a product with a large variation in weight is prone to erroneous recognition in processing by the weight sensor.

If the determination unit 140 of the present embodiment determines that any of the above (1) to (3) applies, the determination described in each of the above embodiments, that is, the information processing using the size or shape of the product A process of determining validity of recognition by the device 10 is executed. Specifically, when the determination unit 140 of the present embodiment determines that any of the above (1) to (3) is applicable, the image recognition unit 110 and the size/shape information acquisition unit 130 receive the above-described Notify the execution request of the processing described in each embodiment. The image recognition unit 110 that has received this notification sends the size or shape of the product recognized from the image generated by the imaging unit 124 to the determination unit 140 . Also, the code acquisition unit 122 that has received this notification sends the size/shape information associated with the product identification code acquired by the code acquisition unit 122 to the determination unit 140 . Using the product size or shape received from the image recognition unit 110 and the size/shape information received from the size/shape information acquisition unit 130, the determination unit 140 uses the size/shape information received from the size/shape information acquisition unit 130 to , the determination process is executed.

[Hardware configuration]
The information processing apparatus 10 of this embodiment also has a hardware configuration as shown in FIG. 4, like the first and second embodiments. The storage 103 further stores program modules that implement the functions of the determination unit 140 of the present embodiment, and the CPU 101 executes the program modules to implement the determination unit 140 of the present embodiment. Also, the storage 103 can implement the function of the weight information storage unit 142 .

[Example of operation]
An operation example of the information processing apparatus 10 of this embodiment will be described with reference to FIG. FIG. 10 is a flow chart showing the processing flow of the information processing apparatus 10 of the fourth embodiment. In the following description, based on the flowchart of FIG. 4, the processing (S301 to S303) added in this embodiment will be mainly described.

Using the product identification code obtained by the code obtaining unit 122 in S101, the determination unit 140 refers to the weight information storage unit 142 and obtains weight information or weight deviation information associated with the product identification code (S301). . Then, the determination unit 140 uses the acquired weight information or weight deviation information to determine whether the product with the product identification code acquired by the code acquisition unit 122 in S101 is not suitable for determination processing using a weight sensor. is determined (S302). Specifically, the determination unit 140 determines whether (1) the weight associated with the acquired product identification code is equal to or greater than a reference value, or (2) the weight associated with the acquired product identification code is not set. and (3) that the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold.

If the product with the product identification code acquired by the code acquisition unit 122 in S101 is not suitable for determination processing using a weight sensor (S302: YES), the determination unit 140 determines the image recognition unit 110 and the size/ The shape information acquiring unit 130 is notified to that effect, and the processing described in the first embodiment is executed (S102 to S107). On the other hand, if the product with the product identification code acquired by the code acquisition unit 122 in S101 is a product that does not cause any problem even if the determination process using the weight sensor is performed (S302: NO), the determination process using the weight sensor is performed. is executed (S303).

[Action and effect of the fourth embodiment]
As described above, in the present embodiment, it is determined whether or not the product having the product identification code acquired by the code acquisition unit 122 is suitable for the determination process using the weight sensor. Then, when it is determined that the product with the product identification code acquired by the code acquisition unit 122 is not suitable for the determination process using the weight sensor, the size or shape of the product recognized from the image and the product identification code Determination processing based on the associated size/shape information is executed. As a result, it is possible to prevent erroneous recognition that may occur when a weight sensor or the like is used, and improve the accuracy of determining the validity of the product recognized by the information processing apparatus 10 .

[Modification of the fourth embodiment]
In the present embodiment, the determination unit 140 basically performs determination processing based on the amount of change in weight measured by the weight sensor, and for products that meet predetermined conditions, the size or shape is recognized from the image. An example of performing the determination process based on is shown. However, not limited to this, the determination unit 140 basically performs determination processing based on the size or shape recognized from the image, and the weight sensor is used for products that satisfy conditions different from the above predetermined conditions. It may be configured to perform determination processing based on the amount of change in the measured weight. Here, "products that meet different conditions" are products that are not suitable for image recognition processing, such as thin products that have no thickness. . In such a case, the information processing apparatus 10 stores a product identification code of a product not suitable for image recognition processing in a storage unit (not shown), and the determination unit 140 determines whether the product identification code stored in the storage unit When the same product identification code is acquired by the code acquisition unit 122, the validity of the product recognized by the information processing device 10 is determined based on the change in weight measured by the weight sensor.

[Fifth embodiment]
In this embodiment, an information processing apparatus 10 having a minimum configuration capable of obtaining the effects of the present invention will be described. A device (for example, a server device) separate from a device for registering products to be settled (hereinafter referred to as a product registration device) is provided so as to be communicable with the product registration device, for example, via a LAN (Local Area Network).

[Processing configuration]
FIG. 11 is a diagram conceptually showing the processing configuration of the information processing apparatus 10 according to the fifth embodiment. As shown in FIG. 11 , the information processing apparatus 10 of this embodiment includes an image recognition section 110 , a code acquisition section 122 , a size/shape information acquisition section 130 and a determination section 140 .

The image recognition unit 110 recognizes the size or shape of the product in the image using the image associated with the depth information indicating the distance in the depth direction of the product included in the imaging range. The code acquisition unit 122 and the size/shape information acquisition unit 130 acquire size/shape information linked to the acquired product identification code. The determination unit 140 determines whether or not the size or shape of the product recognized from the image and the size/shape information obtained using the product identification code satisfy predetermined criteria.

[Hardware configuration]
The information processing apparatus 10 of this embodiment has a hardware configuration as shown in FIG. Note that the input/output interface 104 of the information processing apparatus 10 of the present embodiment is not connected to the imaging device 1043 and the reading device 1044, and can acquire images and product identification codes from other devices via the communication module 105. may be configured to The image recognition unit 110, the code acquisition unit 122, the size/shape information acquisition unit 130, and the determination unit 140 of this embodiment are program modules that the CPU 101 implements the functions of each processing unit, as in the above-described embodiments. It is realized by executing

[Example of operation]
An operation example of the information processing apparatus 10 according to the present embodiment will be described with reference to FIG. 12 . FIG. 12 is a flow chart showing the processing flow of the information processing apparatus 10 of the fifth embodiment.

The code acquisition unit 122 communicates with the product registration device via a network such as a LAN, for example, and acquires the product identification code of the product for payment read by the product registration device (S401). Then, the size/shape information acquisition unit 130 uses the product identification code acquired in S401 to read the size/shape information associated with the product identification code (S402). In this embodiment, the size/shape information is stored, for example, in a storage unit (not shown) of another device that can communicate with the information processing device 10, in association with the product identification code.

Also, the image recognition unit 110 communicates with the product registration device via a network such as a LAN, and obtains an image generated by an imaging unit provided in the product registration device (S403). The imaging means provided in this product registration apparatus is a so-called 3D camera, which can generate an image and depth information in association with each other. Then, the image recognition unit 110 recognizes the shape of the product in the image using known image recognition processing, and recognizes the shape of the product in the image based on the depth information associated with the image and the size of the area of the product in the image. The actual size of the product is calculated (S404).

The determination unit 140 determines whether or not the size/shape information acquired in S402 and the size and shape of the product acquired in S404 satisfy a predetermined standard (S405), and sends the result to the product registration device. (S406).

[Actions and Effects of the Fifth Embodiment]
As described above, in this embodiment, the actual size or shape of the product recognized from the image is compared with the theoretical size or shape of the product linked to the product identification code. As a result, it is possible to accurately determine the validity of the product recognized by the product registration device.

[Sixth Embodiment]
In this embodiment, a settlement system including a product registration device that performs registration processing of products to be settled and a settlement device that performs settlement processing of the price of the products registered as settlement targets will be described.

〔System configuration〕
FIG. 13 is a diagram conceptually showing the system configuration of the settlement system 1 in the sixth embodiment. As shown in FIG. 13, the settlement system 1 includes an information processing device 10 that functions as a product registration device, a settlement device 30 that performs settlement processing for the price of the product registered in the information processing device 10, and a server device 40. ,including. In this case, the information processing device 10, the settlement device 30, and the server device 40 are communicably connected via a network such as a LAN. Also, the information processing apparatus 10 has a configuration similar to that of any one of the first to fourth embodiments. In the following description, the information processing apparatus 10 of this embodiment has the configuration of FIG. 1 as in the first embodiment. Note that the image recognition unit 110, the code acquisition unit 122, the size/shape information acquisition unit 130, and the determination unit 140 of the information processing apparatus 10 are not limited to the example of FIG. 13, as described in the fifth embodiment. A device (for example, server device 40) separate from the product registration device may be provided so as to be communicable with the product registration device via, for example, a LAN. In this case, the product registration device has a processing unit that acquires the product identification code from the product information symbol or the like, in addition to the code acquisition unit 122 of the information processing device 10 .

[Example of operation]
The processing flow of the settlement system 1 of the present embodiment will be described with reference to FIG. 14 . FIG. 14 is a sequence diagram showing the flow of processing of the settlement system 1 of the sixth embodiment.

The information processing apparatus 10 executes the processes of S101 to S105 in the flowchart of FIG. 4, as described in the first embodiment. Then, the determination of S106 in the flowchart of FIG. 4 is performed. When the determination unit 140 determines that the predetermined criteria are not satisfied (S106: NO), the determination unit 140 performs a predetermined output (for example, an output prompting re-recognition of the product) (S107), and the processing of S101 to S105 is performed. Executed again. On the other hand, when the determination unit 140 determines that the predetermined criteria are satisfied (S106: YES), the product recognition unit 120 determines that the product is normally recognized by the information processing device 10, and the code acquisition unit 122 acquires the code. Product information linked to the product identification code is read (S501). Specifically, using the product identification code acquired by the code acquisition unit 122, the product recognition unit 120 recognizes the product identification code and product information (for example, product name, unit price of product, whether or not there is a discount, discount, etc.). A storage unit (not shown) that associates and stores the information is read, and product information associated with the acquired product identification code is read. Then, the merchandise recognition unit 120 adds the read merchandise information as information (hereinafter referred to as settlement information) used for settlement processing executed by the information processing apparatus 10 (S502). The settlement information includes at least the name of each product to be settled, the unit price and quantity of each product to be settled, the total price of all the products to be settled, and the like. The processes of S101 to S107, S501, and S502 are repeated until the registration of the product to be settled in one transaction is completed (S503: NO). The information processing apparatus 10 detects, for example, an event such as pressing of a "subtotal" button by an input device provided in the information processing apparatus 10, and detects completion of registration of products to be settled in one transaction.

When it is detected that the registration of the product to be settled in one transaction has been completed (S503: YES), the information processing device 10 transmits the settlement information for the transaction generated in S502 via the server device 40, It is transmitted to the settlement device 30 (S504). The information processing apparatus 10 identifies the settlement apparatus 30 to which the settlement information is to be transmitted, and transmits the settlement information, for example, according to the input operation of the operator of the information processing apparatus 10 . In this case, the information processing device 10 transmits the settlement information and information designating the settlement device 30 to which the settlement information is transmitted to the server device 40 . Then, the server device 40 transmits settlement information to the settlement device 30 designated by the information processing device 10, and the settlement device 30 uses the settlement information transmitted from the server device 40 to carry out settlement processing. (S505). In addition to this, the information processing device 10 transmits settlement information to the server device 40, for example, and includes information identifying the settlement information, and dispenses a portable medium readable by the settlement device 30 to the customer, The portable medium may be read into any settlement device 30 by the customer's operation. In this case, the settlement device 30 reads the settlement information from the server device 40 using the identification information contained in the portable medium, and performs settlement processing of the price (S505).

Although the embodiments of the present invention have been described above with reference to the drawings, these are examples of the present invention, and various configurations other than those described above can also be adopted.

For example, in the first embodiment, the imaging unit 124 acquires an image when the product is taken out from the place where the unregistered product is placed (for example, a basket) in order to hold the product before registration over the product recognition unit 120. However, the image recognition unit 110 may be configured to recognize the size or shape of the product from the image. In this case, it is possible to determine whether or not the unregistered product taken out from the placement location matches the product actually registered as the payment object. Further, for example, the imaging unit 124 may be configured to acquire an image when the registered product is placed on the placement location, and the image recognition unit 110 may be configured to recognize the size or shape of the product from the image. . In this case, it is possible to determine whether or not the registered product placed at the placement location matches the product actually registered for settlement. By doing so, it is possible to detect fraudulent acts such as replacement of products at the time of product registration, and to determine whether or not the recognized product is the correct product.

In the flowcharts or sequence diagrams used in the above description, a plurality of steps (processes) are described in order. is not limited to In each embodiment, the order of the illustrated steps can be changed within a range that does not interfere with the content. Moreover, each of the above-described embodiments can be combined as long as the contents do not contradict each other.

Examples of reference forms are added below.
1.
image recognition means for recognizing the size or shape of the product in the image using an image associated with depth information indicating the distance in the depth direction of the product included in the imaging range;
code acquisition means for acquiring a product identification code of a product to be settled;
a size/shape information acquisition means for acquiring size/shape information associated with the acquired product identification code;
determining means for determining whether or not the size or shape of the product recognized from the image and the size/shape information obtained using the product identification code satisfy a predetermined criterion;
A product registration device comprising:
2.
further comprising thermography information acquiring means for acquiring thermography information in association with the image;
The image recognition means is
Using the acquired thermography information, excluding a portion presumed to be a person from the shape of the product recognized from the image;
1. The product registration device described in .
3.
A plurality of pieces of size/shape information are associated with the product identification code,
The determining means determines whether or not the recognized size or shape and each of the plurality of size/shape information satisfy the predetermined criteria.
1. or 2. The product registration device described in .
4.
Further comprising imaging means for imaging the image according to the timing at which the code acquisition means reads the product identification code,
1. to 3. 3. The product registration device according to any one of .
5.
Further comprising a number acquisition means for acquiring the number of products included in the imaging range,
The determination means determines whether or not the number of the obtained product identification codes and the number of the obtained products match.
1. to 4. 3. The product registration device according to any one of .
6.
The product identification code is further associated with the weight or weight deviation of the product of the product identification code,
The determination means determines whether the weight associated with the obtained product identification code is equal to or greater than a reference value, when the weight associated with the obtained product identification code is not set, or when the obtained product identification code When the weight deviation associated with the product identification code obtained is greater than or equal to a predetermined threshold value, the determination is performed in at least one of the following cases;
1. to 5. 3. The product registration device according to any one of .
7.
the computer
recognizing the size or shape of the product in the image using an image associated with depth information indicating the distance in the depth direction of the product included in the imaging range;
Acquire the product identification code of the product for payment,
Acquiring size/shape information associated with the acquired product identification code,
calculating the difference between the size or shape of the product recognized from the image and the size/shape information obtained using the product identification code, and determining whether the difference satisfies a predetermined standard; judge,
control method including
8.
the computer
Acquiring thermography information in association with the image;
Using the acquired thermography information, excluding a portion presumed to be a person from the shape of the product recognized from the image;
7. including The control method described in .
9.
A plurality of pieces of size/shape information are associated with the product identification code,
the computer
Determining whether the recognized size or shape and each of the plurality of size/shape information meet the predetermined criteria;
7. including or 8. The control method described in .
10.
the computer
capturing the image according to the timing of reading the product identification code;
7. including to 9. Control method according to any one of.
11.
the computer
Acquiring the number of products included in the imaging range,
Determining whether the number of the acquired product identification codes and the number of the acquired products match;
7. including to 10. Control method according to any one of.
12.
The product identification code is further associated with the weight or weight deviation of the product of the product identification code,
the computer
When the weight associated with the obtained product identification code is equal to or greater than a reference value, When the weight associated with the obtained product identification code is not set, or When the obtained product identification code When the weight deviation associated with is equal to or greater than a predetermined threshold value, the determination is performed in at least one of the following cases;
7. including to 11. Control method according to any one of.
13.
the computer,
Image recognition means for recognizing the size or shape of the product in the image using an image associated with depth information indicating the distance in the depth direction of the product included in the imaging range;
code acquisition means for acquiring the product identification code of the product to be settled;
Size/shape information acquisition means for acquiring size/shape information associated with the acquired product identification code;
calculating the difference between the size or shape of the product recognized from the image and the size/shape information obtained using the product identification code, and determining whether the difference satisfies a predetermined standard; determination means for determining,
A program to function as
14.
said computer,
thermography information acquisition means for acquiring thermography information in association with the image;
The image recognition means,
means for excluding a part presumed to be a person from the shape of the product recognized from the image, using the acquired thermography information;
13. to further function as program described in .
15.
A plurality of pieces of size/shape information are associated with the product identification code,
said computer,
means for determining whether or not the recognized size or shape and each of the plurality of size/shape information satisfy the predetermined criteria;
13. to further function as or 14. program described in .
16.
said computer,
imaging means for capturing the image according to the timing at which the code acquisition means reads the product identification code;
13. to further function as to 15. A program according to any one of
17.
said computer,
number acquisition means for acquiring the number of products included in the imaging range;
means for determining whether or not the number of the obtained product identification codes matches the number of the obtained products;
13. to further function as to 16. A program according to any one of
18.
The product identification code is further associated with the weight or weight deviation of the product of the product identification code,
said computer,
the determination means, if the weight associated with the obtained product identification code is equal to or greater than a reference value, if the weight associated with the obtained product identification code is not set, or means for executing the determination in at least one of the cases where the weight deviation associated with the acquired product identification code is equal to or greater than a predetermined threshold;
13. for functioning as to 17. A program according to any one of

1 settlement system 10 information processing device 101 CPU
102 memory 103 storage 104 input/output interface 1041 display device 1042 input device 1043 imaging device 1044 reading device 105 communication module 110 image recognition unit 120 product recognition unit 122 code acquisition unit 124 imaging unit 126 thermography information acquisition unit 130 size/shape information acquisition Unit 132 Size/shape information storage unit 140 Judgment unit 142 Weight information storage unit 150 Number acquisition unit 170 Weight information acquisition unit 20 Basket 30 Checkout device 40 Server device

Claims (11)

a first acquisition means for acquiring product information related to products to be settled;
a second acquiring means for acquiring image information including the product;
determination means for determining whether or not there is fraud related to product registration based on the outputs of the first and second acquisition means;
with
The information processing apparatus, wherein the image information includes at least one of an image when the product is placed on the registered product placement location and an image of the product placed on the registered product placement location. Further comprising output means for outputting predetermined information based on the determination result of the determination means,
The information processing device according to claim 1 . The output means outputs a warning when the determination result indicates that it is invalid.
The information processing system according to claim 2. The judging means judges that there is fraud in product registration when the product whose product information is acquired by the first acquisition means and the product included in the image information acquired by the second acquisition means are different. ,
The information processing apparatus according to any one of claims 1 to 3. a first appearance information acquiring means for acquiring appearance information linked to the product information;
a second appearance information acquiring means for acquiring appearance information of the product using the image information;
The determination means uses the information acquired by the first appearance information acquisition means and the second appearance information acquisition means to determine whether or not there is fraud related to product registration.
The information processing apparatus according to any one of claims 1 to 4. Further comprising a number acquiring means for acquiring the number of products included in the image information,
The judging means judges whether or not there is fraud regarding product registration, using the number of products for which the first acquisition means has acquired the product information and the number of products acquired by the number acquisition means.
The information processing apparatus according to any one of claims 1 to 5. The number acquisition means acquires the number of products included in the image when the product is placed on the registered product placement location or the number of products included in the image of the product placed on the registered product placement location. do,
The information processing apparatus according to claim 6. The determination means is
A first determination is made to determine whether or not there is fraud in product registration using the number of products for which the first acquisition means has acquired the product information and the number of products acquired by the number acquisition means. ,
If it is determined in the first determination that there is no fraud related to product registration, it is further determined whether or not there is fraud related to product registration using the information acquired by the first and second acquisition means. making a second determination;
The information processing apparatus according to claim 6 or 7. further comprising a third acquisition means for acquiring weight information of the product placed on the placement place of the registered product;
The determination means uses the weight information to determine whether or not there is fraud related to product registration.
The information processing apparatus according to any one of claims 1 to 8. The information acquired by the first acquisition means is further associated with the weight or weight deviation of the product,
When the weight associated with the information acquired by the first acquisition means is equal to or less than a reference value, the determination means determines that the weight associated with the information acquired by the first acquisition means is not set. or when the weight deviation associated with the information acquired by the first acquisition means is equal to or greater than a predetermined threshold, it is determined whether or not there is fraud related to product registration. judge,
The information processing apparatus according to any one of claims 1 to 9. the computer
Acquire product information related to products for payment,
Acquiring image information including the product,
Determining whether there is fraud related to product registration based on the product information and the image information,
The information processing method, wherein the image information includes at least one of an image when the product is placed on the registered product placement location and an image of the product placed on the registered product placement location.

Images