JP2021128798A - Information processing device, system, image processing method and program - Google Patents

Abstract

To provide a device capable of extracting an image of a commodity from among images captured by imaging means for contribution to improvement in a recognition ratio of a commodity.SOLUTION: An information processing device includes: imaging means for generating an image of an object; distance measurement means for measuring a distant to the object; extraction means for extracting as an image of a commodity a region in which the measured distance is equal to or less than a predetermined value, from the generated images; and commodity recognition means for recognizing a commodity on the basis of the extracted image.SELECTED DRAWING: Figure 1

Description

The present invention relates to information processing devices, systems, image processing methods and programs.

At POS (Point Of Sales) terminals installed at payment locations (payment offices: cashiers) such as supermarkets and mass retailers, the clerk inputs the products with barcodes using the barcode input device, and the bar For products that cannot be coded, the clerk inputs the product data using the keyboard. Therefore, depending on the skill level of the clerk, there is a large difference in the input time of the product without the barcode. It is also practiced for a clerk to attach a barcode for a store to a product without a barcode in advance, which leads to an increase in working time. Furthermore, in recent years, self-checkouts in which customers directly operate POS terminal devices by themselves are increasing. Since it takes time for the customer to determine the position of the bar code on the product, the time required to operate the POS terminal device is further increased.

Therefore, a technique has been proposed in which a product is imaged by a camera or the like built in a POS terminal device, and the product is recognized by using an image recognition technique from the obtained image data. Here, when the product is photographed with the camera, not only the image of the product but also the background image is photographed. In the product recognition process, the background image becomes unnecessary noise. Therefore, it is necessary to remove the background image in order to perform the product recognition process. As a technique for removing a background image, Patent Document 1 describes a technique for removing a background image from an input image by taking a difference between a background image taken in advance and an input image in which the subject is taken. It is disclosed.

Japanese Unexamined Patent Publication No. 2001-52177

The technique of removing the background image (background image) by the difference between the background image taken in advance and the input image in which the subject is taken, as in Patent Document 1, has the following problems. When a product is photographed with the camera of the POS terminal device, an object other than the product may be reflected. For example, when taking a picture of a product with a camera, the body of the person holding the product may be reflected. In such a case, an object other than the product (for example, the body of a person holding the product) is also extracted as a difference from the background image. Therefore, the image of the product also includes an image of an object other than the product (for example, the body of a person holding the product). In addition, the background itself may be different from when the background image is taken in advance, for example, when the color of the background changes due to the influence of external light (for example, sunset). In such a case, since the background is recognized as a difference, the background image is not removed, and the image of the product includes an image of an object other than the product. In this way, when the image of the product includes an image of an object other than the product, the product is recognized with the image of the object other than the product included. In such a case, the accuracy of product recognition deteriorates, so that the product recognition rate may deteriorate.

The present invention has been made to solve such a problem, and in order to contribute to the improvement of the recognition rate of the product, the POS capable of extracting the image of the product from the image taken by the imaging means. The purpose of the present invention is to provide a terminal device, a POS system, an image processing method and a program.

The information processing apparatus according to the present invention includes an imaging means for generating an image of an object, a distance measuring means for measuring the distance to the object, and the measured distance of the generated images when the measured distance is a predetermined value or less. It has an extraction means for extracting a certain area as an image of a product, and a product recognition means for recognizing a product based on the extracted image.

Further, the system according to the present invention includes an information processing device and a management device that communicates with the information processing device.

Further, in the image processing method according to the present invention, an image of an object is generated, a distance to the object is measured, and a region of the generated image in which the measured distance is equal to or less than a predetermined value is used as a product. It is extracted as an image, and the product is recognized based on the extracted image.

Further, the program according to the present invention includes a step of generating an image of an object, a step of measuring a distance to the object, and a region of the generated image in which the measured distance is equal to or less than a predetermined value. A computer is made to execute a step of extracting as an image of a product and a step of recognizing a product based on the extracted image.

According to the present invention, in order to contribute to the improvement of the recognition rate of a product, a POS terminal device, a POS system, an image processing method and a program capable of extracting an image of the product from an image taken by an imaging means are provided. can.

It is a figure which shows the outline of the POS terminal apparatus which concerns on embodiment of this invention. It is a side view which shows the appearance of the POS terminal apparatus which concerns on Embodiment 1. FIG. It is a figure which shows the hardware composition of the POS terminal apparatus which concerns on Embodiment 1. FIG. It is a functional block diagram of the POS terminal apparatus which concerns on Embodiment 1. FIG. It is a flowchart which shows the process of the start control part which concerns on Embodiment 1. FIG. It is a flowchart which shows the processing of the recognition processing part which concerns on Embodiment 1. It is a figure for demonstrating the process of the product image extraction part which concerns on Embodiment 1. FIG. It is a figure which illustrates the product which has the same shape and package but different size. It is a figure which shows the POS system which concerns on Embodiment 2. It is a figure which shows the hardware configuration of the management apparatus which concerns on Embodiment 2. FIG. It is a functional block diagram of the POS terminal apparatus which concerns on Embodiment 2. FIG. It is a functional block diagram of the management apparatus which concerns on Embodiment 2. FIG.

(Overview of Embodiments of the present invention)
Prior to the description of the embodiment, the outline of the embodiment according to the present invention will be described. FIG. 1 is a diagram showing an outline of a POS terminal device 1 according to an embodiment of the present invention. As shown in FIG. 1, the POS terminal device 1 includes an image pickup unit 2, a distance measurement unit 4, and a product image extraction unit 6.

The image pickup unit 2 takes an image of an object and generates an image. The distance measuring unit 4 measures the distance to the position of the object corresponding to the image generated by the imaging unit 2. The product image extraction unit 6 uses an image area corresponding to a position where the distance measured by the distance measurement unit 4 is within a predetermined first range from the image generated by the image pickup unit 2 as an image of the product. Extract. The POS terminal device 1 according to the embodiment of the present invention can extract an image of a product from an image taken by the imaging unit 2 in order to contribute to an improvement in the recognition rate of the product.

(Embodiment 1)
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 2 is a side view showing the appearance of the POS terminal device 100 according to the first embodiment. Further, FIG. 3 is a diagram showing a hardware configuration of the POS terminal device 100 according to the first embodiment. The POS terminal device 100 includes a store clerk display operation unit 102, a customer display unit 104, an information processing device 110, and a product reading device 120. The POS terminal device 100 is placed on, for example, a counter stand (not shown), and the customer faces the left side of FIG. 2 and the clerk faces the right side of the POS terminal device 100.

The display operation unit 102 for a clerk is, for example, a touch panel, an LCD (Liquid Crystal Display), a keyboard, or the like. The store clerk display operation unit 102 displays necessary information for the store clerk under the control of the information processing device 110, and accepts the store clerk's operation.

The customer display unit 104 is, for example, a touch panel or an LCD. The customer display unit 104 displays information necessary for the customer under the control of the information processing device 110. Further, the customer display unit 104 may have an input device, and may accept customer operations as needed.

The information processing device 110 is, for example, a computer. The information processing device 110 includes, for example, a control unit 112 such as a CPU (Central Processing Unit), a storage unit 114 such as a memory or a hard disk, and a communication device 116. The information processing device 110 controls the operations of the store clerk display operation unit 102, the customer display unit 104, and the product reading device 120. In addition, the information processing device 110 performs necessary processing according to the operation received by the store clerk display operation unit 102. Further, the information processing device 110 performs necessary processing such as image processing according to the image information read by the product reading device 120. The communication device 116 performs processing necessary for communicating with a management device such as a server connected via a network.

The product reading device 120 has a housing 122, a product reading surface 124 made of a light-transmitting material, and a three-dimensional camera 130. The product reading surface 124 is provided on the surface of the housing 122 on the clerk side, and the product is directed when the product is imaged (read). The three-dimensional camera 130 is provided inside the housing 122 on the side opposite to the product reading surface 124. When the clerk points the product received from the customer toward the product reading surface 124, the three-dimensional camera 130 reads the image of the product. As a result, the POS terminal device 100 performs the product recognition process. Details will be described later.

The three-dimensional camera 130 has an imaging unit 132 and a distance sensor 134 (distance measuring means). The image pickup unit 132 is, for example, an image pickup device (camera) such as a CCD (Charge-Coupled Device), and performs a process of reading an image of a product. Specifically, the imaging unit 132 images an object directed to the product reading surface 124 and generates a two-dimensional color image or a monochrome image (two-dimensional image) including an image of the object. Hereinafter, the term "two-dimensional image" also means "image data indicating a two-dimensional image" as a processing target in information processing.

The distance sensor 134 measures the distance from the distance sensor 134 to the position of the object directed to the product reading surface 124 by, for example, a TOF (Time Of Flight) method. That is, the distance sensor 134 irradiates a light ray such as infrared rays, and measures the distance from the time required for the irradiated light ray to reciprocate to the object. In the present embodiment, infrared rays are irradiated, but the present invention is not limited to this. For example, the distance sensor 134 may irradiate a laser.

The distance sensor 134 includes an infrared irradiation unit 136, an infrared light receiving unit 138, and a distance image generation unit 140. The infrared irradiation unit 136 irradiates infrared rays toward the outside of the product reading surface 124. When an object exists in the vicinity of the product reading surface 124, the infrared rays emitted by the infrared irradiation unit 136 reflect the object. The infrared light receiving unit 138 is, for example, an image sensor (for example, a CCD or CMOS (Complementary Metal-Oxide Semiconductor)) and receives infrared rays reflected from an object.

The distance image generation unit 140 calculates the distance (depth) to the object from the time difference between the time when the infrared irradiation unit 136 irradiates infrared rays and the time when the infrared light receiving unit 138 receives the reflected light. Specifically, the infrared irradiation unit 136 and the infrared light receiving unit 138 are synchronized with each other, and the distance image generation unit 140 receives the phase of the infrared rays emitted by the infrared irradiation unit 136 and the infrared light receiving unit 138. The time difference is measured by measuring the phase difference with the phase of the reflected light.

More specifically, the distance image generation unit 140 measures the time difference for each reflected light from each position of the object received by the infrared light receiving unit 138, and measures the distance for each position. As a result, the distance image generation unit 140 generates a distance image (three-dimensional image) which is a set of pixels indicating the distance at each position of the object. Hereinafter, the term "distance image" also means "image data indicating a distance image" as a processing target in information processing.

For example, for a position 15 cm from the distance sensor 134, the pixel corresponding to the position in the distance image includes distance information indicating "distance 15 cm". Further, with respect to the position 30 cm from the distance sensor 134, the pixel corresponding to the position in the distance image includes the distance information indicating "distance 30 cm".

If the distance between the distance sensor 134 and the object is too close, the phase difference between the irradiation light and the reflected light cannot be detected, and the time difference cannot be measured. Therefore, the distance to the object cannot be measured. Therefore, the distance sensor 134 (three-dimensional camera 130) keeps the distance between the distance sensor 134 (three-dimensional camera 130) and the object according to the minimum time difference (phase difference) that the distance sensor 134 can measure. , It is installed at a distance L from the product reading surface 124. That is, the distance sensor 134 (three-dimensional camera 130) is installed at a distance of L from the product reading surface 124 so as to maintain the minimum measurable distance. The speed of infrared rays (speed of light) is 300,000 km / s, so infrared rays travel 30 cm in 1 nanosecond. That is, when the distance between the distance sensor 134 and the object changes by 15 cm (reciprocating 30 cm), the time from irradiating the light to receiving the reflected light changes by 1 nanosecond. Therefore, for example, when the minimum measurable time difference is 1 nanosecond, the interval L is 15 cm.

Further, the image pickup unit 132 and the distance sensor 134 are arranged close to each other (that is, at substantially the same position). Therefore, the two-dimensional image obtained by the imaging unit 132 and the distance image obtained by the distance sensor 134 are configured to correspond to each other. That is, when the position P of the object to be imaged corresponds to the pixel position (X1, Y1) of the two-dimensional image, the position P substantially corresponds to the pixel position (X1, Y1) also in the distance image. In other words, each pixel position of the distance image obtained by the distance sensor 134 is configured to correspond to each pixel position of the two-dimensional image obtained by the imaging unit 132. If the position of the image pickup unit 132 and the position of the distance sensor 134 are slightly different from each other, the distance between the image pickup unit 132 and the distance sensor 134 and the visual fields of the image pickup unit 132 and the distance sensor 134, respectively. From the corners, processing may be performed such that the positions of each pixel of the distance image and the positions of each pixel of the two-dimensional image are aligned.

FIG. 4 is a functional block diagram of the POS terminal device 100 according to the first embodiment. 5 and 6 are flowcharts showing the processing of the POS terminal device 100 according to the first embodiment. The POS terminal device 100 has a start control unit 200 and a recognition processing unit 220. The start control unit 200 includes a distance image acquisition unit 202, an object approach determination unit 204, and a recognition process execution control unit 206. Further, the recognition processing unit 220 includes a product image shooting control unit 222, a distance image shooting control unit 224, a product image extraction unit 226, a product recognition processing unit 228, and a product information storage unit 230.

The start control unit 200 and the recognition processing unit 220 can be realized by, for example, executing a program under the control of the control unit 112. More specifically, the program stored in the storage unit 114 is realized by executing the program under the control of the control unit 112. Further, each component is not limited to being realized by software by a program, and may be realized by a combination of hardware, firmware, software, or the like. Further, each component of the start control unit 200 and the recognition processing unit 220 may be realized by using an integrated circuit that can be programmed by the user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of each of the above components.

The start control unit 200 determines whether or not the object has approached the product reading surface 124 (that is, the three-dimensional camera 130), and controls whether or not the recognition processing unit 220 executes the process. The recognition processing unit 220 performs a product recognition process (product recognition process) when the start control unit 200 controls the process to be executed.

Hereinafter, the processing of the start control unit 200 and the recognition processing unit 220 will be specifically described. FIG. 5 is a flowchart showing the processing of the start control unit 200 according to the first embodiment. Further, FIG. 6 is a flowchart showing the processing of the recognition processing unit 220 according to the first embodiment.

The distance image acquisition unit 202 controls the distance sensor 134 to acquire a distance image (S102). Specifically, the distance image acquisition unit 202 controls the distance sensor 134 so as to capture a distance image in front of the product reading surface 124. The distance sensor 134 captures a distance image in front of the product reading surface 124 under the control of the distance image acquisition unit 202, and generates a distance image (image data indicating the distance image). The distance image acquisition unit 202 acquires the generated distance image and outputs it to the object approach determination unit 204.

The object approach determination unit 204 determines whether or not the object has approached within the threshold value Th1 (first threshold value) by using the distance image (S104). For example, the object approach determination unit 204 analyzes the distance image and determines whether or not there is a pixel indicating a distance within the threshold Th1 from the distance sensor 134. The object approach determination unit 204 determines that the objects are approaching when there are pixels indicating a distance within the threshold Th1. On the other hand, the object approach determination unit 204 determines that the object is not approaching when there is no pixel indicating the distance within the threshold Th1.

The threshold value Th1 is determined in consideration of the distance from the product reading surface 124 to the product (object) when the store clerk or the like points the product at the product reading surface 124 to recognize the product. Further, the threshold value Th1 is determined between the product reading surface 124 and the position of the threshold value Th1 so that no object exists except when the store clerk or the like points the product toward the product reading surface 124. For example, assuming that the distance (Th1-L) from the product reading surface 124 to the position of the threshold Th1 is 25 cm, the threshold Th1 is 40 cm when the interval L is 15 cm.

When the object approach determination unit 204 determines that the object is approaching within the threshold Th1 (YES in S104), the recognition process execution control unit 206 causes the recognition process unit 220 to start the product recognition process. Control (S106).

On the other hand, when it is determined by the object approach determination unit 204 that the object is not approaching within the threshold Th1 (NO in S104), the recognition process execution control unit 206 determines whether the recognition process 220 is performing the product recognition process. Is determined (S108). When the recognition processing unit 220 has not performed the product recognition processing (NO in S108), the processing returns to S102.

On the other hand, when the recognition processing unit 220 is performing the product recognition processing (YES in S108), the recognition processing execution control unit 206 controls the recognition processing unit 220 to end the product recognition process (S110). That is, the processing of the start control unit 200 may always be performed while the POS terminal device 100 is running. Even if the product recognition process by the recognition processing unit 220 is started because the object (product) has approached the product reading surface 124 once, when the recognition process is completed or while the recognition process is being performed. When the object (product) is separated from the product reading surface 124 (the distance from the distance sensor 134 to the object exceeds the threshold Th1), the start control unit 200 ends the product recognition process for the recognition processing unit 220. Control to do.

As described above, the POS terminal device 100 according to the present embodiment performs the product recognition process only when the object (product) approaches the product reading surface 124 (distance sensor 134). When the product recognition process is performed, the load on the POS terminal device 100 (particularly, the imaging unit 132, the control unit 112, and the storage unit 114) increases. Therefore, with such a configuration, it is possible to reduce the resource load of the POS terminal device 100 when it is not necessary to perform the product recognition process. The "resource" here includes not only the hardware resource of the POS terminal device 100 itself but also the network resource.

The product image shooting control unit 222 causes the imaging unit 132 to shoot a two-dimensional image including the product image (product image) (S202). Specifically, the product image capturing control unit 222 controls the image pickup unit 132 to image the product directed to the product reading surface 124. Then, the product image capturing control unit 222 acquires the two-dimensional image generated by the imaging unit 132 and outputs the two-dimensional image to the product image extraction unit 226. The two-dimensional image may include a background image (background image) in addition to the product image.

The distance image capturing control unit 224 causes the distance sensor 134 to capture a distance image corresponding to the two-dimensional image (S204). Specifically, the distance image capturing control unit 224 controls the distance sensor 134 to capture a distance image including the product directed at the product reading surface 124. Then, the distance image capturing control unit 224 acquires the distance image generated by the distance sensor 134 and outputs it to the product image extraction unit 226. As described above, since the imaging unit 132 and the distance sensor 134 are arranged close to each other, the distance image corresponds to the two-dimensional image. The process (S204) of the distance image capturing control unit 224 may be replaced by the process of S102. That is, the distance image acquired by the distance image acquisition unit 202 may be output to the product image extraction unit 226.

The product image extraction unit 226 determines an area in which the distance from the distance sensor 134 is within the range S1 (first range) in the distance image, and extracts an image area corresponding to the area from the two-dimensional image as a product image. (S206). Further, the product image extraction unit 226 outputs the extracted product image to the product recognition processing unit 228.

Specifically, the product image extraction unit 226 superimposes the two-dimensional image and the distance image. In addition, the product image extraction unit 226 determines which area in the two-dimensional image corresponds to the area in which the distance is within the range S1 in the distance image. Further, the product image extraction unit 226 determines that the corresponding area is an image area corresponding to the product image. Then, the product image extraction unit 226 extracts the image area as a product image.

More specifically, the product image extraction unit 226 extracts pixels (pixels within the range) indicating the distance within the range S1 from the distance image. In addition, the product image extraction unit 226 determines which pixel of the two-dimensional image each of the extracted pixels in the range corresponds to. Further, the product image extraction unit 226 extracts pixels corresponding to the pixels within the range from the two-dimensional image. The product image extraction unit 226 extracts a set of pixels extracted from the two-dimensional image as an image area corresponding to the product image.

The lower limit of the range S1 may be, for example, the distance (interval L) from the distance sensor 134 to the product reading surface 124. Further, the upper limit of the range S1 is set in consideration of how close the product is to the product reading surface 124 when the store clerk or the like directs the product toward the product reading surface 124 in order to read the product. For example, the upper limit of the range S1 may be set to be smaller than the threshold value Th1. For example, assuming that the distance from the product reading surface 124 to the upper limit position of the range S1 is 15 cm, when the interval L is 15 cm, the range S1 is 15 cm <S1 <30 cm. The lower limit of the range S1 does not have to be set, and in that case, the lower limit is 0 cm.

FIG. 7 is a diagram for explaining the processing of the product image extraction unit 226. FIG. 7A is a diagram illustrating a two-dimensional image Im2 including a product image taken by the imaging unit 132 under the control of the product image shooting control unit 222. The two-dimensional image Im2 includes a product image A (indicated by a solid line) and a background image B (indicated by an alternate long and short dash line). The product corresponding to the product image A is at a position where the distance from the distance sensor 134 (product reading surface 124) is within the range S1. The object corresponding to the background image B is located at a position far enough away from the distance sensor 134 (product reading surface 124) so as to deviate from the range S1.

FIG. 7B is a diagram illustrating a distance image Im3 corresponding to a two-dimensional image captured by the distance sensor 134 under the control of the distance image capturing control unit 224. Here, the area indicated by C indicates an area in which the distance from the distance sensor 134 is within the range S1. The shaded area indicated by D indicates an area where the distance from the distance sensor 134 deviates from the range S1.

The product image extraction unit 226 superimposes the two-dimensional image Im2 and the distance image Im3. Then, the product image extraction unit 226 extracts the image region corresponding to the area indicated by C of the distance image Im3 from the two-dimensional image Im2. As a result, the product image E as illustrated in FIG. 7C is extracted. Here, the product image E does not include the background image. That is, the product image extraction unit 226 removes the background image B from the two-dimensional image Im2.

The product recognition processing unit 228 (FIG. 4) performs product recognition processing using the product image extracted by the product image extraction unit 226 (S208). In addition, the product recognition processing unit 228 stores the product information obtained by the product recognition processing in the product information storage unit 230. The product information includes the information included in the barcode attached to the product. Further, the product information may include the size (capacity) of the product. The POS terminal device 100 uses the product information obtained by the product recognition process by the product recognition processing unit 228 to perform payment processing and the like for the product.

Regarding the product recognition process, specifically, for example, the product recognition processing unit 228 stores the product name and the information related to the product (reference product information) in advance in association with each other. The product recognition processing unit 228 performs pattern matching between the extracted product image and the reference product information stored in advance. The standard product information is illustrated below.

For example, the reference product information may be an image (reference product image) that serves as a reference for the product. In that case, the product recognition processing unit 228 collates the extracted product image with the reference product image. Then, the product recognition processing unit 228 associates the product with the product name corresponding to the reference product image when the similarity between the two satisfies the permissible value.

Further, for example, the reference product information may be data (product feature data) indicating a feature that serves as a reference for the product. The product feature data includes, for example, information indicating the shape of the product, information indicating the color of the product, information indicating the texture (gloss, etc.) of the product, and character information and patterns attached to the package of the product. May contain at least one of. In this case, the product recognition processing unit 228 extracts the features of the image from the extracted product image. Then, the product recognition processing unit 228 collates the features of the extracted image with the product feature data. Then, the product recognition processing unit 228 associates the product with the product name corresponding to the product feature data when the similarity between the two satisfies the permissible value. In addition, the product recognition processing unit 228 may recognize the product name by reading the character information attached to the product package with an OCR (Optical Character Reader).

Here, the background of the product image extracted by the product image extraction unit 226 is removed. Therefore, when the product recognition processing unit 228 performs the product recognition processing, it is not necessary to exclude the background. When the two-dimensional image includes not only the product image but also the background image, in the product recognition process, it is first necessary to recognize where the product image is in the two-dimensional image. In particular, when an unspecified number of customers use the POS terminal device 100, such as a self-checkout, the position of the product reading surface 124 at which the product is directed differs depending on the customer. In this process of recognizing where the product image is, for example, the reference product information must be collated with respect to all the images included in the two-dimensional image. Therefore, the processing time becomes enormous.

On the other hand, in the present embodiment, since the product image itself is used, it is not necessary to recognize where the product image is in the two-dimensional image. Therefore, the POS terminal device 100 according to the present embodiment can improve the processing speed of the product recognition process. In other words, the POS terminal device 100 according to the present embodiment can reduce the load of resources in the product recognition process. Further, the amount of data of the product image is smaller than the amount of data of the two-dimensional image because the background is removed. Therefore, since the amount of data to be processed can be reduced, it is possible to reduce resources and load. Therefore, a resource-poor device such as a tablet terminal can be used as the POS terminal device 100 according to the present embodiment. The "resource" here includes not only the hardware resource of the POS terminal device 100 itself but also the network resource. That is, in the present embodiment, it is also possible to reduce the network load.

Further, if the product image includes a background image, the background image is taken into consideration in the product recognition process. Therefore, the recognition rate deteriorates. On the other hand, since the background of the product image extracted by the product image extraction unit 226 is removed, the recognition rate can be improved.

In addition, the two-dimensional image including the product image taken by the imaging unit 132 may include an image of the body of a clerk or the like holding the product. Here, in the method of using the difference from the background image taken in advance when extracting the product image, the body of the clerk or the like is also recognized as the difference. Therefore, the extracted product image also includes the image of the body of the clerk or the like, and the image of the body of the clerk or the like becomes noise and the recognition rate is lowered.

Here, when a person points the product at the product reading surface 124, he / she usually reaches out and points the product. Therefore, the body of the clerk or the like is usually away from the product reading surface 124 (distance sensor 134). Therefore, in the present embodiment, the product image extraction unit 226 can remove the image of the body of the clerk or the like. Therefore, the product recognition processing unit 228 can perform the product recognition process only with the image of the product without considering the image of the body of the clerk or the like. Therefore, the POS terminal device 100 according to the present embodiment can further improve the recognition rate.

In addition, in the method using the difference from the background image taken in advance, the product image is extracted even if the color of the background is different from the background image taken in advance due to the influence of external light (for example, sunset). At that time, this background is also recognized as a difference. Therefore, the background image is also included in the product image, and the background image becomes noise and the recognition rate is lowered.

Here, the object corresponding to the background is separated from the product reading surface 124 (distance sensor 134). Therefore, in the present embodiment, the product image extraction unit 226 can reliably remove the background regardless of the change in the background color. Therefore, the POS terminal device 100 according to the present embodiment can further improve the recognition rate.

In addition, some products have the same shape and package but different sizes. For example, as shown in FIG. 8, a plurality of types of PET bottled beverages having the same contents but different sizes (capacities) are sold. Such products generally vary in price depending on their size. In such a case, the size of the product cannot be recognized simply by performing the product recognition process using the product image. Therefore, in order to settle at an appropriate price, it is necessary for a clerk or the like to manually input the price or capacity.

On the other hand, the POS terminal device 100 according to the present embodiment can measure the distance to the product by the distance sensor 134 as described above. Even if the actual product dimensions are the same, the dimensions of the product image in the two-dimensional image decrease as the distance (depth) increases, and increase as the distance (depth) decreases. That is, the actual size of the product can be geometrically grasped from the dimensions of the product image in the two-dimensional image and the distance to the product.

Therefore, the product recognition processing unit 228 may recognize the size of the product by acquiring information indicating the distance to the product from the distance image shooting control unit 224 and measuring the dimensions of the product image. Specifically, the product recognition processing unit 228 acquires a distance image from the distance image capturing control unit 224. In addition, the product recognition processing unit 228 uses the distance image to acquire distance information for each pixel in the area of the product image extracted by the product image extraction unit 226. Further, the product recognition processing unit 228 calculates the distance to the product from the acquired distance information of each pixel. As the calculation method, for example, the distance indicated by the pixels corresponding to the edges of the product image may be the distance to the product, or the average of the distances indicated by each pixel in the area of the product image may be the distance to the product. ..

Further, the product recognition processing unit 228 measures the size of the product image in the two-dimensional image. For the size of the product image, for example, the vertical dimension and the horizontal dimension are measured. Then, the product recognition processing unit 228 calculates the actual product dimensions from the size of the product image and the distance to the product. Here, the reference product information that serves as the reference for the product recognition process may include the dimensions and capacity of the product. Therefore, the product recognition processing unit 228 can grasp the name and capacity of the product (in the example of FIG. 8, "capacity of product name ABC 500 ml").

(Embodiment 2)
Next, the second embodiment will be described. The second embodiment is different from the first embodiment in that the POS terminal device 100 does not perform the product recognition process, as will be described later. The same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 9 is a diagram showing a POS system 300 according to the second embodiment. As shown in FIG. 9, the POS system 300 includes a POS terminal device 100 and a management device 310. The POS terminal device 100 and the management device 310 are communicably connected to each other. The communication between the two may be either wired communication or wireless communication, and various communication standards may be applied. The POS terminal device 100 and the management device 310 may be connected to each other via a network (for example, a wireless LAN (Local Area Network) or the Internet). Further, the POS terminal device 100 and the management device 310 may communicate with each other by infrared communication or a short-range wireless communication method such as Bluetooth (registered trademark).

The POS terminal device 100 according to the second embodiment has substantially the same hardware configuration as the POS terminal device 100 according to the first embodiment. The POS terminal device 100 communicates with the management device 310 by using the communication device 116. In this case, the communication device 116 performs the processing necessary for communicating with the management device 310.

The management device 310 is an information processing device that manages product information and the like. The management device 310 may be arranged in the store where the POS terminal device 100 is arranged. Further, the management device 310 may collectively manage each POS terminal device 100 arranged in a plurality of stores. In this case, the management device 310 is different from the store in which the POS terminal device 100 is arranged. Can be placed in place. Further, the management device 310 may be, for example, a server or a cloud server.

FIG. 10 is a diagram showing a hardware configuration of the management device 310 according to the second embodiment. The management device 310 includes, for example, a control unit 312 such as a CPU, an input / output unit 314 which is a user interface such as a touch panel, an LCD or a keyboard, a storage unit 316 such as a memory or a hard disk, and a communication device 318. The communication device 318 performs processing necessary for communicating with the POS terminal device 100 (or another management device 310).

FIG. 11 is a functional block diagram of the POS terminal device 100 according to the second embodiment. The POS terminal device 100 has a start control unit 200 and a recognition processing unit 240. The start control unit 200 includes a distance image acquisition unit 202, an object approach determination unit 204, and a recognition process execution control unit 206. Further, the recognition processing unit 240 includes a product image shooting control unit 222, a distance image shooting control unit 224, a product image extraction unit 226, and a product image transmission unit 248. As described above, the start control unit 200 and the recognition processing unit 240 can be realized by, for example, executing a program under the control of the control unit 112.

The recognition processing unit 240 according to the second embodiment is different from the recognition processing unit 220 according to the first embodiment in that it does not have the product recognition processing unit 228 and the product information storage unit 230 and has the product image transmission unit 248. .. The product image extraction unit 226 outputs the extracted product image to the product image transmission unit 248. The product image transmission unit 248 transmits the product image (image data of the product image) to the management device 310. When transmitting the product image, the product image transmission unit 248 may also transmit the current time, the identification information of the POS terminal device 100, and the like to the management device 310. Further, the product image transmission unit 248 may also transmit a distance image (image data of the distance image) to the management device 310 so that the size of the product can be calculated by the management device 310.

FIG. 12 is a functional block diagram of the management device 310 according to the second embodiment. The management device 310 has a recognition processing unit 320. Further, the recognition processing unit 320 has a product image receiving unit 322, a product recognition processing unit 328, and a product information storage unit 330.

The recognition processing unit 320 can be realized, for example, by executing a program under the control of the control unit 312. More specifically, the program stored in the storage unit 316 is realized by executing the program under the control of the control unit 312. Further, each component is not limited to being realized by software by a program, and may be realized by a combination of hardware, firmware, software, or the like. Further, each component of the recognition processing unit 320 may be realized by using an integrated circuit programmable by the user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of each of the above components.

The product image receiving unit 322 receives the product image (product image data) or the like transmitted by the POS terminal device 100 and outputs it to the product recognition processing unit 328. The product recognition processing unit 328 has substantially the same function as the product recognition processing unit 228 according to the first embodiment. Therefore, the product recognition processing unit 328 uses the product image extracted by the product image extraction unit 226 to perform the product recognition processing in the same manner as in the first embodiment described above. Further, the product recognition processing unit 328 stores the product information obtained by the product recognition processing in the product information storage unit 330. Further, the management device 310 transmits the obtained product information to the POS terminal device 100. The POS terminal device 100 uses the product information received from the management device 310 to perform payment processing and the like for the product.

By performing the product recognition process not in the POS terminal device 100 but in the management device 310 as in the second embodiment, it is not necessary for each POS terminal device 100 to store the reference product information required for the product recognition process. , The POS terminal device 100 does not need to perform the product recognition process. Therefore, it is possible to save the resources of the POS terminal device 100. Further, the present embodiment can be applied to the POS terminal device 100, which has scarce resources such as a tablet terminal. Further, also in the second embodiment, the product image is extracted by the product image extraction unit 226. Therefore, as in the first embodiment, in the product recognition process by the management device 310, the resource load is reduced, the processing speed is improved, the recognition rate is improved, and the size (capacity) of the product is determined. It becomes possible to grasp.

Further, as described above, in the product image extracted by the product image extraction unit 226, the background image is removed from the two-dimensional image obtained by the photographing of the image pickup unit 132. Therefore, the amount of data of the product image is smaller than the amount of data of the two-dimensional image. When the product recognition process is performed by the management device 310, if the POS terminal device 100 transmits image data of a two-dimensional image to the management device 310, the amount of data is large, so that the load on the communication network increases. On the other hand, when the POS terminal device 100 transmits the image data of the product image to the management device 310, the load on the communication network is reduced because the amount of data is small.

(Modification example)
The present invention is not limited to the above embodiment, and can be appropriately modified without departing from the spirit. For example, the order of processing in the above-mentioned flowchart can be changed as appropriate. Further, at least one of the plurality of processes in the above-mentioned flowchart may be omitted. For example, in the flowchart of FIG. 6, the process of S202 may be performed after the process of S204. Further, in the flowchart of FIG. 5, the processing of S108 and S110 may not be necessary.

Further, the configuration according to the present embodiment is applied to the POS terminal device, but the present invention is not limited to this. For example, it can be applied to a general object recognition device such as an object recognition device used for sorting luggage in a warehouse or the like, and a system including this object recognition device.

Further, the POS terminal device 100 according to the present embodiment can be applied to, for example, self-registration. When a customer uses a POS terminal as in self-checkout, the customer is not accustomed to having a reader read the barcode attached to the product. Therefore, in self-checkout, a method that does not use a barcode, that is, a method that allows the product to be read directly is required. Therefore, by applying the POS terminal device 100 according to the present embodiment to the self-registration, the problem caused by directly reading the product as described above is solved.

Further, as described above, the POS terminal device 100 according to the present embodiment can be applied to a terminal having scarce resources such as a tablet terminal (tablet POS). In this case, the three-dimensional camera 130 does not have to be built in the tablet terminal, and may be a device separate (external) from the tablet terminal.

Further, in the above-described embodiment, the start control unit 200 controls the start or end of the product recognition process, but the present invention is not limited to this. That is, the start control unit 200 is not an indispensable component, and the processing of the start control unit 200 shown in FIG. 5 is not always necessary. However, as described above, it is possible to reduce resources by performing the processing of the start control unit 200. In addition, instead of the start control unit 200, a store clerk or the like may control the start or end of the product recognition process by operating a switch for starting or ending the product recognition process. However, by processing the start control unit 200, it is not necessary for the store clerk or the like to operate the switch.

Further, the configuration of the first embodiment and the configuration of the second embodiment may be combined. For example, the product recognition process may also be performed in the POS terminal device 100 according to the second embodiment. In other words, the POS terminal device 100 according to the second embodiment may have a product recognition processing unit 228. In this case, when the load of the POS terminal device 100 increases from the predetermined first load value, the POS terminal device 100 transmits the product image to the management device 310, and the management device 310 performs the product recognition process. You may do so. On the other hand, when the load of the management device 310 is higher than the predetermined second load value, or when the load of the communication network is higher than the predetermined third load value, the POS The terminal device 100 may perform the product recognition process on its own POS terminal device 100 without transmitting the product image to the management device 310.

In this way, it is possible to appropriately distribute the load according to the load of the POS terminal device 100, the load of the management device 310, and the load of the communication network. In this case, the POS terminal device 100 or the management device 310 measures the load of the POS terminal device 100, the load of the management device 310, and the load of the communication network, and the measured load and the first to third load values. You may have a means to compare with each other.

Programs can also be stored and supplied to a computer using various types of non-transitory computer readable medium. Non-temporary computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs. It includes a CD-R / W and a semiconductor memory (for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (random access memory)). The program may also be supplied to the computer by various types of transient computer readable medium. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

1 POS terminal device 2 Imaging unit 4 Distance measurement unit 6 Product image extraction unit 100 POS terminal device 110 Information processing device 120 Product reading device 124 Product reading surface 130 Three-dimensional camera 132 Imaging unit 134 Distance sensor 136 Infrared irradiation unit 138 Infrared light receiving unit 140 Distance image generation unit 200 Start control unit 202 Distance image acquisition unit 204 Object approach discrimination unit 206 Recognition processing execution control unit 220 Recognition processing unit 222 Product image shooting control unit 224 Distance image shooting control unit 226 Product image extraction unit 228 Product recognition processing Unit 230 Product information storage unit 240 Recognition processing unit 248 Product image transmission unit 300 POS system 310 Management device 312 Control unit 320 Recognition processing unit 322 Product image receiving unit 328 Product recognition processing unit 330 Product information storage unit

Claims (10)

An imaging means that generates an image of an object,
A distance measuring means for measuring the distance to the object and
An extraction means for extracting a region of the generated image in which the measured distance is equal to or less than a predetermined value as an image of a product,
A product recognition means that recognizes a product based on the extracted image, and
Information processing device with. When the measured distance becomes equal to or less than a predetermined threshold value, the product recognition means recognizes the product.
The information processing device according to claim 1. Further, it has a discriminating means for discriminating that an object has approached so that the measured distance is equal to or less than the threshold value.
The information processing apparatus according to claim 2, wherein the extraction means executes the extraction process when it is determined by the determination means that the object has approached. The product recognition means
The size of the image of the extracted product in the image is measured, and
The size of the product is recognized based on the size of the measured image of the product and the distance to the measured object.
The information processing device according to claim 3, wherein the product is recognized based on the size of the recognized product. Any one of claims 1 to 4, further comprising a transmission means for transmitting data indicating the image of the extracted product to a management device that performs recognition processing of the product based on the image of the extracted product. The information processing device described in the section. The information processing device according to any one of claims 1 to 5.
A system having a management device that communicates with the information processing device. The information processing device and the management device are connected via a communication network.
When the load of the information processing device increases from a predetermined first load value, the information processing device transmits data indicating an image of the extracted product to the management device. The management device performs recognition processing of the product.
The system according to claim 6. The information when the load of the management device is higher than a predetermined second load value, or when the load of the communication network is higher than a predetermined third load value. The system according to claim 7, wherein the processing device does not transmit data indicating an image of the extracted product to the management device, and the product recognition means performs the product recognition process. Generate an image of an object,
Measure the distance to the object and
From the generated image, a region where the measured distance is equal to or less than a predetermined value is extracted as an image of the product.
The product is recognized based on the extracted image.
Image processing method. Steps to generate an image of an object,
The step of measuring the distance to the object and
A step of extracting a region of the generated image in which the measured distance is equal to or less than a predetermined value as an image of a product, and
The step of recognizing the product based on the extracted image and
A program that causes a computer to run.

Images