JP6318557B2 - POS terminal device, POS system, image processing method and program - Google Patents

Description

  The present invention relates to a POS (Point Of Sales) terminal device, a POS system, a product extraction method, and a program, and more particularly to a POS terminal device, a POS system, an image processing method, and a program that are used for settlement of products.

  In a POS (Point Of Sales) terminal installed at a payment place (charge payment place: cash register) such as a supermarket or a mass retailer, the store clerk inputs products with barcodes using a barcode input device. For products that cannot be coded, the store clerk inputs product data using the keyboard. For this reason, a big difference arises in the input time of the goods to which the barcode is not attached depending on the skill level of the store clerk. Although a store clerk has added a bar code for a store in advance to a product without a bar code, it leads to an increase in work time. Furthermore, in recent years, self-checkout, in which customers directly operate POS terminal devices themselves, is increasing. Since it takes time for the customer to determine at which position of the product the barcode is attached, the time required for operating the POS terminal device further increases.

  For this reason, a technology has been proposed in which a product is imaged with a camera or the like built in the POS terminal device, and the product is recognized from the obtained image data using an image recognition technology. Here, when the product is photographed by the camera, not only the product image but also the background image is photographed. In the product recognition process, the background image becomes unnecessary noise. Therefore, it is necessary to remove the background image in order to perform the product recognition process. As a technique for removing a background image, Patent Document 1 discloses a technique for removing a background image from an input image by taking a difference between a background image taken in advance and an input image obtained by photographing a subject. It is disclosed.

JP 2001-52177 A

  The technique for removing a background image (background image) based on a difference between a background image captured in advance and an input image obtained by capturing a subject, such as Patent Document 1, has the following problems. When a product is photographed with the camera of the POS terminal device, an object other than the product may be reflected. For example, when a product is photographed with a camera, the body of the person holding the product may be reflected. In such a case, an object other than the product (for example, the body of a person who has the product) is also extracted as a difference from the background image. Therefore, the image of the product includes an image of an object other than the product (for example, a human body having the product). Also, the background itself may be different from when the background image is taken in advance, for example, when the color of the background changes due to the influence of outside light (for example, sunset). In such a case, since the background is recognized as a difference, the background image is not removed, and an image of an object other than the product is included in the product image. As described above, when an image of an object other than the product is included in the product image, the product recognition is performed in a state where the image of the object other than the product is included. In such a case, since the accuracy of product recognition is deteriorated, the recognition rate of the product may be deteriorated.

  The present invention has been made to solve such a problem, and in order to contribute to an improvement in the recognition rate of a product, a POS capable of extracting a product image from an image taken by an imaging unit. It is to provide a terminal device, a POS system, an image processing method, and a program.

  The POS terminal device according to the present invention includes an imaging unit that images an object to generate an image, a distance measuring unit that measures a distance to the position of an object corresponding to the image generated by the imaging unit, and the imaging unit Product image extracting means for extracting an image area corresponding to a position where the distance measured by the distance measuring means is within a first range determined in advance as an image of the product.

  The POS system according to the present invention includes a POS terminal device and a management device that communicates with the POS terminal device.

  Further, the image processing method according to the present invention captures an object to generate an image, measures a distance to the position of the object corresponding to the generated image, and measures the measured from the generated image. An image area corresponding to a position where the distance is within a predetermined first range is extracted as an image of the product.

  The program according to the present invention includes a step of causing an imaging unit to capture an object and generating an image, a step of causing a distance measurement unit to measure a distance to a position of an object corresponding to the generated image, and the generation A step of causing the computer to execute, as a product image, an image area corresponding to a position where the measured distance is within a predetermined first range from the measured image.

  According to the present invention, there is provided a POS terminal device, a POS system, an image processing method, and a program capable of extracting an image of a product from an image taken by an imaging unit in order to contribute to an improvement in the recognition rate of the product. it can.

It is a figure which shows the outline | summary of the POS terminal device concerning embodiment of this invention. 1 is a side view showing an external appearance of a POS terminal device according to a first embodiment; 2 is a diagram illustrating a hardware configuration of a POS terminal device according to a first embodiment; FIG. 2 is a functional block diagram of a POS terminal device according to a first embodiment; FIG. 3 is a flowchart showing processing of a start control unit according to the first exemplary embodiment; 3 is a flowchart showing processing of a recognition processing unit according to the first exemplary embodiment. FIG. 6 is a diagram for explaining processing of a product image extraction unit according to the first embodiment; It is a figure which illustrates the goods from which size differs even if a shape and a package are the same. It is a figure which shows the POS system concerning Embodiment 2. FIG. FIG. 3 is a diagram illustrating a hardware configuration of a management apparatus according to a second embodiment. FIG. 4 is a functional block diagram of a POS terminal device according to a second embodiment. FIG. 6 is a functional block diagram of a management apparatus according to a second embodiment.

(Outline of the embodiment of the present invention)
Prior to the description of the embodiment, an outline of the embodiment according to the present invention will be described. FIG. 1 is a diagram showing an outline of a POS terminal device 1 according to an embodiment of the present invention. As illustrated in FIG. 1, the POS terminal device 1 includes an imaging unit 2, a distance measurement unit 4, and a product image extraction unit 6.

  The imaging unit 2 captures an object and generates an image. The distance measuring unit 4 measures the distance to the position of the object corresponding to the image generated by the imaging unit 2. The product image extraction unit 6 uses, as an image of the product, an image area corresponding to a position where the distance measured by the distance measurement unit 4 is within a predetermined first range from the image generated by the imaging unit 2. Extract. The POS terminal device 1 according to the embodiment of the present invention can extract an image of a product from an image captured by the imaging unit 2 in order to contribute to an improvement in the recognition rate of the product.

(Embodiment 1)
Embodiments of the present invention will be described below with reference to the drawings.
FIG. 2 is a side view showing an appearance of the POS terminal apparatus 100 according to the first embodiment. FIG. 3 is a diagram illustrating a hardware configuration of the POS terminal device 100 according to the first embodiment. The POS terminal device 100 includes a store clerk display operation unit 102, a customer display unit 104, an information processing device 110, and a product reading device 120. The POS terminal device 100 is mounted on, for example, a counter stand (not shown), and a customer on the left side of FIG.

  The salesclerk display operation unit 102 is, for example, a touch panel, an LCD (Liquid Crystal Display), or a keyboard. The clerk display operation unit 102 displays information necessary for the clerk and receives operations of the clerk under the control of the information processing apparatus 110.

  The customer display unit 104 is, for example, a touch panel or an LCD. The customer display unit 104 displays information necessary for the customer under the control of the information processing apparatus 110. The customer display unit 104 may have an input device, and may accept a customer operation as necessary.

  The information processing apparatus 110 is a computer, for example. The information processing apparatus 110 includes a control unit 112 such as a CPU (Central Processing Unit), a storage unit 114 such as a memory or a hard disk, and a communication device 116. The information processing apparatus 110 controls the operations of the display operation unit 102 for clerk, the display unit 104 for customer, and the product reading device 120. Further, the information processing apparatus 110 performs necessary processing in accordance with the operation received by the store clerk display operation unit 102. Further, the information processing apparatus 110 performs necessary processing such as image processing in accordance with the image information read by the commodity reading device 120. The communication device 116 performs processing necessary to communicate with a management device such as a server connected via a network.

  The product reading device 120 includes a housing 122, a product reading surface 124 formed of a light transmissive material, and a three-dimensional camera 130. The product reading surface 124 is provided on the surface of the housing 122 on the clerk side, and the product is directed when the product is imaged (read). The three-dimensional camera 130 is provided inside the housing 122 on the side opposite to the product reading surface 124. When the clerk points the product received from the customer to the product reading surface 124, the three-dimensional camera 130 reads the product image. As a result, the POS terminal apparatus 100 performs merchandise recognition processing. Details will be described later.

  The three-dimensional camera 130 includes an imaging unit 132 and a distance sensor 134 (distance measuring means). The imaging unit 132 is an imaging device (camera) such as a CCD (Charge-Coupled Device), for example, and performs a process of reading an image of a product. Specifically, the imaging unit 132 images an object directed to the product reading surface 124, and generates a two-dimensional color image or monochrome image (two-dimensional image) including the image of the object. Hereinafter, the term “two-dimensional image” also means “image data indicating a two-dimensional image” as a processing target in information processing.

  The distance sensor 134 measures the distance from the distance sensor 134 to the position of the object directed to the product reading surface 124 by, for example, the TOF (Time Of Flight) method. That is, the distance sensor 134 irradiates light rays such as infrared rays, and measures the distance from the time required for the irradiated light rays to reciprocate to the object. In the present embodiment, infrared light is irradiated, but the present invention is not limited to this. For example, the distance sensor 134 may irradiate a laser.

  The distance sensor 134 includes an infrared irradiation unit 136, an infrared light reception unit 138, and a distance image generation unit 140. The infrared irradiation unit 136 irradiates infrared rays toward the outside of the product reading surface 124. When an object exists in the vicinity of the product reading surface 124, the infrared light irradiated by the infrared irradiation unit 136 reflects the object. The infrared light receiving unit 138 is, for example, an image sensor (for example, a CCD or a CMOS (Complementary Metal-Oxide Semiconductor)), and receives infrared light reflected from an object.

  The distance image generation unit 140 calculates the distance (depth) to the object from the time difference between the time when the infrared irradiation unit 136 irradiates infrared light and the time when the infrared light reception unit 138 receives reflected light. Specifically, the infrared irradiation unit 136 and the infrared light receiving unit 138 are synchronized, and the distance image generation unit 140 receives the phase of the infrared ray irradiated by the infrared irradiation unit 136 and the infrared light receiving unit 138 receives light. The time difference is measured by measuring the phase difference from the phase of the reflected light.

  More specifically, the distance image generation unit 140 measures a time difference for each reflected light from each position of the object received by the infrared light receiving unit 138, and measures a distance for each position. Thereby, the distance image generation unit 140 generates a distance image (three-dimensional image) that is a set of pixels indicating the distance of each position of the object. Hereinafter, the term “distance image” also means “image data indicating a distance image” as a processing target in information processing.

  For example, for a position 15 cm from the distance sensor 134, the pixel corresponding to the position in the distance image includes distance information indicating “distance 15 cm”. For the position 30 cm from the distance sensor 134, the pixel corresponding to the position in the distance image includes distance information indicating “distance 30 cm”.

  If the distance between the distance sensor 134 and the object is too close, the phase difference between the irradiated light and the reflected light cannot be detected, and the time difference cannot be measured. Therefore, the distance to the object cannot be measured. Therefore, the distance sensor 134 (three-dimensional camera 130) is adjusted so that the distance between the distance sensor 134 (three-dimensional camera 130) and the object is maintained in accordance with the minimum time difference (phase difference) that can be measured by the distance sensor 134. The product reading surface 124 is spaced apart by an interval L. That is, the distance sensor 134 (three-dimensional camera 130) is installed at a distance L from the product reading surface 124 so as to maintain a minimum measurable distance. The speed of the infrared ray (the speed of light) is 300,000 km / second, so the infrared ray travels 30 cm in 1 nanosecond. That is, when the distance between the distance sensor 134 and the object changes by 15 cm (round trip 30 cm), the time from when the light is irradiated until when the reflected light is received changes by 1 nanosecond. Therefore, for example, when the minimum time difference that can be measured is 1 nanosecond, the interval L is 15 cm.

  Further, the imaging unit 132 and the distance sensor 134 are arranged close to each other (that is, at substantially the same position). Therefore, the two-dimensional image obtained by the imaging unit 132 and the distance image obtained by the distance sensor 134 are configured to correspond to each other. That is, when the position P where the object to be photographed corresponds to the pixel position (X1, Y1) of the two-dimensional image, the position P substantially corresponds to the pixel position (X1, Y1) also in the distance image. In other words, each pixel position of the distance image obtained by the distance sensor 134 is configured to correspond to each pixel position of the two-dimensional image obtained by the imaging unit 132. Note that in the case where the position of the imaging unit 132 and the position of the distance sensor 134 are slightly deviated, the distance between the imaging unit 132 and the distance sensor 134 and the field of view of the imaging unit 132 and the distance sensor 134 respectively. From the corner, a process may be performed in which each pixel position of the distance image and each pixel position of the two-dimensional image are aligned.

  FIG. 4 is a functional block diagram of the POS terminal apparatus 100 according to the first embodiment. 5 and 6 are flowcharts showing processing of the POS terminal apparatus 100 according to the first embodiment. The POS terminal device 100 includes a start control unit 200 and a recognition processing unit 220. The start control unit 200 includes a distance image acquisition unit 202, an object approach determination unit 204, and a recognition process execution control unit 206. The recognition processing unit 220 includes a product image shooting control unit 222, a distance image shooting control unit 224, a product image extraction unit 226, a product recognition processing unit 228, and a product information storage unit 230.

  The start control unit 200 and the recognition processing unit 220 can be realized by causing a program to be executed under the control of the control unit 112, for example. More specifically, the program stored in the storage unit 114 is realized by executing the program under the control of the control unit 112. In addition, each component is not limited to being realized by software by a program, but may be realized by any combination of hardware, firmware, and software. Each component of the start control unit 200 and the recognition processing unit 220 may be realized by using an integrated circuit that can be programmed by the user, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of the above-described components.

  The start control unit 200 determines whether or not the object has approached the commodity reading surface 124 (that is, the three-dimensional camera 130), and controls whether or not the recognition processing unit 220 executes the process. The recognition processing unit 220 performs a product recognition process (product recognition process) when the start control unit 200 is controlled to execute the process.

  Hereinafter, the processes of the start control unit 200 and the recognition processing unit 220 will be specifically described. FIG. 5 is a flowchart illustrating processing of the start control unit 200 according to the first embodiment. FIG. 6 is a flowchart illustrating processing of the recognition processing unit 220 according to the first embodiment.

  The distance image acquisition unit 202 controls the distance sensor 134 to acquire a distance image (S102). Specifically, the distance image acquisition unit 202 controls the distance sensor 134 so as to capture a distance image in front of the product reading surface 124. The distance sensor 134 captures a distance image in front of the product reading surface 124 under the control of the distance image acquisition unit 202 and generates a distance image (image data indicating the distance image). The distance image acquisition unit 202 acquires the generated distance image and outputs it to the object approach determination unit 204.

  The object approach determination unit 204 determines whether or not an object has approached within the threshold Th1 (first threshold) using the distance image (S104). For example, the object approach determination unit 204 analyzes the distance image and determines whether or not there is a pixel indicating a distance within the threshold Th1 from the distance sensor 134. The object approach determination unit 204 determines that an object is approaching when there is a pixel indicating a distance within the threshold Th1. On the other hand, the object approach determination unit 204 determines that the object is not approaching when there is no pixel indicating a distance within the threshold Th1.

  The threshold value Th1 is determined in consideration of the distance from the product reading surface 124 to the product (object) when a store clerk or the like intends to recognize the product with the product reading surface 124 facing the product. Further, the threshold Th1 is determined so that there is no object between the product reading surface 124 and the position of the threshold Th1, except when the store clerk or the like directs the product to the product reading surface 124. For example, if the distance (Th1-L) from the product reading surface 124 to the position of the threshold Th1 is 25 cm, the threshold Th1 is 40 cm when the distance L is 15 cm.

  When the object approach determining unit 204 determines that an object is approaching within the threshold Th1 (YES in S104), the recognition process execution control unit 206 starts the product recognition process for the recognition processing unit 220. Control is performed (S106).

  On the other hand, when the object approach determining unit 204 determines that the object is not approaching within the threshold Th1 (NO in S104), the recognition process execution control unit 206 determines whether the recognition processing unit 220 is performing the product recognition process. Is determined (S108). If the recognition processing unit 220 has not performed the product recognition process (NO in S108), the process returns to S102.

  On the other hand, when the recognition processing unit 220 is performing product recognition processing (YES in S108), the recognition process execution control unit 206 controls the recognition processing unit 220 to end the product recognition processing (S110). In other words, the process of the start control unit 200 may be always performed while the POS terminal device 100 is activated. Even when the recognition processing unit 220 starts the product recognition process because the object (product) once approaches the product reading surface 124, the recognition process is completed or while the recognition process is being performed. When the object (product) is separated from the product reading surface 124 (the distance from the distance sensor 134 to the object exceeds the threshold Th1), the start control unit 200 ends the product recognition process for the recognition processing unit 220. Control to do.

  As described above, the POS terminal device 100 according to the present embodiment performs the product recognition process only when the object (product) approaches the product reading surface 124 (distance sensor 134). When the product recognition process is performed, the load on the POS terminal device 100 (in particular, the imaging unit 132, the control unit 112, and the storage unit 114) increases. Therefore, with this configuration, it is possible to reduce the resource load of the POS terminal device 100 when it is not necessary to perform the product recognition process. The “resource” here includes not only hardware resources of the POS terminal apparatus 100 itself but also network resources.

  The product image capturing control unit 222 causes the imaging unit 132 to capture a two-dimensional image including a product image (product image) (S202). Specifically, the product image capturing control unit 222 controls the imaging unit 132 to image the product directed to the product reading surface 124. Then, the product image photographing control unit 222 acquires the two-dimensional image generated by the imaging unit 132 and outputs it to the product image extraction unit 226. Note that the two-dimensional image may include a background image (background image) in addition to the product image.

  The distance image capturing control unit 224 causes the distance sensor 134 to capture a distance image corresponding to the two-dimensional image (S204). Specifically, the distance image capturing control unit 224 controls the distance sensor 134 to capture a distance image including a product directed to the product reading surface 124. Then, the distance image capturing control unit 224 acquires the distance image generated by the distance sensor 134 and outputs it to the product image extraction unit 226. As described above, since the imaging unit 132 and the distance sensor 134 are arranged close to each other, the distance image corresponds to a two-dimensional image. Note that the processing of the distance image capturing control unit 224 (S204) may be replaced with the processing of S102. That is, the distance image acquired by the distance image acquisition unit 202 may be output to the product image extraction unit 226.

  The product image extraction unit 226 discriminates an area in the distance image whose distance from the distance sensor 134 is within the range S1 (first range), and extracts an image area corresponding to the area from the two-dimensional image as a product image. (S206). Further, the product image extraction unit 226 outputs the extracted product image to the product recognition processing unit 228.

  Specifically, the product image extraction unit 226 superimposes the two-dimensional image and the distance image. In addition, the product image extraction unit 226 determines which region in the two-dimensional image corresponds to the area whose distance is in the range S1 in the distance image. Further, the product image extraction unit 226 determines that the corresponding area is an image area corresponding to the product image. Then, the product image extraction unit 226 extracts the image area as a product image.

  More specifically, the product image extraction unit 226 extracts a pixel (in-range pixel) indicating a distance in the range S1 from the distance image. In addition, the product image extraction unit 226 determines which pixel in the two-dimensional image each extracted in-range pixel corresponds to. Further, the product image extraction unit 226 extracts pixels corresponding to the in-range pixels from the two-dimensional image. The product image extraction unit 226 extracts a set of pixels extracted from the two-dimensional image as an image area corresponding to the product image.

  The lower limit of the range S1 may be a distance (interval L) from the distance sensor 134 to the product reading surface 124, for example. In addition, the upper limit of the range S1 is set in consideration of how close the product is to the product reading surface 124 when a clerk or the like directs the product to the product reading surface 124 in order to read the product. For example, the upper limit of the range S1 may be set to be smaller than the threshold Th1. For example, if the distance from the product reading surface 124 to the upper limit position of the range S1 is 15 cm, the range S1 is 15 cm <S1 <30 cm when the distance L is 15 cm. In addition, the lower limit of range S1 does not need to be set, In that case, a lower limit will be 0 cm.

  FIG. 7 is a diagram for explaining the processing of the product image extraction unit 226. FIG. 7A is a diagram illustrating a two-dimensional image Im2 including a product image captured by the imaging unit 132 under the control of the product image capturing control unit 222. The two-dimensional image Im2 includes a product image A (indicated by a solid line) and a background image B (indicated by a one-dot chain line). The product corresponding to the product image A is at a position where the distance from the distance sensor 134 (product reading surface 124) is within the range S1. The object corresponding to the background image B is located farther away from the distance sensor 134 (product reading surface 124) than the range S1.

  FIG. 7B is a diagram illustrating a distance image Im3 corresponding to a two-dimensional image captured by the distance sensor 134 under the control of the distance image capturing control unit 224. Here, the region indicated by C indicates an area where the distance from the distance sensor 134 is within the range S1. A region indicated by diagonal lines indicated by D indicates an area where the distance from the distance sensor 134 deviates from the range S1.

  The product image extraction unit 226 superimposes the two-dimensional image Im2 and the distance image Im3. Then, the product image extraction unit 226 extracts an image region corresponding to the area indicated by C in the distance image Im3 from the two-dimensional image Im2. As a result, a product image E as illustrated in FIG. 7C is extracted. Here, the product image E does not include a background image. That is, the product image extraction unit 226 removes the background image B from the two-dimensional image Im2.

  The product recognition processing unit 228 (FIG. 4) performs product recognition processing using the product image extracted by the product image extraction unit 226 (S208). In addition, the product recognition processing unit 228 stores the product information obtained by the product recognition processing in the product information storage unit 230. The product information includes information included in a barcode attached to the product. Further, the product information may include the size (capacity) of the product. The POS terminal apparatus 100 uses the product information obtained by the product recognition processing by the product recognition processing unit 228 to perform settlement processing for the product.

  Regarding the product recognition process, specifically, for example, the product recognition processing unit 228 previously stores a product name and information (reference product information) related to the product in association with each other. The product recognition processing unit 228 performs pattern matching between the extracted product image and reference product information stored in advance. The reference product information is exemplified below.

  For example, the reference product information may be an image (reference product image) that serves as a reference for the product. In that case, the product recognition processing unit 228 collates the extracted product image with the reference product image. Then, the product recognition processing unit 228 associates the product with the product name corresponding to the reference product image when the similarity between the two satisfies an allowable value.

  Further, for example, the reference product information may be data (product feature data) indicating a feature that is a reference of the product. The product feature data includes, for example, information indicating the shape of the product, information indicating the color of the product, information indicating the texture of the product (such as gloss), information indicating character information and a pattern attached to the package of the product, May be included. In this case, the product recognition processing unit 228 extracts the feature of the image from the extracted product image. Then, the product recognition processing unit 228 collates the extracted image feature with the product feature data. Then, the product recognition processing unit 228 associates the product with the product name corresponding to the product feature data when the similarity between the two satisfies the allowable value. In addition, the product recognition processing unit 228 may recognize the product name by reading character information attached to the product package using an OCR (Optical Character Reader).

  Here, the background of the product image extracted by the product image extraction unit 226 is removed. Therefore, when the product recognition processing unit 228 performs the product recognition process, it is not necessary to exclude the background. If the 2D image includes not only the product image but also the background image, in the product recognition process, it is first necessary to recognize where the product image is in the 2D image. In particular, when an unspecified number of customers use the POS terminal device 100, such as self-checkout, the position on the product reading surface 124 on which the product is directed differs depending on the customer. In this process of recognizing where the product image is, for example, the reference product information must be collated for all images included in the two-dimensional image. Therefore, the processing time becomes enormous.

  On the other hand, in the present embodiment, since the product image itself is used, it is not necessary to recognize where the product image is in the two-dimensional image. Therefore, the POS terminal device 100 according to the present embodiment can improve the processing speed of the product recognition process. In other words, the POS terminal apparatus 100 according to the present embodiment can reduce the resource load in the product recognition process. In addition, the data amount of the product image is smaller than the data amount of the two-dimensional image because the background is removed. Therefore, since the amount of data to be processed can be reduced, it is possible to realize resource reduction and load reduction. Therefore, it is also possible to use a resource-poor device such as a tablet terminal as the POS terminal device 100 according to the present embodiment. The “resource” here includes not only hardware resources of the POS terminal apparatus 100 itself but also network resources. That is, in this embodiment, it is possible to reduce the network load.

  Furthermore, when a background image is included in the product image, the background image is considered in the product recognition process. As a result, the recognition rate deteriorates. On the other hand, since the background of the product image extracted by the product image extraction unit 226 is removed, the recognition rate can be improved.

  In addition, a two-dimensional image including a product image photographed by the imaging unit 132 may include a body image of a store clerk who has the product. Here, when a product image is extracted, the method of using a difference from a background image captured in advance recognizes the body of the store clerk as a difference. Therefore, the extracted product image includes an image of the body of the store clerk and the body image of the store clerk becomes noise and the recognition rate decreases.

  Here, when a person directs a product to the product reading surface 124, the product is usually extended with his hand directed. Therefore, a body such as a store clerk is usually away from the product reading surface 124 (distance sensor 134). Therefore, in this embodiment, the product image extraction unit 226 can remove an image of a body such as a store clerk. Therefore, the merchandise recognition processing unit 228 can perform merchandise recognition processing using only the merchandise image without considering the image of the body of a store clerk or the like. Therefore, the POS terminal device 100 according to the present embodiment can further improve the recognition rate.

  Further, in the method using a difference from a previously captured background image, the product image is extracted even when the background color is different from the previously captured background image due to the influence of external light (for example, sunset). At this time, this background is also recognized as a difference. Therefore, the background image is also included in the product image, and the background image becomes noise and the recognition rate is lowered.

  Here, the object corresponding to the background is separated from the product reading surface 124 (distance sensor 134). Therefore, in the present embodiment, the product image extraction unit 226 can reliably remove the background regardless of the change in the background color. Therefore, the POS terminal device 100 according to the present embodiment can further improve the recognition rate.

  Some products have different sizes even if the shape and package of the product are the same. For example, as shown in FIG. 8, a plurality of types of plastic bottle drinks having different sizes (capacities) are sold even if the contents are the same. Such products generally have different prices depending on their sizes. In such a case, the size of the product cannot be recognized simply by performing the product recognition process using the product image. Therefore, in order to make a settlement at an appropriate price, it is necessary for a clerk or the like to manually input the price or capacity.

  On the other hand, as described above, the POS terminal apparatus 100 according to the present embodiment can measure the distance to the product by the distance sensor 134. The size of the product image in the two-dimensional image becomes smaller as the distance (depth) becomes longer, and becomes larger as the distance (depth) becomes shorter, even if the actual product has the same size. That is, the actual size of the product can be grasped geometrically from the size of the product image in the two-dimensional image and the distance to the product.

  Therefore, the product recognition processing unit 228 may acquire information indicating the distance to the product from the distance image capturing control unit 224 and measure the size of the product image to recognize the size of the product. Specifically, the product recognition processing unit 228 acquires a distance image from the distance image shooting control unit 224. In addition, the product recognition processing unit 228 acquires distance information for each pixel in the product image area extracted by the product image extraction unit 226 using the distance image. Further, the product recognition processing unit 228 calculates the distance to the product from the acquired distance information of each pixel. In the calculation method, for example, the distance indicated by the pixel corresponding to the edge of the product image may be the distance to the product, or the average of the distances indicated by each pixel in the area of the product image may be the distance to the product. .

  Further, the product recognition processing unit 228 measures the size of the product image in the two-dimensional image. As the size of the product image, for example, a vertical dimension and a horizontal dimension are measured. Then, the product recognition processing unit 228 calculates the actual product dimensions from the size of the product image and the distance to the product. Here, the reference product information serving as a reference for the product recognition process may include the size and capacity of the product. Therefore, the product recognition processing unit 228 can grasp the name and capacity of the product (“product name ABC capacity 500 ml” in the example of FIG. 8).

(Embodiment 2)
Next, a second embodiment will be described. As described later, the second embodiment is different from the first embodiment in that the product recognition process is not performed by the POS terminal apparatus 100. Note that components that are substantially the same as those in the first embodiment are given the same reference numerals, and descriptions thereof are omitted.

  FIG. 9 is a diagram illustrating a POS system 300 according to the second embodiment. As illustrated in FIG. 9, the POS system 300 includes a POS terminal device 100 and a management device 310. The POS terminal device 100 and the management device 310 are communicably connected. The communication between the two may be either wired communication or wireless communication, and various communication standards can be applied. The POS terminal device 100 and the management device 310 may be connected to each other via a network (for example, a wireless local area network (LAN) or the Internet). Further, the POS terminal device 100 and the management device 310 may communicate with each other by near field communication such as infrared communication or Bluetooth (registered trademark).

  The POS terminal apparatus 100 according to the second embodiment has substantially the same hardware configuration as the POS terminal apparatus 100 according to the first embodiment. The POS terminal device 100 communicates with the management device 310 using the communication device 116. In this case, the communication device 116 performs processing necessary for communicating with the management device 310.

  The management device 310 is an information processing device that manages product information and the like. The management device 310 may be disposed in a store where the POS terminal device 100 is disposed. Further, the management device 310 may collectively manage the POS terminal devices 100 arranged in a plurality of stores. In this case, the management device 310 is different from the store in which the POS terminal devices 100 are arranged. Can be placed in place. Moreover, the management apparatus 310 is a server, for example, and may be a cloud server.

  FIG. 10 is a diagram illustrating a hardware configuration of the management apparatus 310 according to the second embodiment. The management device 310 includes a control unit 312 such as a CPU, an input / output unit 314 that is a user interface such as a touch panel, an LCD, or a keyboard, a storage unit 316 such as a memory or a hard disk, and a communication device 318. The communication device 318 performs processing necessary to communicate with the POS terminal device 100 (or other management device 310).

  FIG. 11 is a functional block diagram of the POS terminal apparatus 100 according to the second embodiment. The POS terminal device 100 includes a start control unit 200 and a recognition processing unit 240. The start control unit 200 includes a distance image acquisition unit 202, an object approach determination unit 204, and a recognition process execution control unit 206. The recognition processing unit 240 includes a product image shooting control unit 222, a distance image shooting control unit 224, a product image extraction unit 226, and a product image transmission unit 248. As described above, the start control unit 200 and the recognition processing unit 240 can be realized by causing a program to be executed under the control of the control unit 112, for example.

  The recognition processing unit 240 according to the second embodiment is different from the recognition processing unit 220 according to the first embodiment in that it does not include the product recognition processing unit 228 and the product information storage unit 230 but includes a product image transmission unit 248. . The product image extraction unit 226 outputs the extracted product image to the product image transmission unit 248. The product image transmission unit 248 transmits a product image (product image image data) to the management apparatus 310. Note that the product image transmission unit 248 may transmit the current time and the identification information of the POS terminal device 100 to the management device 310 when transmitting the product image. Further, the product image transmission unit 248 may also transmit a distance image (image data of the distance image) to the management device 310 so that the size of the product can be calculated in the management device 310.

  FIG. 12 is a functional block diagram of the management apparatus 310 according to the second embodiment. The management device 310 has a recognition processing unit 320. The recognition processing unit 320 includes a product image receiving unit 322, a product recognition processing unit 328, and a product information storage unit 330.

  Note that the recognition processing unit 320 can be realized, for example, by executing a program under the control of the control unit 312. More specifically, the program stored in the storage unit 316 is realized by executing the program under the control of the control unit 312. In addition, each component is not limited to being realized by software by a program, but may be realized by any combination of hardware, firmware, and software. Each component of the recognition processing unit 320 may be realized by using an integrated circuit that can be programmed by the user, such as a field-programmable gate array (FPGA) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of the above-described components.

  The product image receiving unit 322 receives the product image (product image data) transmitted by the POS terminal device 100 and outputs the product image to the product recognition processing unit 328. The product recognition processing unit 328 has substantially the same function as the product recognition processing unit 228 according to the first embodiment. Therefore, the product recognition processing unit 328 performs product recognition processing using the product image extracted by the product image extraction unit 226 in the same manner as in the first embodiment. Further, the product recognition processing unit 328 stores the product information obtained by the product recognition processing in the product information storage unit 330. Furthermore, the management apparatus 310 transmits the obtained product information to the POS terminal apparatus 100. The POS terminal device 100 uses the product information received from the management device 310 to perform settlement processing for the product.

  As in the second embodiment, the product recognition process is performed not by the POS terminal apparatus 100 but by the management apparatus 310, so that it is not necessary for each POS terminal apparatus 100 to store reference product information necessary for the product recognition process. The POS terminal device 100 does not need to perform product recognition processing. Therefore, resources of the POS terminal device 100 can be saved. Also, the present embodiment can be applied to a POS terminal device 100 having a scarce resource, such as a tablet terminal. Also in the second embodiment, a product image is extracted by the product image extraction unit 226. Therefore, as in the first embodiment, in the product recognition process by the management apparatus 310, the resource load is reduced, the processing speed is improved, the recognition rate is improved, and the product size (capacity) is set. It is possible to grasp.

  Further, as described above, the product image extracted by the product image extraction unit 226 has the background image removed from the two-dimensional image obtained by the imaging unit 132. Therefore, the data amount of the product image is smaller than the data amount of the two-dimensional image. When the product recognition process is performed by the management device 310, if the POS terminal device 100 transmits image data of a two-dimensional image to the management device 310, the load of the communication network increases because the data amount is large. On the other hand, when the POS terminal device 100 transmits the image data of the product image to the management device 310, the data amount is small, so the load on the communication network is reduced.

(Modification)
Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention. For example, the order of processing in the flowcharts described above can be changed as appropriate. Further, at least one of the plurality of processes in the above-described flowchart may not be provided. For example, in the flowchart of FIG. 6, the process of S202 may be performed after the process of S204. Further, in the flowchart of FIG. 5, the processing of S108 and S110 may be omitted.

  Moreover, although the structure concerning this Embodiment was applied to the POS terminal device, it is not restricted to this. For example, the present invention can be applied to a general object recognition device such as an object recognition device used for sorting packages in a warehouse or the like, and a system including the object recognition device.

  Further, the POS terminal device 100 according to the present embodiment can be applied to, for example, a self-checkout. When the customer uses the POS terminal as in the self-checkout, the customer is not accustomed to having the reading device read the barcode attached to the product. For this reason, self-checkout requires a method that does not use a barcode, that is, a method that allows a product to be read directly. Therefore, by applying the POS terminal device 100 according to the present embodiment to the self-registration, the problem caused by causing the commodity to be read directly as described above is solved.

  Further, as described above, the POS terminal device 100 according to the present embodiment can also be applied to a terminal with scarce resources such as a tablet terminal (tablet POS). In this case, the three-dimensional camera 130 may not be built in the tablet terminal, and may be a separate (external) device from the tablet terminal.

  In the above-described embodiment, the start control unit 200 controls the start or end of the product recognition process. However, the present invention is not limited to this. That is, the start control unit 200 is not an essential component, and the processing of the start control unit 200 shown in FIG. 5 is not necessarily required. However, as described above, resource reduction can be realized by performing the processing of the start control unit 200. Instead of the start control unit 200, the salesclerk or the like may control the start or end of the product recognition process by operating a switch that starts or ends the product recognition process. However, by performing the process of the start control unit 200, it becomes unnecessary for a store clerk or the like to operate the switch.

  Further, the configuration of the first embodiment and the configuration of the second embodiment may be combined. For example, in the POS terminal device 100 according to the second embodiment, the product recognition process may be performed. In other words, the POS terminal apparatus 100 according to the second embodiment may include the product recognition processing unit 228. In this case, when the load of the POS terminal device 100 increases from a predetermined first load value, the POS terminal device 100 transmits a product image to the management device 310, and the management device 310 performs product recognition processing. You may do it. On the other hand, when the load of the management device 310 is increased from a predetermined second load value, or when the load of the communication network is increased from a predetermined third load value, the POS The terminal device 100 may perform product recognition processing with its own POS terminal device 100 without transmitting the product image to the management device 310.

  Thus, load distribution can be performed as appropriate according to the load on the POS terminal device 100, the load on the management device 310, and the load on the communication network. In this case, the POS terminal device 100 or the management device 310 measures the load of the POS terminal device 100, the load of the management device 310, and the load of the communication network, and the measured load and the first to third load values. And a means for comparing each of the above.

  The program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

DESCRIPTION OF SYMBOLS 1 POS terminal device 2 Image pick-up part 4 Distance measurement part 6 Product image extraction part 100 POS terminal device 110 Information processing apparatus 120 Goods reading apparatus 124 Goods reading surface 130 Three-dimensional camera 132 Imaging part 134 Distance sensor 136 Infrared irradiation part 138 Infrared light receiving part 140 Distance image generation unit 200 Start control unit 202 Distance image acquisition unit 204 Object approach determination unit 206 Recognition processing execution control unit 220 Recognition processing unit 222 Product image shooting control unit 224 Distance image shooting control unit 226 Product image extraction unit 228 Product recognition processing Unit 230 product information storage unit 240 recognition processing unit 248 product image transmission unit 300 POS system 310 management device 312 control unit 320 recognition processing unit 322 product image reception unit 328 product recognition processing unit 330 product information storage unit

Claims (16)

Imaging means for imaging an object to generate an image;
Distance measuring means for measuring the distance to the position of the object corresponding to the image generated by the imaging means;
Product image extracting means for extracting, from the image generated by the imaging means, an image area corresponding to a position where the distance measured by the distance measuring means is within a predetermined first range as a product image; A POS terminal device. A determination unit for determining that the object has approached the distance measured by the distance measurement unit within a predetermined first threshold;
The POS terminal apparatus according to claim 1, wherein the product image extraction unit performs the extraction process when the determination unit determines that the object has approached. The POS terminal device according to claim 1, further comprising a recognition processing unit configured to perform a recognition process of the product based on the extracted product image. The recognition processing means includes
Measure the size of the product image extracted by the product image extraction unit in the image captured and generated by the imaging unit,
Wherein the size of the measured product images, based on the distance to the position of said distance thus measured to the measuring unit object, recognizing the size of the product,
The POS terminal device according to claim 3, wherein recognition processing of the product is performed based on the size of the recognized product. 5. The transmission apparatus according to claim 1, further comprising: a transmission unit configured to transmit data indicating the image of the extracted product to a management device that performs recognition processing of the product based on the extracted product image. Item POS terminal device. The imaging means and the distance measuring means are provided inside a housing,
The housing has a reading surface to which the product is directed when the product is imaged,
6. The distance measurement unit according to claim 1, wherein the distance measurement unit is installed away from the reading surface by a predetermined distance according to a minimum distance that can be measured by the distance measurement unit. POS terminal device. The POS terminal device according to any one of claims 1 to 6,
A POS system comprising: a management device that communicates with the POS terminal device. The POS terminal device further includes recognition processing means for performing recognition processing of the product based on the extracted product image,
The POS terminal device and the management device are connected via a communication network,
If the load of the POS terminal device is increased than the first load value set in advance, before Symbol POS terminal device to the management device, and transmits data indicating an image of the extracted product The management device performs recognition processing for the product,
If the load of the management device is increased than a second load value set in advance, or when said load on the communication network is increased from the third load value set in advance, before Symbol 8. The POS system according to claim 7, wherein the POS terminal device does not transmit data indicating the image of the extracted product to the management device, and the recognition processing unit performs a recognition process of the product. Image an object to generate an image,
Measure the distance to the position of the object corresponding to the generated image,
An image processing method for extracting, from the generated image, an image area corresponding to a position where the measured distance is within a predetermined first range as a product image. Determining that the measured distance has approached an object within a predetermined first threshold;
The image processing method according to claim 9, wherein a process of extracting an image of the product is executed when it is determined that the object has approached. The image processing method according to claim 9 or 10, wherein recognition processing of the product is performed based on the extracted product image. Measure the size of the image of the extracted product in the captured and generated image,
Recognizing the size of the product based on the size of the image of the measured product and the distance to the position of the measured object;
The image processing method according to claim 11, wherein the product is recognized based on the size of the recognized product. Causing the imaging means to image an object to generate an image;
Causing the distance measuring means to measure the distance to the position of the object corresponding to the generated image;
A program for causing a computer to execute, from the generated image, a step of extracting an image area corresponding to a position where the measured distance is within a predetermined first range as a product image. Further causing the computer to execute a step of determining that an object has approached the measured distance within a predetermined first threshold;
The program according to claim 13, wherein a step of extracting an image of the product is executed when it is determined that the object has approached. The program according to claim 13 or 14, further causing the computer to execute a step of recognizing the product based on the image of the extracted product. Measuring the size of the image of the extracted product in the captured and generated image;
A step of recognizing the size of the product based on the measured image size of the product and the distance to the position of the measured object, and a recognition process for the product based on the size of the recognized product The program according to claim 15, further causing the computer to execute:

Images