JP6628336B2 - Information processing system - Google Patents

Description

  The present invention relates to an information processing system for specifying an area in captured image information.

  In the retail industry, it is known that the display status of a product affects the sale of the product. Therefore, as a product manufacturing company and a sales company, it is possible to link to a development strategy and a sales strategy of the company's product by grasping what kind of product of the company or another company is displayed in the store.

  On the other hand, in order to achieve this, it is important to accurately identify the products displayed at stores. Therefore, it is conceivable that a human may manually specify the displayed product from the image information of the display shelf of the store. In this case, the product can be specified almost accurately. However, in order to continuously grasp the display status of products, it is necessary to grasp the display status at regular intervals. However, every time a human identifies a product from image information obtained by photographing the display shelf of a store. Is burdensome and inefficient.

  Therefore, it is necessary to automatically identify the products and product tags displayed there from image information obtained by photographing the display shelves of the store, and to grasp information on the displayed products. For example, there is a method of using an image recognition technique for an image of a display shelf of a store based on a sample image of each product. As these prior arts, for example, there is a system that manages the display status of commodities by using the technologies described in Patent Literatures 1 to 3 below.

JP-A-5-342230 JP-A-5-334409 International Publication 2012/029548

  The invention of Patent Literature 1 is a system that supports even a person who does not have knowledge of a display shelf on which a product should be displayed, so that the product can be displayed. Therefore, it is not possible to grasp the information of the actually displayed product. Patent Document 2 discloses a shelf layout support system that supports display of products, which is a system that supports input of product images. However, the system of Patent Document 2 only supports input of a product image when using the shelf layout support system, and even if this system is used, it is necessary to grasp the information of the products actually displayed. Can not.

  Patent Literature 3 is an invention in which, when there is an empty space in a display shelf, a product to be displayed in the empty space is specified, or when a product to be displayed on the display shelf is misplaced, the notification is made. In this invention as well, the product displayed on the display shelf is specified by performing image matching processing between the product and the product displayed on the display shelf, but the actual situation is that the recognition accuracy is low.

  When using image recognition technology based on a sample image of a product on an image of a display shelf, the similarity between the image features of the sample image and the area of the display portion of the product in the image of the display shelf Recognize products by using their characteristics. The image feature amount of the region of the display portion of the product may vary depending on the range of the region specified as the display portion of the product from the image of the display shelf. Therefore, if the partial image can be specified more accurately, it is possible to improve the accuracy in matching with the sample image. Also, in the case where the information described in the product tag is OCR-recognized from the image of the display shelf, the recognition accuracy may vary depending on the accuracy of the area to be recognized as the product tag.

  The inventor of the present invention has invented an information processing system for accurately specifying an area where products are displayed or an area of a product tag in image information obtained by photographing a display shelf.

The invention according to claim 1 is an information processing system that uses image information and depth information, wherein the information processing system specifies a product surface or a depth face area using the depth information. When, using said identified product surface or depth face area, and an image recognition processing unit that specifies an image face area in the image information, which have a, the depth processing unit, the depth information A surface is detected based on the detected surface, the type of the surface is specified using the context of the detected surface, and the surface type is a product surface, and a comparison is made based on a determination condition of a depth face area for each package type. Thus, the information processing system specifies the depth face area .

In the present invention, since the product surface or the depth face area is specified by physical depth information and the image face area in the image information is specified using the same, the conventional method of specifying only the image information is used. Can also be specified with high accuracy. Further, there are various methods for specifying the depth face area, but by using the processing of the present invention, the depth face area can be specified with high accuracy.

  In the above invention, the determination condition is that, when the package type is a can or a bottle, the cylindrical area within a predetermined size is specified as a depth face area of the can or the bottle. If both were cans and bottles, there was a cylinder within a certain size range, the area without a small cylinder above it was the depth face area of the can, and the cylinder was within a certain size range. If the package type is a box, the area with a small cylinder above it is identified as the depth face area of the bottle. If the package type is a hanging product, or if the package type is a hanging product, an area of a rectangular flat surface or uneven surface within a certain size range is used as the depth face area of the hanging product. Identified as including one or more one of the can be configured as an information processing system.

  Although there are various conditions for the determination, the determination conditions of the present invention can be used for general purposes.

The invention according to claim 3 is an information processing system using image information and depth information, wherein the information processing system uses a depth processing unit that specifies a product surface or a depth face area using the depth information. If the identified using a product surface or depth face area, and an image recognition processing unit that specifies an image face area in the image information, which have a, the image recognition processing unit, the depth processor In the above, if the product surface or the depth face area could not be specified, or if the depth face area was equal to or larger than a predetermined width, the image information may include a shadow between the products and the repetition of the image. Identifying an image face area based on at least one of a pattern, a step on an upper side of a package, and a product width, and determining a similarity between the identified image face area and specimen information. , To identify the product identification information of commodities displayed on the image face area, an information processing system.

  Even when the depth information is used, the product surface or the depth face area cannot always be specified. Therefore, the method may coexist with a method of specifying an image face area from image information.

The invention of claim 1 can be realized by causing a computer to read and execute the program of the present invention. That is, a computer is used to execute a depth processing unit that specifies a product surface or a depth face area using depth information, and an image that specifies an image face area in image information using the specified product surface or depth face area. An information processing program functioning as a recognition processing unit, wherein the depth processing unit detects a surface based on the depth information, specifies a type of the surface using the context of the detected surface, The information processing program can be configured as an information processing program that specifies the depth face region by comparing the surface type, which is the product type, with the determination condition of the depth face region for each package type .

The invention of claim 3 can be realized by causing a computer to read and execute the program of the present invention. That is, a computer is used to execute a depth processing unit that specifies a product surface or a depth face area using depth information, and an image that specifies an image face area in image information using the specified product surface or depth face area. An information processing program functioning as a recognition processing section , wherein the image recognition processing section has failed to identify the product surface or the depth face area in the depth processing section, or has determined that the depth face area has a predetermined depth. If the width is not less than the width of the image, the image face area is specified in the image information based on at least one of a shadow between products, a repetition pattern of the image, a step on the upper side of the package, and a product width. Then, by determining the similarity between the specified image face area and the specimen information, the product identification of the product displayed in the image face area is determined. Specifying the information, it can be configured as an information processing program.

  By using the information processing system of the present invention, a predetermined area can be specified with high accuracy from image information obtained by photographing an object to be photographed. In particular, it is possible to accurately specify a product display portion and a product tag area in an image of a display shelf.

It is a block diagram which shows an example of the processing function of the information processing system of this invention typically. It is a block diagram which shows an example of the processing function of the depth processing part in the information processing system of this invention typically. It is a block diagram which shows an example of the processing function of the image recognition processing part in the information processing system of this invention typically. FIG. 3 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. 5 is a flowchart illustrating an example of an entire processing process in the information processing system of the present invention. It is a figure which shows typically the correspondence of the captured image information and the depth map by depth information. FIG. 4 is a diagram illustrating an example of captured image information. It is a figure which shows an example of the depth map which mapped depth information typically. FIG. 8 is a diagram illustrating an example of normal image information obtained by normalizing the captured image information of FIG. 7. It is a figure which shows typically performing the same emplacement process as the emplacement process performed with respect to the captured image information about a depth map. FIG. 9 is a diagram schematically illustrating an example of a depth map in a state where normalization processing has been performed on the depth map of FIG. 8. FIG. 6 is a diagram illustrating an example of a surface type specifying process in a surface type specifying processing unit. It is an image figure which specifies a face when a package type is a can. It is an image figure which specifies a face when a package type is a bottle. It is an image figure which specifies a face when a package type is a box. It is an image figure which specifies a face when a package type is a hanging product. It is a figure which shows the image information which binarized the image information of the goods tag area | region. It is a figure showing the state where a box was generated. FIG. 7 is a diagram illustrating a state in which a block is generated. It is a figure which shows an example of the product name described in the product tag, the product name with the shortest edit distance among the product names stored in a product dictionary as a result of OCR recognition, and the edit distance. It is a figure showing an example of a table of correspondence of edit distance which may be decided for every character number of the last candidate. FIG. 4 is a diagram illustrating an example of sample information stored in a sample information storage unit. It is a figure which shows the face area in a display shelf typically. FIG. 4 is a diagram illustrating an example of captured image information. FIG. 9 is a diagram illustrating an example of photographed image information placed image information obtained by performing a normalization process on photographed image information. FIG. 13 is a block diagram illustrating an example of a processing function of an image recognition processing unit according to the second embodiment. FIG. 13 is a block diagram illustrating an example of a processing function of an image recognition processing unit according to a third embodiment. FIG. 26 is a diagram illustrating a state where a product tag area is cut out from the normal image information of FIG. 25. It is a figure which shows the image information of the product tag area | region which was set as the normal. It is a figure in Example 4 which shows a goods tag area | region, a goods surface, etc. typically. FIG. 14 is a diagram schematically illustrating a process according to a fourth embodiment. FIG. 11 is a diagram schematically illustrating an example of a process of cutting out an image face area corresponding to a depth face area when the package type is a can or a bottle. It is a figure which shows typically an example of the process which cuts out the image face area | region corresponding to a depth face area | region when a package type is a box or a hanging product.

  Examples of the processing functions of the information processing system 1 of the present invention are shown in block diagrams in FIGS. The information processing system 1 uses a management terminal 2 and an input terminal 3. FIG. 1 is a block diagram showing an example of an overall processing function of the information processing system 1, FIG. 2 is a block diagram showing an example of a processing function of a depth processing unit 23 described later, and FIG. FIG. 3 is a block diagram illustrating an example of a processing function of a processing unit 24.

  The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The input terminal 3 is a terminal for inputting image information and depth information of a display shelf of a store.

  The management terminal 2 and the input terminal 3 in the information processing system 1 are realized using a computer. FIG. 4 schematically shows an example of a hardware configuration of a computer. The computer includes an arithmetic unit 70 such as a CPU that executes arithmetic processing of a program, a storage unit 71 such as a RAM or a hard disk that stores information, a display unit 72 such as a display that displays information, and a computer that can input information. It has an input device 73 such as a keyboard and a mouse, and a communication device 74 for transmitting and receiving processing results of the arithmetic device 70 and information stored in the storage device 71 via a network such as the Internet or a LAN.

  When the computer has a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, for example, but are not limited thereto.

  The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that an input can be directly performed on the display by a predetermined input device (such as a touch panel pen) or a finger.

  The input terminal 3 includes an image capturing device such as a camera in addition to the devices described above. As the input terminal 3, a portable communication terminal such as a mobile phone, a smartphone, and a tablet computer can be used. The input terminal 3 captures image information (photographed image information) using visible light with the photographing device, and acquires distance information (depth information) between the photographing target and the photographing device as two-dimensional information. The information to which the depth information is mapped is called a depth map in this specification. The depth map corresponds to the photographed image information, and the depth information to each object shown in the photographed image information is mapped to a position corresponding to the photographed image information. In the depth map, at least a range corresponding to the captured image information is divided into meshes, and depth information is given to each mesh. The vertical and horizontal sizes of the mesh depend on the accuracy of the depth information that can be obtained. The smaller the vertical and horizontal sizes of the mesh, the higher the accuracy. Usually, the range is from 1 mm to several mm, but may be less than 1 mm or 1 cm or more.

  When depth information is acquired, the amount of reflected light from each direction and the amount of time required for reflected light to reach when irradiating a light beam of a specific wavelength, for example, infrared light, when photographing a display shelf in the photographing device are measured. Obtain distance information (depth information) to the imaging target by measuring, or irradiate a light beam of a specific wavelength, for example, an infrared dot pattern, and distance information (depth information) from the reflection pattern to the imaging target In addition to the method of measuring the parallax, there is a method of using the parallax of the stereo camera, but the method is not limited thereto. If the depth information is given by the distance from the imaging device (one point on three dimensions), the depth information is converted into depth information from the imaging surface (plane) of the imaging device. . The input terminal 3 includes a device (depth detecting device) for acquiring depth information in this manner.

  The positional relationship between the captured image information and the depth information corresponds to each other. FIGS. 6A and 6B schematically show the correspondence between the captured image information and the depth map based on the depth information.

  Each means in the present invention is only logically distinguished in its function, and may have the same physical or practical area. The processing order in each means of the present invention can be changed as appropriate. Also, part of the processing may be omitted. For example, a normalization process described later can be omitted. In that case, it is possible to execute processing on image information that has not been subjected to normalization processing.

  The information processing system 1 includes a photographed image information input reception processing unit 20, a depth input reception processing unit 21, a normalization processing unit 22, a depth processing unit 23, an image recognition processing unit 24, an image information storage unit 25, and sample information. It has a storage unit 26 and a product identification information storage unit 27.

  The photographed image information input reception processing unit 20 receives input of image information (photographed image information) of a display shelf of a store photographed by the input terminal 3 and stores the image information in an image information storage unit 25 described later. The input terminal 3 may receive an input together with the photographed image information, the photographing date and time, store identification information such as a store name, image information identification information for identifying image information, and the like. FIG. 7 shows an example of photographed image information. In FIG. 7, there are three shelves on the display shelf, and this is the photographed image information in which products are displayed.

  The depth input reception processing unit 21 receives input of depth information and a depth map for the display shelf of the store photographed by the input terminal 3, and corresponds to the captured image information received by the captured image information input reception processing unit 20. In addition, it is stored in an image information storage unit 25 described later. FIG. 8 schematically shows an example of a depth map in which depth information is mapped. When depth information is received from the input terminal 3 instead of the depth map, the depth information is mapped to generate a depth map.

  The normalization processing unit 22 performs processing (correction processing) for correcting the photographed image information stored in the image information storage unit 25 so that the object to be photographed faces directly, for example, trapezoidal correction processing. Generate the executed normal image information. The trapezoidal correction process is a correction process performed so that the shelf of the display shelf shown in the captured image information is horizontal and the product tag for the product displayed there is vertical. Orientation is to transform the image information so that the optical axis of the lens of the imaging device is along the direction perpendicular to the plane to be imaged so as to be the same as when the image is taken from a sufficiently long distance. There is a correction process.

  The trapezoid correction processing executed by the normalization processing unit 22 receives an input of designation of four vertices in the captured image information, and executes the trapezoid correction processing using each of the vertices. The four vertices for receiving the designation may be the four vertices of the shelf of the display shelf or the four vertices of the shelf position of the display shelf. Alternatively, four vertices of a set of two or three shelves may be used. Any four points can be designated as the four vertices. FIG. 9 shows an example of photographed image information (normal image information) obtained by normalizing the photographed image information of FIG.

  The normalization processing unit 22 also performs the same normalization processing (the same conversion processing) as the normalization processing performed on the captured image information on the depth map. FIGS. 10A and 10B schematically show this. FIG. 10A shows the normal image information obtained by performing the normalization process on the captured image information of FIG. 6A. FIG. 10B is a diagram illustrating a depth map obtained by performing the same emplacement processing as the emplacement processing of FIG. 10A on the depth map of FIG. 6B. FIG. 11 schematically shows an example of a depth map in a state where the normalization processing has been performed on the depth map of FIG.

  By performing the same processing as that for the normal image information on the depth map as described above, the depth information can be specified with the same positional relationship as the normal image information.

  The depth processing unit 23 specifies an area (depth face area) where a product may be displayed based on the depth map. In addition, an area where a product tag is likely to be placed (a product tag placement area) is specified.

  The depth processing unit 23 includes an abnormal value removal processing unit 230, a surface type specification processing unit 231, and a depth face specification processing unit 232.

  The abnormal value removal processing unit 230 removes an abnormal value of the depth information in the depth map. The abnormal value can be determined, for example, by deviating from the average value of the depth information by a predetermined range or more. When it is determined as an abnormal value, the depth information in the mesh may be an average value of the depth information around the mesh.

  The surface type identification processing unit 231 detects a surface (including a curved surface as well as a flat surface) based on the depth information in the depth map, and detects the type of the detected surface, for example, the front surface of the shelf, the product tag surface, the product surface, and the like. Identify the rear plate surface of the shelf. FIG. 12 shows an example of the process of specifying the surface type in the surface type specifying processing unit 231. FIG. 12A is a diagram in which a surface detected based on depth information is imaged obliquely from above, and FIG. 12B is a diagram in which a surface detected based on depth information is imaged from a side surface. It is.

  The surface type identification processing unit 231 filters out high frequencies in order to exclude fine irregularities, and then forms a three-dimensional surface model based on the depth information, so as to be close to a vertical, facing, and a certain level. Surface, an upright cylindrical surface, and a surface of a polyhedron region that is entirely convex and distorted, and a position and range in a three-dimensional space are specified. For example, if the depth information is within a certain range and the normal is stably facing the front, there is a strong correlation in the vertical direction with respect to the depth information. In both the vertical direction and the convex direction, or if the normal is not stable, the surface is specified as a surface of a convexly distorted polyhedron region. Then, each detected surface is labeled. FIG. 12 shows a state in which five surfaces are detected and labeled with 1 to 5, respectively. Then, with respect to the detected surface, the distance from the foreground to the foreground (this is referred to as a level) is calculated in order from the foreground based on, for example, the average value of the depth information. Then, the front-rear relationship of each surface is specified based on the calculated level. In FIG. 12, the levels of the planes 1 and 2 are 0 cm, the levels of the planes 3 and 4 are 3 cm, and the level of the plane 5 is 20 cm. Then, the front and rear relations of the five planes are specified, with the frontmost plane being 1, 2, the next rearmost plane being 3, 4 and the rearmost plane being 5.

  The surface type specification processing unit 231 specifies the type of the surface using the context of the surface specified in this way. In other words, the front and the narrow horizontal plane in the depth map are the same as the front of the shelf and the product tag surface, the rear is the rear plate of the shelf, and the depth information is the same as the front of the shelf. Or, it is close to (a certain range, for example, within several cm), and a vertical rectangle in the hollow is specified as the surface of the product tag. Then, preferably, the other surface is specified as the product surface (the surface on which the product may be displayed). Therefore, in the example of FIG. 12, the surfaces 1 and 2 are specified as the front surface of the shelf and the product tag surface, the surfaces 3 and 4 as the product surface, and the surface 5 as the rear plate surface of the shelf.

  The depth face specification processing unit 232 performs a process of specifying a depth face area on the product surface specified by the surface type specification processing unit 231.

  The processing of specifying the depth face area in the depth face specification processing unit 232 is performed as follows.

  First, corresponding processing is executed based on the package type of the product whose input has been received at any timing. Package types include, but are not limited to, cans, bottles (including PET bottles; the same applies hereinafter), coexistence of cans and bottles, boxes, and hanging products.

  If the package type is a can or bottle, specify the area of the cylinder within a certain size range as the depth face area of the can or bottle. FIG. 13 and FIG. 14 schematically show this. FIG. 13 is an image diagram showing a case where the package type is a can, and FIG. 14 is an image diagram showing a case where the package type is a bottle.

  If the package type includes both cans and bottles, there is a cylinder within a certain size, the area without a small cylinder above is the depth face area of the can, and the cylinder is within a certain size. Yes, the area with a small cylinder above it is specified as the bottle depth face area.

  If the package type is a box, the area of the vertical rectangular plane within a certain size is specified as the depth face area of the box. This is schematically shown in FIG.

  If the package type is a hanging product, a rectangular flat or uneven surface area within a certain size and floating in the air is specified as a depth face region of the hanging product. This is schematically shown in FIG.

  The depth map specification processing unit may specify the depth face area without accepting the selection of the package type. In this case, it is determined whether or not the condition as the depth face area is satisfied by executing the determination process for each of the product surface area and each package type. If there is a face constituting the depth face area, the depth face area constituted by the face is specified, and the depth face area is determined based on a predetermined package type priority condition. The package type.

  The image recognition processing unit 24 is based on the photographed image information or the normal image information received by the photographed image information input reception processing unit 20 and the surface specified by the depth processing unit 23, and displays the product tag placement area on the display shelf. Recognize the product identification information described in the product tag in. Further, based on the photographed image information or the normal image information received by the photographed image information input reception processing unit 20 and the product surface or the depth face area specified by the depth processing unit 23, the photographed image information information or the normal face A process is executed for recognizing the product identification information of the product displayed on the shelf of the display shelf shown in the image information. The product identification information includes, but is not limited to, a product name and a JAN code assigned to the product. Any information may be used as long as it can identify the product.

  The image recognition processing unit 24 includes a product tag information specification processing unit 240 and a product identification information specification processing unit 241.

  The product tag information specification processing unit 240 cuts out a region (product tag region) of the photographed image information corresponding to the product tag surface region specified by the depth processing unit 23, and converts the information described in the product tag region into an OCR. A process specified by recognition or the like is executed. The OCR recognition can be performed for all or a part of the area specified as the product tag area. If it is performed on all the areas specified as the product tag areas, there is a high possibility that erroneous recognition will occur due to noise or the like. Therefore, it is preferable to limit the target area for OCR recognition. Various methods can be used to perform OCR recognition on the product tag area. For example, one of the following two methods can be used. Note that in this specification, extracting image information includes not only separating the image information from the image information as an actual processing target, but also specifying the image information of the area as a processing target range without separating the image information.

  As a first method, first, binarization processing is performed in the specified product tag area. As the binarization processing, one of a method using the lightness of the HSV color space, a method of Otsu, and a method of extracting red (a black and a red are frequently used for numbers and character strings of product tags) are used. There is a method of selecting, as candidates, those extracted by combining a plurality of them.

  Then, a labeling process is performed on the binarized image information. The labeling process is a process of assigning the same number (identification information) to continuous white or black pixels in binary image information, thereby forming continuous islands into one island (group). is there. Then, a rectangular area (box) including the island detected by the labeling process is generated, and the height, width, and baseline of the box are obtained. When generating a box, it is preferable, but not limited to, to generate a rectangle enclosed by the smallest, vertical and horizontal line segments surrounding the area labeled with the same number. When a box is generated, an island that does not satisfy a predetermined threshold height and width is not generated as a noise but is removed from the processing target without generating a box. For example, islands that are too small may be horizontal ruled lines or dust on the image, while islands that are too wide may be logos, which are removed as noise.

  In general, characters used in a product tag are often used in a font such as a bold Gothic font in which the outline of the entire character is clear. Therefore, even if the image information has some blur, the character group forming one character string can be detected as an island having the same height as the baseline.

  Then, the information-in-product-tag identification processing unit 240 merges adjacent boxes having a predetermined similarity to form a block. That is, a continuous box whose height coincides with the height of the base line within a predetermined range and whose height and width are within a certain threshold range is merged to form a block. At this time, small boxes between blocks to be merged are also collectively configured as one block. As a result, a cloud point, a semi-voice point, a hyphen, and the like are also taken into one block. A block is an area to be subjected to OCR recognition. Then, of the blocks, the block having the highest height is estimated as a price area (price area without tax), and OCR recognition is performed. OCR recognition is similarly performed for the other blocks. By performing the above-described processing, the character recognition processing can be performed with higher accuracy than when performing OCR recognition on a document portion freely laid out over a plurality of lines in the product tag area. 17 to 19 schematically show this processing. FIG. 17 shows binarized image information, and FIG. 18 shows a state in which a box (region indicated by a broken line) is generated. FIG. 19 is a diagram showing a state in which blocks (regions indicated by broken lines) have been generated.

  As described above, the information specification processing unit 240 within a product tag can perform OCR recognition on the information described in the product tag.

  The second method is a process of holding image information of a product tag as a template and comparing the image information used as a template with the image information of a product tag area. The image information of the product tag used as a template includes the product identification information such as the tax-excluded price, tax-included price, manufacturer name, product name, etc., and the position and size (height, width) of each character frame on which the rating is written. Is set. Then, the image information of the corresponding portion is cut out from the product tag area specified by the product tag specifying processing unit, and the OCR recognition processing is executed. At this time, product identification information such as price, manufacturer name, product name, etc., and the character type used by the rating (for example, whether it contains numbers, Roman characters, symbols, or Kana-Kanji characters) are defined as constraints. The accuracy of the OCR recognition processing can be improved.

  As described above, the in-product-tag information specification processing unit 240 may execute the process of checking the consistency of the read information. As the consistency check processing, it is preferable to perform two types of consistency check processing by dictionary matching and logical check.

  The process of checking the consistency by dictionary matching is executed, for example, as follows. The information processing system 1 stores a product dictionary (not shown) in which product identification information such as a product name of a product that may be displayed on a display shelf is associated with code information (for example, a JAN code) corresponding thereto. Zu). Then, character string similarity evaluation between a character string other than the character string read from the area indicating the price recognized by the information-in-product-tag processing unit 240 and the product identification information such as all the product names registered in the product dictionary I do. For example, an edit distance (Levenshtein distance), a Jaro-Winkler distance, or the like can be used.

  If the maximum value of the character string similarity calculated above is one, the product identification information such as the product name is determined as the final candidate. The allowable similarity is determined in advance for the length of the character string of the product identification information such as the product name as the final candidate, and if the similarity is equal to or greater than the allowable similarity, the product identification information such as the product name is identified. I do. If the allowable similarity has not been reached, the read character string is not determined. If there are a plurality of products having the maximum similarity, the read character string is not determined.

  The edit distance, which is one of the character string similarity evaluations, is a type of distance that indicates how different two character strings are. Specifically, the edit distance is determined by inserting, deleting, or replacing one character. The minimum number of steps required to transform one string into another. FIG. 20 shows an example of the product name described in the product tag, the product name with the smallest editing distance among the product names stored in the product dictionary as a result of OCR recognition, and the editing distance. FIG. 21 is a table showing the correspondence between edit distances that may be determined for each final candidate character number. In this specification, a case in which the processing is performed using the edit distance will be described. However, a character string similarity evaluation defined so that the substitution has a larger similarity than the insertion or deletion may be used.

  After obtaining the product identification information such as the product name of the minimum edit distance, a rating dictionary (not shown) is provided separately for the character string obtained by removing the relevant part of the product identification information such as the product name from the OCR recognition result. ), And the edit distance to each maker name stored in a maker name dictionary (not shown). For example, in the example of "350 ml of throat raw" in FIG. 20, "350 ml" matches with "350 ml" in the rated dictionary at an editing distance of 0, and identifies the character string of the rated part. Similarly, the editing distance is obtained for the maker name, and the character string at the minimum editing distance is identified as the maker name. Then, the character strings of the rated part and the manufacturer name part are removed from the OCR recognition result, and the character string at the shortest edit distance in the product name dictionary is obtained for the remaining character strings, and is the edit distance allowable? Determine whether or not. If the editing distance is within the allowable range, the product identification information such as the product name, the manufacturer name, and the rating are determined. By performing such processing, it is possible to appropriately determine even when the rating and the manufacturer name are included in the product identification information such as the product name. The rating dictionary is a dictionary that stores ratings (capacity, etc.) of products that may be displayed on the display shelf. The maker name dictionary is a dictionary that stores maker names of products that are likely to be displayed on display shelves.

  In order to allow the user to finally judge and approve the recognition result by the information identification processing unit 240 in the product tag, when performing display for accepting an input such as selection of the recognition result, a confirmed character string, an unconfirmed character string Is preferably displayed in a manner that can be specified. For example, the colors of the determined character string and the undetermined character string are divided, and the determined character string and / or the undetermined character string are attached with information indicating the determined or undetermined character string. For an undetermined character string, if there is a plurality of product identification information such as a product name having the minimum editing distance, each product identification information may be displayed.

  The logical check (the process of determining the logical consistency of the recognized information) in the product tag information specification processing unit 240 can be performed as follows. For example, when two prices are read as price areas, one is the tax-excluded price and one is the tax-included price, but if the tax-excluded price is higher than the tax-included price, they are reversed and recognized. Let it. If the value obtained by multiplying the tax-excluded price by the consumption tax rate does not match the tax-included price, it is determined that one or both of the tax-recognitions are erroneously recognized. Further, when the product identification information such as the product name is recognized, it is determined whether the product or the product category falls within the normal price range. Further, it may be determined whether the correspondence between the product identification information such as the product name, the maker name, and the rating match.

  As described above, by checking the consistency of the information in the product tag, the information described in the product tag in the captured image information can be determined. The information determined in this way can be output in, for example, a table format.

  The product identification information specifying processing unit 241 cuts out the area of the normal image information (image face area) corresponding to the depth face area specified by the depth processing unit 23. This is schematically shown in FIGS. 32 and 33. FIG. FIG. 32A is a diagram schematically showing a state where an image face area corresponding to a depth face area is cut out when the package type of a product is a can. FIG. 32B is a diagram schematically illustrating a state where an image face area corresponding to a depth face area is cut out when the package type of the product is a bottle. FIG. 33A is a diagram schematically showing a state where an image face area corresponding to a depth face area is cut out when the package type of the product is a box. FIG. 33B is a diagram schematically illustrating a state where an image face area corresponding to a depth face area is cut out when the package type of the product is a hanging product. Note that it is preferable to set in advance a rule for clipping as a template depending on the package type of the product. For example, when the package type of the product is a bottle, a rule may be set to cut out from the bottle body to the position of the cap as one.

  By performing the matching process between the image face area cut out from the normal image information and the sample information stored in the sample information storage unit 26 described later, the product identification of the product displayed on the shelf of the display shelf is performed. Identify information. Specifically, first, the similarity between the image information of the image face area to be processed and the sample information stored in the sample information storage unit 26 is determined, and the product identification corresponding to the sample information having the highest similarity is determined. The information is specified, and if the specified similarity is equal to or greater than a predetermined threshold, the information is specified as the product identification information of the product displayed on the face constituted by the coordinates of the image face area.

  It should be noted that a correction input by a person in charge may be received for the automatically specified image face area. Further, the input of the position of the image face area from the person in charge may be received. The coordinates of the specified image face area are as follows: shooting date and time information, store information, image information identification information of shot image information, image identification information of normal image information, depth information and depth map, face identification for face identification Manage by associating with information. The coordinates indicating the image face area are the coordinates of each point for specifying the area. When the image face area is rectangular, for example, four points in the normal image information, upper right and lower left, upper left and right The coordinates of the lower two points may be used. When the image face area is a polygon, the coordinates of each vertex may be used. In addition, the coordinates are relative coordinates with respect to a predetermined location (for example, an upper left vertex of the display shelf) in the image information such as a display shelf in the normal image information.

  The following processing is performed to determine the similarity between the image face area and the specimen information in the normal image information. First, in the processing up to the processing for specifying the product identification information in the product identification information specifying processing unit 241, the direction of the image face area in the shelf of the normal image information and the direction of the sample information are the same (they are not turned over or inverted). ), And their sizes are substantially the same (when the sizes of the image information are different from each other over a predetermined range, the respective sizes are within the predetermined range before the similarity determination). Adjust the size so that it becomes).

  The product identification information specifying processing unit 241 determines the similarity between the image face area in the normal image information and the sample information based on the image feature amount (for example, local feature amount) of the image face area in the normal image information. A feature point and a feature point based on an image feature amount (for example, a local feature amount) of the sample information are respectively extracted. Then, a pair having the highest similarity between the feature point of the image face area and the feature point of the sample information is detected, and the difference between the coordinates of the corresponding points is calculated. Then, an average value of the differences is obtained. The average value of the difference indicates the average moving amount of the entire image information of the image face area and the sample information. Then, the coordinate difference between all feature point pairs is compared with the average coordinate difference, and pairs having a large degree of deviation are excluded. Then, similarity is ranked based on the number of remaining corresponding points.

  The similarity between the image face area of the normal image information and the specimen information can be calculated by the above method. In addition, in order to improve the accuracy, an EMD (Earth Movers Distance) between the color histograms may be obtained and used as a similarity measure. This makes it possible to compare the similarity relatively strong to environmental changes such as the brightness information of the captured image information, and to specify with high accuracy.

  As a similarity determination, an EMD between signatures (a set of image feature amounts and weights) of image information of each image face area may be obtained and used as a similarity measure. As the signature image feature amount, for example, a frequency distribution of the image face area in the HSV color space is obtained, and hue and saturation are grouped to obtain an image feature amount based on the number of features and the area in the HSV color space. Can be. The reason why the hue and the saturation are grouped is to reduce the dependence on the lightness so as not to be largely influenced by the photographing conditions.

  Further, in order to speed up the processing, similarity such as L2 distance between image features such as a color correlogram and a color histogram of image information in an appropriate color space can be used instead of the signature and the EMD.

  The determination of similarity is not limited to the above. The identified product identification information is associated with shooting date / time information, store information, image information identification information of shot image information, image identification information of normal image information, and face identification information for identifying a face, and a product identification information storage unit. 27.

  The image information storage unit 25 stores the photographed image information, depth information and depth map, photographing date and time, store identification information, image information identification information, etc. received from the input terminal 3 in association with each other. The captured image information may be any image information to be processed by the present invention. In general, it is difficult to take a photographing object with the photographing object directly facing when simply photographing. Therefore, it is necessary to execute a correction process for correcting the photographing object to the facing state, for example, a trapezoidal correction process. Is good. When one display shelf is photographed with a plurality of images, image information obtained by synthesizing it as one image information is also included. The image information after the distortion correction processing has been executed is also included in the captured image information.

  The specimen information storage unit 26 stores specimen information for identifying which commodity is displayed on the shelf of the display shelf shown in the normal image information. The sample information is image information obtained by photographing a product that may be displayed on a display shelf from a plurality of angles such as up, down, left, right, and oblique. FIG. 22 shows an example of the sample information stored in the sample information storage unit 26. FIG. 22 shows a case where canned beer is photographed from various angles as specimen information, but is not limited to canned beer. The sample information storage unit 26 stores sample information and product identification information in association with each other.

  The sample information storage unit 26 stores information necessary for calculating similarity extracted from the sample image information together with the sample image information of the product or instead of the sample image information, such as an image feature amount and its position. May be stored. The sample information may be information necessary for calculating the similarity in addition to the sample image information. In this case, the product identification information identification processing unit 241 does not have to calculate the image feature amount for each sample information every time the matching process is performed between the image face area and the sample information, thereby shortening the calculation time. Can be.

  The product identification information storage unit 27 stores the product identification information of the product displayed on each face of the display shelf. For example, image date and time information, store information, image information identification information of photographed image information, image identification information of normal image information, identification information of depth information and depth map, and face identification are associated with product identification information. In the product identification information storage unit 27 in association with the face identification information. FIG. 23 schematically shows an image face area in the photographed image information of the display shelf.

  Next, an example of a processing process of the information processing system 1 of the present invention will be described with reference to a flowchart of FIG.

  A depth map in which the photographed image information of the display shelves of the store and the depth information are mapped is input from the input terminal 3 and the photographed image information input reception processing section 20 and the depth input reception processing section 21 of the management terminal 2. Accepts the input (S100). In addition, the input of the photographing date and time, store identification information, and image information identification information of the photographed image information is received. FIG. 24 shows an example of captured image information. FIG. 8 shows an example of the depth map. Then, the photographed image information input reception processing unit 20 and the depth input reception processing unit 21 perform the input of received photographed image information, depth information and depth map, photographing date and time, store identification information, and image information identification of photographed image information. The information is stored in the image information storage unit 25 in association with the information.

  When a predetermined operation input is received at the management terminal 2, the normalization processing unit 22 extracts the photographed image information stored in the image information storage unit 25, and uses the vertex for performing the normalization processing such as the trapezoidal correction processing. The input of four points at a certain shelf position (display shelf position) is received, and a normalization process is executed (S110). FIG. 25 shows an example of photographed image information (normal image information) on which the normalization processing has been executed in this manner.

  The normalization processing unit 22 performs the same normalization processing on the depth map corresponding to the captured image information subjected to the normalization processing as described above (S110). Thus, the normal image information and the depth map can be associated with the same positional relationship.

  As described above, after performing the normalization process on the captured image information and the depth map, the abnormal value removal processing unit 230 in the depth processing unit 23 determines the depth that deviates from the average value of the depth information by a predetermined range or more. The information is removed as an abnormal value (S120).

  Then, the surface type identification processing unit 231 in the depth processing unit 23 detects the surface based on the depth information of the depth map, and identifies the surface type (S130).

  That is, the surface type specifying processing unit 231 forms a three-dimensional surface model based on the depth information after performing low-pass filtering, and specifies a plane, a cylindrical surface, and a surface of a convexly distorted polyhedral region. . Then, each detected surface is labeled, and a level is calculated based on the depth information of each surface. Then, based on the calculated level, the context of each surface is specified. Using the front-back relation of the surfaces specified in this way, the type of each surface is specified as a front surface of the shelf, a product tag surface, a rear plate surface of the shelf, and a product surface.

  When each surface is specified in this way, the depth face specification processing unit 232 specifies the type of the surface based on the package type of the product displayed on the shelf of the display shelf, which has been received by the management terminal 2 (S140). The processing for specifying the depth face area is performed on the product surface specified by the processing unit 231 (S150).

  Then, when the depth face specification processing unit 232 specifies the depth face area for the product surface, the product identification information specification processing unit 241 of the image recognition processing unit 24 determines the depth image area corresponding to the specified depth face area. The region is cut out as an image face region (S160), and by performing matching processing with the sample information stored in the sample information storage unit 26, the product identification information of the product in the image face region is specified (S170). By performing this for each image face area, the product identification information of the products displayed on the shelf of the display shelf can be specified.

  That is, the similarity between the image face area in the normal image information to be processed and the sample information stored in the sample information storage unit 26 is determined, and the product identification information corresponding to the sample information having the highest similarity is identified. If the specified similarity is equal to or greater than a predetermined threshold, the similarity is specified as the product identification information of the product displayed in the image face area including the coordinates. Then, the product identification information specifying processing unit 241 converts the specified product identification information into shooting date / time information, store information, image information identification information of shot image information, image identification information of normal image information, depth information and depth map. , Are stored in the product identification information storage unit 27 in association with face identification information for identifying faces.

  By executing the above-described processing, it is possible to appropriately set an area in which the product is displayed in the captured image information, so that it is possible to improve accuracy in comparing the similarity with the sample information. .

  On the other hand, when the product tag surface is specified by the surface type specification processing unit 231, the in-product tag information specification processing unit 240 of the image recognition processing unit 24 determines the area of the normal image information (product tag area) corresponding to the specified product tag surface. ) Is cut out (S180), and processing for specifying information described in the product tag area by OCR recognition or the like is executed (S190).

  That is, OCR recognition is performed on the product tag area of the normal image information to be processed, and processing for specifying information such as product identification information and price described on the product tag is executed. Then, the information described in the specified product tag is processed by the information specification processing unit 240 within the product tag to obtain the shooting date / time information, the store information, the image information identification information of the shot image information, the image identification information of the normal image information, and the depth information. The information is stored in the product identification information storage unit 27 in association with the image data, the depth map, and the face identification information for identifying the face. Note that the product identification information is also recognized by the product identification information identification processing unit 241. Therefore, if the two recognition results match, the registration is performed as it is, and if the recognition results are different, an error occurs. Display may be performed. Further, priorities may be set, and any of the recognition results may be registered with priority.

  By executing the above-described processing, the area of the product tag can be appropriately set in the captured image information, so that the recognition accuracy of the information described on the product tag by OCR recognition can be improved. .

  In the above-described first embodiment, the product identification information specifying processing unit 241 determines whether or not the image face area of the normal image information corresponding to the depth face area specified by the depth face specifying processing unit 232 and the sample information are similar. Although the processing has been described, instead of or in addition to this, the corresponding area is specified by the normal image information based on the area of the product surface specified by the surface type specifying processing unit 231, and the image face You may comprise so that the area | region may be specified and the determination process of similarity with sample information may be performed.

  An example of the processing function of the image recognition processing unit 24 in the present embodiment is shown in the block diagram of FIG. The image recognition processing unit 24 of the present embodiment further includes an image face specification processing unit 242.

  The image face specification processing unit 242 specifies a corresponding area (product display area) in the normal image information based on the product surface area specified by the depth face specification processing unit 232. The image face area specifying processing in the image face specifying processing unit 242 is performed in the product display area of the normal image information corresponding to the product surface specified by the depth face specifying processing unit 232, in a narrow and narrow area generated between the products. The image face area is specified by specifying a shadow, specifying a repetitive pattern of an image, specifying a step on the upper side of a package, specifying a break position based on constraints such as the same product width, and the like.

  As the process of specifying the image face area, an arbitrary method can be adopted depending on the category of the product and the form of the product, and is not limited to the above method.

  In general, one shelf often contains similar types of products. In that case, the size of the product is almost the same size. Therefore, when one image face area is specified in the product display area, the image face area in the product display area may be specified by repeatedly setting the area in the shelf.

  The second and subsequent image face area specification processing by the image face specification processing unit 242 is performed for the product display area of the same shelf on the same display shelf using the image specified by the previous (N-1) -th normal image information. The coordinates of the face area may be the coordinates of the image face area specified by the current (N-th) normal image information.

  The product identification information specifying processing unit 241 determines whether the image face area specified by the image face specifying processing unit 242 is similar to the sample information stored in the sample information storage unit 26, and thereby determines whether the image face area is in the image face area. Specify the product identification information of the product.

  As in the present embodiment, the process of specifying the image face area in the normal image information using the product surface specified based on the depth map in the image face specification processing unit 242 is performed by, for example, the depth face specification processing unit 232. May be executed when the depth face area cannot be specified by the processing described above, or may coexist with the processing of the depth face specification processing unit 232. As a case where the processes coexist, for example, there is a case where the depth face area specified by the depth face specification processing unit 232 is equal to or larger than a predetermined width. In this case, the processing of the image face specification processing unit 242 may be executed over the entire product surface (the entire product display area), or the image face specification processing unit may be executed on a depth face area having a predetermined width or more. 242 may be executed.

  In the above-described first embodiment, the processing for directly specifying the area (product tag area) corresponding to the product tag surface specified by the depth processing unit 23 from the normal image information is executed. However, in many cases, the product tag is attached to the front of the shelf so as to be slightly inclined upward so as to be easily recognized by the purchaser of the product. In this case, the product tag is slightly inclined upward at the front of the shelf only if the photographed image information is subjected to the normalization processing. Therefore, there is a possibility that the recognition accuracy when performing the OCR recognition process in the product tag information specification processing unit 240 may be affected. Therefore, in the present embodiment, a case will be described in which an area (product tag area) corresponding to the product tag surface in the normal image information is further set to correct the product tag so that the product tag faces directly.

  An example of the processing function of the image recognition processing unit 24 in the present embodiment is shown in the block diagram of FIG. The image recognition processing unit 24 of this embodiment further includes a product tag area normalization processing unit 243.

  The product tag area normalization processing unit 243 cuts out the area (product tag area) of the normal image information corresponding to the product tag surface specified by the surface type specifying processing unit 231 and normalizes the image information of the product tag area. Execute the normalization process. FIG. 28 is a diagram showing a state where a product tag area is cut out from the normal image information of FIG. FIG. 28A shows the product tag area in the upper part of FIG. 25, and FIG. 28B shows the product tag area in the lower part of FIG. In FIG. 28, the one-dot chain line indicates the area of the normal image information corresponding to the front of the shelf, and the solid line indicates the image information of the product tag area. In the following description, the normalization process is performed on the product tag surface, but the product tag is often located on the front of the shelf. Therefore, a normalization process may be performed on the front surface of the shelf and / or the product tag area. In that case, the product tag area may be replaced with a normal image information area corresponding to the front of the shelf.

  The product tag area normalization processing unit 243 performs the following processing to normalize the product tag area in the normal image information. That is, edge detection is performed in the product tag area in the normal image information, and near a right or left end, a nearly vertical contour line having a certain length or more (for example, a vertical (90 From (degree), a contour line within a predetermined range of angle is specified. In addition, it is preferable to extract the contour lines near the left and right ends, but the present invention is not limited to this. In the case of the product tag area of the normal image information in FIG. 28, L1 to L4 are specified respectively. Note that the specified contour lines L1 to L4 are not actually drawn in the product tag area. Then, trapezoid correction processing is performed on the product tag area such that L1 and L2 in FIG. 28A and L3 and L4 in FIG. 28B respectively become vertical lines.

  By executing such a process, the image information of the product tag area is normalized, and the image information of the normalized product tag area shown in FIG. 29 can be obtained. By executing the process of the product tag area normalization processing unit 243, the accuracy of OCR recognition by the product tag information specification processing unit 240 can be improved.

  Then, the in-product-tag information specifying processing unit 240 executes the OCR recognition process in the first embodiment on the product tag area subjected to the normalization process.

  In the above-described third embodiment, an example in which the normalization process is performed on the product tag area in order to correct a state in which the product tag is attached to the front of the shelf in a slightly upwardly inclined state has been described. In the present embodiment, as another normalization processing, a product which is an area of the normal image information corresponding to the product tag surface such that the normal line of the product tag surface specified by the surface type specifying processing unit 231 faces the viewpoint direction. By rotating the tag area in the three-dimensional space, the normalization processing can be executed more accurately.

  FIG. 30 schematically shows a product tag area, a product surface, and the like. FIG. 30A is a diagram in which the product tag surface and the product surface are imaged obliquely from above, and FIG. 30B is a diagram in which the product tag surface and the product surface are imaged from the side.

  The product tag area normalization processing unit 243 calculates a normal vector of the product tag surface specified by the surface type specifying processing unit 231, and the normal vector is a depth for acquiring depth information in the imaging device. A conversion is specified that rotates the product tag surface in a three-dimensional space vertically and / or horizontally so as to face a detection device, for example, an infrared irradiation device or a stereo camera. Then, conversion for rotating in the specified vertical and / or horizontal direction is performed on the product tag area in the normal image information corresponding to the product tag surface specified by the surface type specifying processing unit 231. This is schematically shown in FIG. FIG. 31A is a view from the front, and FIG. 31B is a view from the side.

  By performing the normalization processing as in the present embodiment, even when the contour of the product tag surface is irregular, or when the product tag is installed at the front of the display shelf in the left and right direction, it is more effective. It can be accurately positioned.

  Note that the product tag surface is specified by the depth processing unit 23 in the product identification information specification processing unit 241 based on the surface detected by the surface type specification processing unit 231 and the context of the level of the surface. A surface on the front side (a surface with a lower level) than the surface in the specified depth face area may be specified as the product tag surface. For example, the surface in the depth face area specified in the previous (N-1st) process may be used to specify the product tag surface and the product surface in the current (Nth) process.

  The information processing system according to the present invention may be configured so that a missing item of a product displayed on the display shelf can be specified. In this case, in the depth processing unit 23, the surface is a portion of the surface whose level is later (larger) by a predetermined value or a predetermined ratio than the product surface detected by the surface type specifying processing unit 231 using the depth information. , An area in which the product surface and the horizontal direction are aligned is determined as “sky”. Then, the image recognition processing unit 24 determines a region in the normal image information corresponding to the region determined to be “empty” by the depth processing unit 23 as a region where no merchandise is displayed on the display shelf.

  By executing such a process, it is possible to determine that the display shelf has a missing product ("empty").

  Usually, in order to determine “sky” only from image information, the position where the product should be originally is specified based on the darker position than the surroundings, the same color as the background color, etc. It may be difficult to maintain accuracy depending on the angle of the camera. However, as in the present embodiment, the level of the product surface is specified using the depth information corresponding to the captured image information or the normal image information, and the area that is receded to a certain extent from the specified product surface is defined as “empty”. , It is possible to make a determination with higher accuracy than to determine “sky” from only image information.

  The processes of the above-described first to fifth embodiments can be appropriately combined. In addition, the respective processes are not limited to the order described in the specification of the present invention, and can be appropriately changed as long as the object is achieved. In addition, each processing in the depth processing unit 23 and the image recognition processing unit 24 is performed on the fixed image information that has been subjected to the normalization processing on the captured image information. May be. In that case, the normal image information may be replaced with the photographed image information. Further, the photographed image information and the normal image information are collectively referred to simply as image information.

  Further, the information processing system of the present invention is effective in the case of image information obtained by photographing a display shelf of a store, but is not limited thereto. That is, when a certain photographing target is photographed, it can be widely used when specifying a region where the desired target is photographed from the photographed image information. By executing each of the above-described processes, a region where a desired target object is captured can be specified with high accuracy from the image information.

  By using the information processing system 1 of the present invention, a predetermined area can be specified with high accuracy from image information obtained by photographing an object to be photographed. In particular, it is possible to accurately specify a product display portion and a product tag area in an image of a display shelf.

1: information processing system 2: management terminal 3: input terminal 20: photographed image information input reception processing unit 21: depth input reception processing unit 22: normalization processing unit 23: depth processing unit 24: image recognition processing unit 25 : Image information storage unit 26: Specimen information storage unit 27: Product identification information storage unit 70: Arithmetic device 71: Storage device 72: Display device 73: Input device 74: Communication device 230: Abnormal value removal processing unit 231: Surface type identification Processing unit 232: depth face specification processing unit 240: product tag information specification processing unit 241: product identification information specification processing unit 242: image face specification processing unit 243: product tag area normalization processing unit

Claims (5)

An information processing system using image information and depth information,
The information processing system comprises:
A depth processing unit that specifies a product surface or a depth face area using the depth information;
Using said identified product surface or depth face region, and have a, and an image recognition processing unit that specifies an image face area in the image information,
The depth processing unit comprises:
Detecting a surface based on the depth information;
Identify the type of surface using the context of the detected surface,
On the side where the type of the side is a product side, the depth face area is specified by comparing with the determination condition of the depth face area for each package type.
An information processing system, comprising: The judgment condition is
If the package type is a can or bottle, specify the cylindrical area within a certain size range as the depth face area of the can or bottle;
If the package type includes both cans and bottles, there is a cylinder within a certain size, and the area without a small cylinder above is the depth face area of the can, within the certain size. The area with the cylinder at the top and the small cylinder above is identified as the bottle depth face area,
If the package type is a box, the area of a vertical rectangular plane within a certain size is specified as the depth face area of the box.
Alternatively, when the package type is a hanging product, a rectangular flat surface or an uneven surface area within a certain size range is specified as a depth face region of the hanging product.
Including one or more of
The information processing system according to claim 1 , wherein: An information processing system using image information and depth information,
The information processing system comprises:
A depth processing unit that specifies a product surface or a depth face area using the depth information;
Using said identified product surface or depth face region, and have a, and an image recognition processing unit that specifies an image face area in the image information,
The image recognition processing unit includes:
In the depth processing unit, if the product surface or the depth face area could not be specified, or if the depth face area was equal to or larger than a predetermined width, the image information indicates that a product Identify the image face area based on at least one of shading, image repetition pattern, step on top of package, and product width,
By determining the similarity between the specified image face area and the specimen information, the product identification information of the product displayed in the image face area is specified.
An information processing system, comprising: Computer
A depth processing unit that specifies a product surface or a depth face area using the depth information;
An information processing program that functions as an image recognition processing unit that specifies an image face area in image information using the specified product surface or depth face area,
The depth processing unit comprises:
Detecting a surface based on the depth information;
Identify the type of surface using the context of the detected surface,
On the side where the type of the side is a product side, the depth face area is specified by comparing with the determination condition of the depth face area for each package type.
An information processing program characterized by the following. Computer
A depth processing unit that specifies a product surface or a depth face area using the depth information;
An information processing program that functions as an image recognition processing unit that specifies an image face area in image information using the specified product surface or depth face area,
The image recognition processing unit includes:
In the depth processing unit, if the product surface or the depth face area could not be specified, or if the depth face area was equal to or larger than a predetermined width, the image information indicates that a product Identify the image face area based on at least one of shading, image repetition pattern, step on top of package, and product width,
By determining the similarity between the specified image face area and the specimen information, the product identification information of the product displayed in the image face area is specified.
An information processing program characterized by the following.