JP2019125394A - System, server, program and method for object acquisition confirmation - Google Patents

Abstract

To provide a system, a server, a program and a method for object acquisition confirmation.SOLUTION: The system includes portable terminal devices and a server for confirming that any kind of an object among a plurality of kinds of objects is acquired. In the system, the portable terminal device determines whether any kind of an object among the plurality of kinds of objects is picked up in an image of a space imaged by an imaging device of the portable terminal device on the basis of a local feature amount about a shape for detecting the plurality of kinds of objects, acquires the image determined to have imaged the object as an acquisition object image, and transits the image to the server, and the server receives the transmitted acquisition object image, determines a feature correlation amount of the acquisition object image to the object on the basis of an image feature amount of one kind of an object among the plurality of kinds of objects, and determines whether the acquisition object image is an image obtained by imaging the one kind of an object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a system, a server, a program and a method for confirming that a user has acquired an object.

  There is known a check-in service for providing a user such as an online game with a user visiting a predetermined place such as a store. According to Patent Document 1, a notice stored in a portable terminal storage unit when providing a service for giving a user such as an online game privilege to a user of a portable terminal visiting a place where a predetermined notice is posted When it is determined based on the reference image that is an image of the message that the message is captured within the imaging area, the image of the message and its periphery are imaged, and this is already authenticated as a normalized image that is normalized. A system is disclosed that verifies a user's visit based on its similarity compared to an authenticated image stored on a server.

Patent No. 6181893

  However, the system is for confirming that a predetermined place has been visited, and it is an object of privilege provision for a check-in service for confirming a user who has acquired by purchasing a product etc. and having an advantage. It is not always suitable for accurately identifying the type of an object such as a commodity.

  By attaching a two-dimensional code identifying the type of product to a product package or the like, it is possible to specify the product type of the captured image. However, in order to identify the type of a product by a two-dimensional code, a two-dimensional code different for each product must be displayed on a packaging sheet or the like. Since changing the design of the existing packaging sheet and printing the two-dimensional code greatly increases the unit price, it is difficult to realize this for an event or the like with a limited period. Moreover, when pasting a two-dimensional code for identifying each product etc. to existing packaging paper, the unit price is greatly increased for the work, and it is required to change the manufacturing / distribution route of the product. It is not target.

  The present invention has been made in view of the above problems, and has the following features. That is, a system according to an embodiment of the present invention is a system including a portable terminal device and a server for confirming that one of a plurality of types of objects has been acquired. The portable terminal device is configured to generate an image of a space captured by the imaging device of the portable terminal device based on the local feature amount of the shape for detecting the plurality of types of objects. It is determined whether or not an object of a type is imaged, and if it is determined that an object of any type is imaged, an image determined to be imaged of the object is an acquisition object Captured as an image, and transmitted the acquisition target object image to the server, the server receives the transmitted acquisition target object image, and the image characteristic of one type of target object among the plurality of types of target objects. The feature correlation amount of the acquisition object image with respect to the one type of object is determined based on the amount, and the acquisition object image is the one type of object based on the determined feature correlation amount. It is determined whether or not the image is captured.

  The image feature amount of the one type of object includes an inherent feature amount and a non-unique feature amount, and the inherent feature amount includes a characteristic color value indicating a color that is a feature of the one type of object; The non-unique feature amount includes non-feature color values indicating a color that is not a feature of the one type of object, and the feature correlation amount is an image feature of the one type of object in the acquired object image. An amount of features determined to indicate a color that is characteristic of said one type of object based on an amount and an amount of non-features determined to indicate an uncharacteristic color of said one type of object And determining whether the acquisition object image is an image obtained by imaging the one type of object, depending on whether or not the feature correlation amount is within a predetermined range. May be

  The non-unique feature amount is not included in the color value indicating the color common to the plurality of types of objects and the unique feature amount of the one type of object but is included in the unique feature amounts of other types of objects It may include at least one of the color values.

  The image feature amount of the one type of object includes a histogram of the shape included in the reference image of the type of object, and the feature correlation amount is included in the histogram of the reference image and the acquisition object image It may be determined based on the difference between the shape and the histogram.

  It is determined whether or not an object of any of a plurality of types of objects is imaged in an image of a space imaged by an imaging device of the portable terminal device. It may be determined whether or not the image is captured in the area of.

  One type of the plurality of types of objects may be designated by the user in advance via the mobile terminal device.

  Determining whether or not the acquisition object image is an image obtained by imaging the one type of object is performed for each of the plurality of types of objects, and an image obtained by imaging an object having the highest correlation It may be determined that

  The local feature may be a common local feature of the plurality of types of objects or may be a unique local feature of each of the plurality of types of objects.

  Further, the server in one embodiment of the present invention is a server used in a system for confirming that one of a plurality of types of objects has been acquired, and the server is used by a portable terminal device. An acquisition target object image determined to be imaged that has been imaged is received from the portable terminal device, and the image characteristic amount of one type of target object among the plurality of types of target objects is received. The feature correlation amount of the acquisition object image for one type of object is determined, and the acquisition object image is an image obtained by imaging the one type of object based on the determined feature correlation amount. It is determined whether or not.

  Furthermore, a set of programs according to an embodiment of the present invention is a set of programs for confirming that one of a plurality of types of objects has been acquired, and one or more In the computer, based on the local feature amount of the shape for detecting the plurality of types of objects, the image of the space captured by the imaging device of the portable terminal device is one of the types of the plurality of types of objects A step of determining whether an object is being imaged or not, and when it is determined that an object of any type is being imaged, an image determined to be imaging of the object is obtained as an object of acquisition Capturing the image as an image, and determining a feature correlation amount of the acquisition object image with respect to the one type of object based on an image feature amount of the one type of the plurality of types of objects , Based on the feature amount of correlation the determined, to execute the steps determines whether the acquired object image is an image obtained by capturing the one type of object.

  A program according to an embodiment of the present invention is a program for confirming that an object of any type among a plurality of types of objects has been acquired, and an object imaged by a mobile terminal device in a server An acquisition object image determined to be imaged is received from the portable terminal device, and based on the image feature amount of the one type of the plurality of types of the object, the one type of The feature correlation amount of the acquisition target object image with respect to the target object is determined, and based on the determined feature correlation amount, whether or not the acquisition target object image is an image obtained by imaging the one type of target object Make a decision.

  A method according to an embodiment of the present invention is a method for confirming that one of a plurality of types of objects has been acquired, and one or more computers are provided with the plurality of types of objects. Whether or not an object of any of a plurality of types of objects is imaged in the image of the space captured by the imaging device of the mobile terminal device based on the local feature amount of the shape for detecting the object Determining whether an object of any type is captured, capturing an image determined to be capturing the object as an acquisition object image, and Determining a feature correlation amount of the acquired object image for the one type of object based on an image feature amount of one type of the object, and the determined feature correlation Based on the amount There are, to execute the steps determines whether the acquired object image is an image obtained by imaging an object of the one type.

  A method according to an embodiment of the present invention is a method for confirming that an object of any type among a plurality of types of objects has been acquired, and an object captured by a mobile terminal device in a server. An acquisition object image determined to be imaged is received from the portable terminal device, and based on the image feature amount of the one type of the plurality of types of the object, the one type of The feature correlation amount of the acquisition target object image with respect to the target object is determined, and based on the determined feature correlation amount, whether or not the acquisition target object image is an image obtained by imaging the one type of target object Make a decision.

  According to the present invention, it is possible to inexpensively realize a system for identifying the type of an object acquired by the user from among objects such as a plurality of types of goods and confirming that the user has acquired the object. To be possible.

1 is an overall configuration of a system according to an embodiment of the present invention. It is a hardware block diagram of the portable terminal device which concerns on one Embodiment of this invention. It is a hardware block diagram of the server which concerns on one Embodiment of this invention. It is a functional block diagram of the portable terminal device concerning one embodiment of the present invention. It is a functional block diagram of the server concerning one embodiment of the present invention. It is a flowchart which shows the information processing which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing of the portable terminal device which concerns on one Embodiment of this invention. It is a figure which shows the example of a screen display which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing of the server which concerns on one Embodiment of this invention. FIG. 6 is a diagram visualizing features and non-features of an object according to an embodiment of the present invention. It is a figure which shows the concept of the window which concerns on one Embodiment of this invention. It is the figure which showed typically the shape feature-value extraction method which concerns on one Embodiment of this invention.

  Hereinafter, one embodiment of the present invention will be described with reference to the drawings. The acquisition confirmation system 100 for an object according to the present embodiment can be realized by a system including one or more mobile terminal devices 120 and a server 130 connected via a network 110, as shown in FIG. The object may be anything as long as it can confirm that the user has acquired it. In the present embodiment, when it is confirmed that the product to be checked in service is purchased and acquired by the user of the online game at the store, the item in the online game corresponding to the product is a reward Granted as The object does not have to be a product. For example, the card may be distributed free of charge at an event.

  The portable terminal device 120 in the present embodiment is an apparatus for capturing an image of an object and transmitting it to the server 130 via the network 110, and, for example, a portable type such as a portable telephone, a smart phone, a tablet terminal, a portable game machine, and a digital camera. It can be an apparatus. The portable terminal device 120 includes a processing device 201, a display device 202, an input device 203, an imaging device 204, a position information acquisition device 205, a storage device 206, a communication device 207, and a bus 208. The storage device 206 stores the program 209.

  The processing device 201 executes an application such as a web browser based on the program 209, input data from the input device 203, imaging data from the imaging device 204, data received from the position information acquisition device 205 or the communication device 207, etc. Perform various processing such as image processing. The processing device 201 includes a processor that controls each device included in the mobile terminal device 120, and performs various processes using a register included in the processor and the storage device 206 as a work area. Each of these components is connected by a bus 208, but may be individually connected as needed.

  The display device (display) 202 displays an application screen and an image captured by the imaging device 204 according to the control of the processing device 201. A liquid crystal display is preferable, but a display using an organic EL, a plasma display, or the like may be used.

  The input device 203 has a function of receiving an input from the user, such as a touch panel, a touch pad, and an input button. When the portable terminal device 120 includes a touch panel as the input device 203, the touch panel also functions as the display device 202, and the display device 202 and the input device 203 are integrated. The display device 202 and the input device 203 may be separate forms disposed at different positions. The imaging device 204 captures a still image or a moving image of a real space area. The position information acquisition device 205 acquires information indicating the current position of the mobile terminal device, and can be a device that specifies the position based on, for example, a GPS sensor or the SSID of WiFi.

  The storage device 206 includes a hard disk, a main memory, and a buffer memory. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. For example, when the portable terminal device 120 is a smart phone, it includes a ROM and a RAM. The storage device 206 stores a program 209 and various data that can be referred to with the execution of the program. The program 209 includes programs and data for an operating system or any application requiring player input, such as a video game, a web browser, and the like.

  The communication device 207 performs wireless communication such as mobile communication and wireless LAN, and wired communication using an Ethernet (registered trademark) cable, a USB cable, and the like. The communication device 207 can also download a program from the server 130 and store it in the storage device 206. A program may be stored in an optical disc such as a CD / DVD, an optical drive may be connected to the communication device 207, and the program may be read from the optical disc and stored in the storage device 206.

  FIG. 3 is a block diagram showing the hardware configuration of the server 130 according to the present embodiment. The server 130 includes a processing device 301, a display device 302, an input device 303, a storage device 306, and a communication device 307. These components are connected by a bus 308, but may be connected individually as needed.

  The processing device 301 includes a processor that controls each device included in the server 130, and performs various processes with the storage device 306 as a work area. The display device 302 has a function of displaying information to the server user. The input device 303 has a function of receiving an input from a user, such as a keyboard and a mouse.

  The storage device 306 includes a hard disk, a main memory, and a buffer memory. A program 309 is stored on the hard disk. However, the hard disk may be any non-volatile storage or non-volatile memory as long as it can store information, and may be removable. The storage device 306 stores a program 309 and various data that can be referred to with the execution of the program.

  The communication device 307 performs wired communication using an Ethernet (registered trademark) cable or the like, mobile communication, and wireless communication such as a wireless LAN, and connects to the network 110.

  FIG. 4 shows an example of a functional block diagram of the mobile terminal device 120 of the present invention. The mobile terminal device 120 includes a mobile terminal control unit 401, a mobile terminal display unit 402, a mobile terminal input unit 403, an imaging unit 404, a position information acquisition unit 405, a mobile terminal storage unit 406, and a mobile terminal communication unit 407.

  The portable terminal display unit 402 displays information to be presented to the user. The portable terminal input unit 403 receives an input from the user. The imaging unit 404 captures a still image or a moving image of the real space area. The position information acquisition unit 405 acquires current position information of the mobile terminal device 120.

  The portable terminal storage unit 406 stores a reference image, which is an image of each object, in association with object identification information (ID) indicating the type of the object. In the present embodiment, although there are a plurality of types of objects, it is possible to operate even with only one type. If there is only one type of object, the reference image may not be stored in association with the object ID. The reference image can be downloaded from, for example, the server 130 into the acquisition confirmation application to the storage unit 406 of the mobile terminal device 120. It is preferable to update the reference image stored in the portable terminal storage unit 406 at a predetermined timing based on the information received from the server 130 via the communication unit 407. For example, it is possible to download and update the reference image at an update timing of an event for giving a benefit or every fixed period. The reference image may be downloaded when the acquisition confirmation application is used as a web browser and a website for capturing a browse image is accessed.

  The control unit 401 of the mobile terminal device 120 selects one of a plurality of types of objects in the image of the space captured by the imaging unit 404 of the mobile terminal device based on the local feature amount of the shape for detecting the object. It is determined whether or not an object of the type is imaged, and when it is determined that an object of any type is imaged, an image determined to be imaged of the object is acquired It is captured as an object image, and the acquired captured object image is transmitted to the server 130. The mobile terminal position information can also be acquired using the position information acquisition unit 405 and transmitted to the server 130 together with the captured image.

  In the present embodiment, the portable terminal control unit 401 is realized by the processing device 201 executing the program 209 included in the hardware configuration described in FIG. 2, but is also realized by preparing an integrated circuit or the like. it can. The mobile terminal display unit 402, the mobile terminal input unit 403, the imaging unit 404, the position information acquisition unit 405, the mobile terminal storage unit 406, and the mobile terminal communication unit 407 are the display device 202, the input device 203, and the like described in FIG. Although implemented by the hardware of the imaging device 204, the position information acquisition device 205, the storage device 206, and the communication device 207, the program 209 is executed by the processing device 201 or the like and realized by cooperating with the hardware. It is also good.

  FIG. 5 shows an example of a functional block diagram of the server 130 of the present invention. The server 130 includes a server control unit 501, a server display unit 502, a server input unit 503, a server storage unit 506, and a server communication unit 507.

  The server display unit 502 displays information for presentation to a user such as a server administrator. The server input unit 503 receives an input from the user. The server storage unit 506 stores a reference image, which is an image of an object, and an image feature of the object in association with an object ID. When there is only one type of object, the reference image and the image feature amount may not be stored in association with the object ID. Furthermore, the server storage unit 506 also stores the image feature amount for each object in association with the object ID.

  In response to a request from the portable terminal device 120, the server control unit 501 transmits the reference image stored in the storage unit 506 to the portable terminal device 120 via the communication unit 507 together with the linked object ID. The reference image can be read from the storage unit 506 at a predetermined timing such as the timing at which the reference image is updated, transmitted to the mobile terminal device 120, and the reference image in the mobile terminal device 120 can be updated.

  The server control unit 501 receives the acquisition confirmation request transmitted from the mobile terminal device 120 using the server communication unit 507, and, based on the image feature amount of one type of object among a plurality of types of objects, Determine the feature correlation amount of the acquisition object image for the object of the type, and determine whether the acquisition object image is an image of the object of the type, based on the determined feature correlation amount. judge. Furthermore, it may be determined whether or not this acquisition target object image is an image acquired illegally. If it is determined that the target object is captured in the acquisition target object image and it is determined that the image is not an illegally acquired image, it is determined that the user has acquired the target object, and the corresponding benefit etc. Give.

  In the present embodiment, the server control unit 501 is realized by executing the program 309 included in the hardware configuration described in FIG. 3 by the processing device 301, but can also be realized by preparing an integrated circuit or the like. . The server display unit 502, the server input unit 503, the server storage unit 506, and the server communication unit 507 are realized by the respective hardware of the display device 302, the input device 303, the storage device 306 and the communication device 307 described in FIG. However, the program 309 may be executed by the processing device 301 or the like and realized by cooperating with each hardware.

  Next, FIG. 6 shows a flowchart of the operation of the portable terminal device 120 and the server 130 in the present embodiment. In the present embodiment, in order to obtain an item in an online game played by a user of a smart phone that is the portable terminal device 120, the user purchases and obtains one of a plurality of products that are objects of a plurality of types. The case where it is asked to do is explained to an example. Here, since the items in the online game given for each type of product are different, it is necessary to specify which product the user has acquired.

  The user actually visits a store that sells the target product, purchases one of the products, acquires it, images the acquired product using the mobile terminal device 120, and transmits it to the server 130 Do. The server 130 determines whether the acquired product is a product of any type among target products having a plurality of products, and if it is determined that the captured product is one type of the target product, the user can It is determined that the item has been acquired, and an item corresponding to the item type is distributed to the game account of the user. The contents of these processes will be specifically described below.

  First, in the present embodiment, the user downloads and installs the acquisition confirmation application as the program 209 on the portable terminal device 120, visits one of the stores, and purchases a product. When the acquisition confirmation application in the portable terminal device 120 is activated to capture an image of the acquired product, the portable terminal device 120 starts acquisition confirmation request processing (S601). In the acquisition confirmation request process, the control unit 401 of the portable terminal device 120 captures an image including a product using the imaging unit 404, captures it as an acquisition target object image, and transmits an acquisition confirmation request including the acquisition target object image to the server 130. Transmit via the communication unit 407.

  The server 130 receives the acquisition confirmation request and executes acquisition confirmation processing (S602). In the acquisition confirmation process, the server 130 determines whether the product imaged by the user is any one of a plurality of types of products targeted for benefit provision.

  A more specific process content of the acquisition confirmation request process (S601) in the present embodiment is shown in FIG. When the user activates the acquisition confirmation application in the mobile terminal device 120, acquisition confirmation request processing S601 is started.

  The portable terminal control unit 401 sets a counter d for the reference image to 0 (S701). In the present embodiment, in the storage unit 406 of the portable terminal device 120, a reference image is stored in association with each object ID of D types of objects (goods) to be provided with benefits, Based on the determination, it is determined whether the image captured by the user includes a product as an object. The reference image is, for example, an object of comparison in order of object ID numbers.

  The user directs the imaging unit 404 of the mobile terminal device 120 to the acquired commodity to try to image the commodity. The imaging unit 404 captures an image of the real space in the directed direction (S702). The captured image is preferably one frame of a moving image, but may be an image captured by still image shooting. Then, the d-th (0 ≦ d <D) reference image of the reference images stored in the storage unit 406 is compared with the captured image of the space captured by the imaging unit 404 (S703), It is determined whether or not the object linked to is captured within a predetermined region in the image of the space in which the object is captured (S705).

  In the present embodiment, the comparison between the reference image and the captured image of the space captured by the imaging unit 404 is performed based on the local feature amount of the shape. Here, the shape includes not only the shape of the product but also the shape and pattern of characters such as logos and product names described in the product. By using the local feature amount, it can be determined at a high speed whether a product is captured in the captured image. The local feature amount is preferably calculated in advance and stored in the storage unit 406 in advance.

  Here, in order to roughly determine whether or not an object is imaged using the local feature amount, other objects or objects other than the object similar to the object of the reference image being compared are imaged Even in this case, it may be determined that the target of the reference image to be compared is being imaged. In the present invention, the possibility of such erroneous determination in determination using local feature quantities is permitted. Calculation is performed by specifying the region where the object is estimated to be captured at high speed by using the local feature amount, and limiting the target to be calculated using the feature correlation amount for more accurate identification. It is possible to reduce the amount and quickly identify the type of object.

  In the present embodiment, the user is guided to move the position and angle of the mobile terminal device 120 so that the product is captured in the predetermined area, and the area in which the product is captured is limited to the predetermined area. As a result, the position and size of the product in the captured image are geometrically normalized, and the amount of information processing for the subsequent product type identification is reduced.

  The predetermined area where the product is to be imaged is an area smaller than the entire area that can be imaged by the imaging unit 404, and the product is imaged within the predetermined area and the peripheral portion of the product is simultaneously imaged. The image of the peripheral area can be used for detection of a fraudulently acquired object image (S 907) described later.

  When it is not determined that the object of the d-th reference image is captured within a predetermined area in the captured image of the space (S704), the counter d is counted up (S705), and the D type to be determined is determined. It is determined whether the reference image of all the objects of and the captured image have been compared (S706). If not determined for all, it is determined whether the next reference image is an object being captured in the captured image (S703, S704). If it is determined that all the reference images and the captured image have been compared (S706), the counter d is reset (S701), and an image of a new space is captured (S703). Perform a comparison with the reference image. The processing of S701 to S706 is repeatedly executed until it is determined that the target object of any reference image is captured within a predetermined region in the image of the space in which the target object is captured.

  FIG. 8 shows a display screen 800 of the acquisition confirmation application displayed on the display unit 402 as one embodiment. The target object is a hamburger. As shown in FIG. 8A, the display unit 402 displays a frame 801 indicating a predetermined area for containing the hamburger 802 as an object, and the object within a frame together with the image of the space captured by the imaging unit 404. Display a message asking you to enter. Here, the entire imaging area 803 of the imaging unit 404 is displayed on the display unit 402, and the frame 801 is an area smaller than the entire area. Therefore, when the hamburger 802 is placed in the frame 801, an image 805 around the hamburger 802 is displayed in the outer area of the frame 801. When instructed to put the hamburger in the frame 801, the user moves the portable terminal device 120.

  The determination process (S703, S704) using the local feature to determine whether or not the target image of the reference image is captured within a predetermined area in the image of the space in which the target object is captured is, for example, ORB (Oriented FAST and Rotated) BRIEF) It can be realized by using a key point matching technique using feature quantities. Specifically, characteristic points (key points) as local feature amounts in the reference image are detected in advance and stored in the storage unit 406 as part of the acquisition confirmation application. Then, the feature points (key points) of the reference image stored in advance are superimposed in a predetermined area (frame 801), and the feature points of the image obtained by imaging by the imaging unit 404 are superimposed in the predetermined area. When overlapping with the selected feature point, it is determined that the target object of the reference image is captured within a predetermined region in the image of the space in which the target object is captured. The feature points of the reference image and the feature points of the captured image do not have to completely overlap. For example, when a predetermined number of feature points overlap in a predetermined shift range, it can be determined that an object is captured in a predetermined area.

  When the user moves the mobile terminal device 120 and places the target within the predetermined area, it is determined that the target of the reference image is captured within the predetermined area in the image of the space in which the target is captured ( S704) An image of a space including an object imaged in a predetermined region and its peripheral portion is fetched as an acquisition object image (S707). In the present embodiment, the acquisition of the acquisition target object image is to store the pickup image in the storage unit 406 for the subsequent processing. Since each mobile terminal device performs processing for capturing an acquisition target object image, even if a large number of acquisition confirmation requests are made simultaneously, the load is distributed to each mobile terminal device, and the load on the server can be reduced. it can.

  As shown in FIG. 8 (b), whether to transmit the acquisition object image to the server together with a message for informing the user that acquisition of the acquisition object image was successful, or to reacquire an acquisition confirmation request or to retake the image Is displayed on the display unit 402, and when the input unit 403 receives an input for retaking, the process returns to S701, and when an input for transmission to the server 130 is received, The control unit 401 transmits the acquisition confirmation request to the server 130 that transmits the acquisition confirmation request via the network 110 using the communication unit 407 (S 708). The acquisition confirmation request includes a user ID identifying a user and an acquisition target object image.

  In the present embodiment, the reference image of the predetermined object and the captured image are compared to determine whether the object is imaged in the predetermined area, but a plurality of types of objects having common features are determined. With regard to objects, the determination may be made based on data representing feature quantities indicating common features. In this case, all objects having this common feature are detected. Which of these objects is captured in the acquired object image is specified by the process described later.

  Next, a specific process of the acquisition confirmation process (S602) executed by the server 130 will be described based on the flowchart shown in FIG. In the present embodiment, the feature correlation amount of the acquired object image with respect to the k-th object is determined based on the image feature amount of the k-th object of the reference image (S902), and the determined feature correlation amount It is determined whether or not the product included in the acquisition target object image is any one type of target product based on (S903). Thereafter, it is determined whether the image has not been tampered with, or whether the image is an unauthorized image such as a duplicate of an image captured by another user (S 907). If one type of target is included in the acquisition target object image and it is not an incorrect image, a privilege corresponding to the type is given to the user (S 908). Hereinafter, these processes will be described in detail.

  The server control unit 501 sets a counter k for the type of object to 0 (S901). In the present embodiment, in the storage unit 506, the unique feature amount and the non-unique feature amount for each of a plurality of types of objects to be provided with benefits are linked to the object IDs of the respective objects and stored in the storage unit 506. It stores and sequentially determines whether any of these objects are included in the image captured by the user. Here, although both the unique feature and the non-unique feature are unique to each object, only the unique feature is unique to each object, and the non-unique feature is shared for all types. It may be The objects to be compared in S902 and S903 can be, for example, in the order of object ID numbers.

  The unique feature amount is an amount indicating a feature appropriate for identifying a type of an object, and the non-unique feature amount is an amount indicating an inappropriate feature to identify the type of the object. In the present embodiment, the unique feature amount and the non-unique feature amount are feature amounts of the color of the image representing the object. Here, the unique feature amount is a table including one or more characteristic RGB values of the object among the RGB values of each pixel of the image representing the object. For example, it is possible to create a table by extracting the RGB values of pixels which are considered characteristic by visual observation and the RGB values of pixels which are considered not characteristic from an image obtained by imaging the object. This table may be created based on the reference image of each object, but since the RGB values in the object image also change depending on the color of illumination at the time of imaging, etc., the object is imaged under various illumination conditions It is preferable to create a table including a plurality of unique feature quantities based on the plurality of images.

  As an example, when the product name is described on the packaging sheet of the hamburger which is the object, consider the case where the background color of the packaging sheet is common although the character color of the product name is different for each product type. In this case, a unique feature amount table including RGB values indicating character colors unique to each product is created, and a non-unique feature amount table including RGB values indicating background colors is created. Furthermore, since the character color of a certain product is different from the character color of another product, RGB values indicating the character color of another product can be included in the non-unique feature amount table for that product. Since the character color for items other than the product is a color not included in the product, by including such a feature quantity in the non-unique feature quantity, the probability of misidentifying the product can be reduced.

  In the present embodiment, it is assumed that the characters of the trade name "CHEESE BURGER" are green, the letters of the trade name "HUMBERGER" are purple, and the background color of both the CHEEESE BURGER and HUMBERGER wrapping papers is brown. Tables 1 and 2 show the unique feature amount table and the non-unique feature amount table of CHEESE BURGER at this time.

  Table 1 is a table showing unique features of CHEEESE BURGER. For example, 131, 150, 87 of ID = 0 are one RGB value which shows the green used as a character color of CHEEESE BURGER. Depending on the lighting conditions, the RGB values in the imaged image will be different even if the same printed green color, so other RGB values indicating green are also included. Table 2 is a table showing non-unique feature quantities of CHEESE BURGER. For example, 107, 71, 54 of ID = 0 are RGB values indicating a brown color common to both CHEESE BURGER and HUMBERGER packaging sheets. Furthermore, it may include RGB values indicating red, which is the character color of HUMBERGER, not included in the packaging paper of CHEESE BURGER.

  The server control unit 501 determines the feature correlation amount of the obtained object image of the object based on the unique feature amount and the non-unique feature amount of the object. In the present embodiment, the area ratio between the unique feature and the non-unique feature extracted based on the unique feature of the object and the non-unique feature is taken as the feature correlation amount for the subject.

  First, color differences between all the RGB values in the table of the unique feature amount and the non-unique feature amount of the object and the RGB values of each pixel in the obtained object image are calculated. When the table including the RGB value having the smallest color difference with the RGB value at a certain pixel is the unique feature amount table, the pixel is set as the unique feature portion (unique feature pixel) and the non-unique feature amount table Make the pixel a non-unique feature (non-unique pixel). Then, the area ratio of the extracted unique feature to the non-unique feature is calculated.

In the calculation of color difference, the godlove color difference equation (Godlove, IH: Improved Color-Difference Formula, with Applications to Perceptibility and Acceptability of Fadings, Journal of the), which is a distance system of colors on a uniform color space. Optical Society of America, 41 (11), pp. 760-770, (1951)), CIE 2000 (Luo, MR, Cui, G. & Rigg, B., "The development of the CIE 2000 colour-difference formula CIEDE 2000 ", Color Research & Application, Volume 26, Issue 5, October 2001, pp. 340-350, DOI: 10.1002 / col. 1049) can be used. For example, the godlove color difference equation Δ _godlove generally known as a distance on the HSV space can be defined as follows.

  FIG. 10 is a diagram visualizing the unique feature and the non-unique feature determined by the process of this embodiment. Here, unique features (pixels) are shown in white and non-unique features (pixels) in black. Thus, a portion having RGB values close to the RGB values included in the unique feature amount table, that is, a character portion which is a feature color portion of the object is extracted as the unique feature portion and included in the non-unique feature amount table It can be seen that a portion having RGB values close to the RGB values being read, that is, a background portion which is not the characteristic color of the object is extracted as the non-unique feature. The area ratio of the extracted unique feature to the non-unique feature, which is the feature correlation amount, is, for example, the area ratio of the white portion to the black portion in FIG.

  If the difference between the unique feature table and the RGB values in the unique feature table is within the threshold without using the non-unique feature table, it can be determined as an unique feature, otherwise it may be determined as a non-unique feature. However, as described above, even when an object of the same type is imaged, the RGB value of each pixel of the imaged image changes depending on the illumination condition. Therefore, there is a high possibility that an erroneous determination may be made if it is determined whether or not an inherent feature portion is determined based only on whether the inherent feature amount is within a certain range. For example, when the acquired object image photographed under illumination close to green, which is a characteristic color, is determined, all the pixels become colors close to the unique feature amount, so all pixels are determined to be unique features There is a risk of being

  On the other hand, as in the present embodiment, by using the non-unique feature amount table as well, it is possible to perform determination more accurately by performing relative determination of which of the unique feature amount and the non-unique feature amount is closer to It becomes. For example, if a non-unique feature amount is extracted based on an image captured under green illumination, pixels of a greenish background color may be obtained even for an acquisition target object image captured under green illumination. Since the color is close to the RGB values included in the non-unique feature amount table, it can be determined as a non-unique feature portion. The accuracy can be further improved by creating these tables based on images captured under various illuminations. Moreover, the accuracy can be further enhanced by inserting a color not included in the object as the non-unique feature amount.

  In step S903, it is determined whether the k-th object is imaged based on the determined feature correlation amount of the acquired object image with respect to the k-th object. If the feature correlation amount is within a predetermined range for the target object, it is determined that a CHEESE BURGER is captured in the acquisition target object image.

  Then, if it is determined that the k-th object is captured, the process proceeds to fraud detection (S 907), and if it is determined that the k-th object is not captured, the counter k is counted up (S 904) If it has not been determined for all the types K (S 905), the process returns to S 902 to execute the determination processing again for the next type. If it is not determined that the object is imaged even though the determination is made for all types K, it is determined that an error is detected because no object of any type is detected (S906) An error message is transmitted to the terminal device 120, and a notification that the object has not been detected is notified (S910).

  If it is determined that the k-th target is captured, it is determined whether the image is an unauthorized acquisition target image (S907). The fraud detection detects that the image of the object to be acquired has been tampered with and that the image obtained by another user has been illegally acquired by copying or the like. As a method of verifying whether the image has been tampered with, see AE Dirik and N. Memon, "Image tamper detection based on demosaicingartifacts," 2009 16th IEEE International Conference on Image Processing (ICIP), Cairo, 2009, pp. 1497-1500 and Minati Mishra, Flt. Lt. Dr. MC Adhikary, "Digital Image Tamper Detection Techniques-A Comprehensive Study," International Journal of Computer Science and Business Informatics, Vol. As shown in 2013, various methods are known. In the present invention, any method may be used as long as it is a method of verifying whether an image has been tampered with.

  When it is determined that the browsed image has not been tampered with, it has been illegally acquired, for example, by duplicating an image captured by another user based on the image of the peripheral part of the object included in the acquisition object image It is determined whether it is a thing. An image including the periphery of the object may be reflected by a passerby, or the position of a shade may change, so the same image may not be obtained if the time for shooting is different. Further, since the peripheral part of the product is different for each imaging location of the product, it is different for each captured image. In the present invention, the acquisition target object image stored in the server storage unit 506 and stored in the server storage unit 506, which is already included in the acquisition confirmation request that has already been determined to be a valid acquisition confirmation request, By comparing the object image with the already obtained acquired object image, it is possible to determine whether it is an image acquired by duplication. If the similarity between the two is higher than a certain level, it is judged as an incorrect image.

  The fraud detection based on the peripheral image can be realized, for example, by the method described in Patent Document 1 (Japanese Patent No. 6181893). In the present embodiment, since the object is imaged within a predetermined area smaller than the entire area that can be imaged by the imaging unit 404, the peripheral portion of the object is also imaged simultaneously, and the fraud detection disclosed in Patent Document 1 Methods can be incorporated to perform fraud detection.

  If it is determined in the unauthorized acquisition target image detection (S 907) that the image is not an unauthorized acquisition target object image, it is determined that the user has acquired the k-th target object, and the k-th target object is identified. A process for giving the corresponding benefit is executed (S 908), and a notification that the benefit has been given is transmitted to the portable terminal device 120 (S 910). If it is determined that the image is an unauthorized acquisition target object image, processing for the unauthorized acquisition target object image is executed (S 909), and a notification indicating that the image is an unauthorized acquisition target image is transmitted to the mobile terminal device 120. It transmits to (S910).

  The present embodiment uses a local feature amount that is a robust feature amount that is resistant to rotation and movement for detecting an object region in which an objective image is included in a captured image, and an object included in a detected object region The matching process using two feature quantities is performed, in which global color feature quantities are used for object identification to determine one of the variations. This makes it possible to identify various types of objects at high speed and with high accuracy. By combining a local feature that is strong against changes and noise but not sufficiently high in discriminability, and a global color feature that is weak against changes and noise while being high in discriminability, both discriminatoryness and noise tolerance Makes it possible to realize at the same time.

  In particular, when the object is a product packaged in a packaging sheet, etc., noise such as deformation or dirt due to folding of the packaging sheet is included a lot, and the product type can be accurately determined only by the existing feature amount matching. Can not. However, by using this embodiment, even if a large amount of noise is included, it is possible to specify the type of the object at high speed with high accuracy.

  In the present embodiment, if it is determined in S903 that the k-th object is captured in the acquisition object image based on the feature correlation amount of the acquisition object image with respect to the k-th object, the k-th object is determined Although it has been determined that the target object has been imaged, it is also possible to calculate feature correlation amounts for all of a plurality of types of objects and to determine that the target object with the highest correlation is being imaged. For example, it is assumed that an object with the smallest difference between the prestored predetermined value Ak for the k-th object and the area ratio Ck calculated for the acquisition object image is captured in the acquisition object image It can be decided.

  In the present embodiment, the image feature quantity used to obtain the feature correlation quantity in S902 is the color feature quantity, but the feature correlation quantity can also be determined using the shape feature quantity. For example, a window of N × N pixels can be defined and sequentially applied to each pixel of the reference image to extract the pattern of the shape, and based on this, the feature amount of the shape can be obtained.

In order to simplify the description, it is assumed that N is 2 and a case of a black and white image is taken as an example. Define the 2 × 2 pixel window 1100 as shown in FIG. 11, the pattern of pixels included in this window becomes a pattern of 2 ^{quadruplicate} for each pixel is white or black, the identification number PID in each pattern ( Assign 0-15). For example, a pattern in which all pixels (1101 to 1104) are white is PID = 0, and if only the lower right pixel (1101) is black, PID = 1, and if all pixels are black, PID = 15.

  Then, the window 1100 is applied to the reference image 1200 shown in FIG. First, the upper left pixel 1101 of the window 1100 is aligned with the upper left pixel 1201 of the reference image 1200 (FIG. 12B), and a pattern of 2 × 2 pixels at the upper left of the reference image 1200 is extracted and extracted The pattern's PID counter is incremented by one. Here, since the upper left 2 × 2 pixels of the reference image 1200 are all white, the counter CPID_0 with PID = 0 is incremented by one. As shown in FIGS. 12 (c) and 12 (d), the window 1100 is moved one pixel at a time, this process is performed on all pixels of the reference image, and the number of patterns corresponding to each PID is counted. Then, the counters CPID_0 to CPID_15 are acquired. Then, with the counters CPID_0 to CPID_15 as bins, a shape histogram as an image feature amount for the type of the object is acquired.

  A shape histogram is created using this window 1100 also for the acquired object image. Then, the difference between the shape histogram of the acquired object image and the shape histogram of the reference image is calculated, and this is used as the feature correlation amount (S902). Then, if the difference value that is the feature correlation amount is within the predetermined range, it is determined that the target object is captured in the acquisition target object image, and if it is out of the range, it is determined that it is not captured. (S903). Here, the smaller the difference value that is the feature correlation amount, the higher the correlation.

  Furthermore, as a modification, it may be specified in advance which object the user has acquired. In this case, based on only the local feature amount of the specified object, confirmation processing of whether the object is imaged in the predetermined area is performed, and in the acquisition confirmation request, the type of the specified object is Is transmitted to the server 130 including the object ID for specifying the object, and the server 130 determines whether the object is captured in the acquired object image based only on the image feature amount of the object specified based on this. Since the determination may be made, faster processing can be realized.

  It is also possible to limit the number of times for one user to obtain one type of target object at a predetermined store and perform check-in processing. In this case, when the portable terminal device 120 transmits the confirmation request, the position information acquired by the position information acquiring unit 405 at the time of imaging the acquisition target object image is included in the acquisition confirmation request and transmitted, and the server 130 receives the benefit. When the assignment is performed, it is checked whether the user has already checked in based on the image captured near the position for the type of the object. If the check-in has already been performed, the privilege is not given, and if the check-in has not been made, the privilege is given and then the user is already checked in at the corresponding position. The information shown is stored in the server storage unit 506.

  In the case where there is one category of a plurality of types of objects including common features, for example, objects included in an acquisition object image based on a common feature amount such as color at the start of the acquisition confirmation process (S602) It is also possible to determine that is one of the categories, and execute the processing of S701 and later only for the objects in the category. When there are a large number of objects, the object of comparison can be reduced, so the processing can be speeded up.

  In the present embodiment, the case where the server 130 executes the acquisition confirmation process (S602) has been described, but the mobile terminal device 120 may execute the acquisition confirmation process. For example, in the storage unit 406 of the mobile terminal device 120, in addition to the reference image of the object, the unique feature amount and the non-unique feature amount for each of a plurality of types of objects to be benefited It is realizable by stringing to thing ID and memorize | storing. It is preferable that these pieces of information be periodically acquired from the server 130 and updated. The portable terminal device 120 does not transmit the acquisition confirmation request to the server 130 after the image of the space including the target and its peripheral portion captured in the predetermined area in S707 is captured as the acquisition target image. The mobile terminal device 120 can execute the processing of S901 to S909 as acquisition confirmation processing based on the acquired acquisition target object image instead of the server 130, and can confirm that the user has acquired the target object. It is determined that a benefit is given as a result of confirmation as to whether or not the user has acquired an object on the display unit 402 of the portable terminal device 120 without performing notification transmission (S910), and that the request is a fraud check request. You can display messages such as events.

  In the processes or operations described above, the processes or operations can be freely changed as long as no contradiction arises. Each embodiment described above is an illustration for describing the present invention, and the present invention is not limited to these embodiments. The present invention can be practiced in various forms without departing from the scope of the invention. Further, the effects described in the present embodiment only list the most preferable effects arising from the present invention, and the effects according to the present invention are not limited to those described in the present embodiment.

100 acquisition confirmation system 110 network 120 portable terminal device 130 server 201 processing device 202 display device 203 input device 204 imaging device 205 position information acquisition device 206 storage device 207 communication device 208 bus 209 program 301 processing device 302 display device 303 input device 306 storage Device 307 Communication device 308 Bus 309 Program 401 Other control unit 402 Display unit 403 Input unit 404 Imaging unit 405 Position information acquisition unit 406 Storage unit 407 Communication unit 501 Control unit 502 Display unit 503 Input unit 506 Storage unit 507 Communication unit 800 Display screen 801 frame 802 object 803 whole image 805 peripheral image 1100 window 1101 pixel 1102 pixel 1103 pixel 1104 pixel 1200 reference image 1201 pixel

Claims (13)

A system comprising a portable terminal device and a server for confirming that one of a plurality of types of objects has been acquired,
The portable terminal device is
An object of any type of a plurality of types of objects is present in an image of a space captured by the imaging device of the mobile terminal device based on the local feature amount of the shape for detecting the plurality of types of objects. Determine whether or not it has been imaged,
If it is determined that an object of any type is imaged, an image determined to be imaged of the object is fetched as an acquisition object image,
Sending the acquisition object image to the server;
The server is
Receiving the transmitted acquisition target object image;
The feature correlation amount of the acquired object image with respect to the one type of object is determined based on the image feature amount of the one type of the plurality of types of objects,
It is determined based on the determined feature correlation amount whether or not the acquisition target object image is an image obtained by imaging the one type of target object.
A system characterized by The image feature quantities of the one type of object include unique feature quantities and non-unique feature quantities,
The unique feature includes a feature color value indicating a color that is a feature of the one type of object, and the non-unique feature indicates a color that is not a feature of the one type of object Contains the value,
The feature correlation amount is an amount of a feature portion determined to indicate a color that is a feature of the one type of target object based on the image feature amount of the one type of target object in the acquisition target object image A ratio to the amount of non-features determined to indicate the non-feature color of the one type of object,
Determining whether or not the acquisition target object image is an image obtained by imaging the one type of target object is determined depending on whether or not the feature correlation amount is within a predetermined range.
The system of claim 1, wherein:   The non-unique feature amount is not included in the color value indicating the color common to the plurality of types of objects and the unique feature amount of the one type of object but is included in the unique feature amounts of other types of objects The system according to claim 2, characterized in that it comprises at least one of the color values. The image feature of the one type of object includes a histogram of the shape included in the reference image of the type of object;
The feature correlation amount is determined based on a difference between a histogram of the reference image and a histogram of a shape included in the acquired object image.
The system of claim 1, wherein:   It is determined whether or not an object of any of a plurality of types of objects is imaged in an image of a space imaged by an imaging device of the portable terminal device. 5. A system according to any one of the preceding claims, characterized in that it is to determine whether or not it is imaged within the region of.   The system according to any one of claims 1 to 5, wherein one type of the plurality of types of objects is specified by a user in advance via the mobile terminal device.   Determining whether or not the acquisition object image is an image obtained by imaging the one type of object is performed for each of the plurality of types of objects, and an image obtained by imaging an object having the highest correlation The system according to any one of claims 1 to 5, wherein it is determined that   The local feature amount is a common local feature amount of the plurality of types of objects, or a local feature amount unique to each of the plurality of types of objects. Or the system described in paragraph 1. A server used in a system for confirming that one of a plurality of types of objects has been acquired,
Receiving from the mobile terminal apparatus an acquisition target object image determined to be imaged by the mobile terminal apparatus;
The feature correlation amount of the acquired object image with respect to the one type of object is determined based on the image feature amount of the one type of the plurality of types of objects,
It is determined based on the determined feature correlation amount whether or not the acquisition target object image is an image obtained by imaging the one type of target object.
Server characterized by A set of programs for confirming that one of a plurality of types of objects has been acquired, on one or more computers,
An object of any type of a plurality of types of objects is present in an image of a space captured by the imaging device of the mobile terminal device based on the local feature amount of the shape for detecting the plurality of types of objects. Determining whether or not an image is captured;
When it is determined that an object of any type is imaged, the step of capturing an image determined to be imaged of the object as an acquisition object image;
Determining a feature correlation amount of the acquired object image with respect to the one type of object based on an image feature amount of the one type of the plurality of types of objects;
Determining whether the acquired object image is an image obtained by imaging the one type of object based on the determined feature correlation amount;
A program that runs A program for confirming that one of a plurality of types of objects has been acquired, the server
Receiving from the mobile terminal apparatus an acquisition target object image determined to be imaged by the mobile terminal apparatus;
The feature correlation amount of the acquired object image with respect to the one type of object is determined based on the image feature amount of the one type of the plurality of types of objects,
It is determined based on the determined feature correlation amount whether or not the acquisition target object image is an image obtained by imaging the one type of target object.
A program that lets you do things. A method for confirming that one of a plurality of types of objects has been acquired, the method comprising:
An object of any type of a plurality of types of objects is present in an image of a space captured by the imaging device of the mobile terminal device based on the local feature amount of the shape for detecting the plurality of types of objects. Determining whether or not an image is captured;
When it is determined that an object of any type is imaged, the step of capturing an image determined to be imaged of the object as an acquisition object image;
Determining a feature correlation amount of the acquired object image with respect to the one type of object based on an image feature amount of the one type of the plurality of types of objects;
Determining whether the acquired object image is an image obtained by imaging the one type of object based on the determined feature correlation amount;
How to do it. A method for confirming that one of a plurality of types of objects has been acquired, the server comprising:
Receiving from the mobile terminal apparatus an acquisition target object image determined to be imaged by the mobile terminal apparatus;
The feature correlation amount of the acquired object image with respect to the one type of object is determined based on the image feature amount of the one type of the plurality of types of objects,
It is determined based on the determined feature correlation amount whether or not the acquisition target object image is an image obtained by imaging the one type of target object.
How to do things.

Images