JP6365117B2 - Information processing apparatus, image determination method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, an image determination method, and a program.

  Image matching technology has been widely used in various fields. For example, in the fields of face authentication and biometric authentication, a technique for matching an image captured with a visible light camera or an infrared camera and a template image is used. In a more general application, a technique for matching an image captured by a mobile terminal with a camera or a digital camera with a landmark image is used.

  For matching between images, for example, a feature amount at a feature point of an input image (hereinafter referred to as a local feature amount) is compared with a local feature amount at a feature point of a reference image, and a reference corresponding to the feature point of the input image is compared. A technique for searching for feature points (hereinafter referred to as corresponding points) of an image is used. By statistically processing the set of corresponding points found by the search, it is possible to recognize the presence of the reference image and the position of the reference image in the input image. As an example of statistical processing, the relative position between the reference point set in the reference image and each feature point is held, and the position of the reference point from each feature point extracted as a corresponding point candidate is voted, A processing method for determining a threshold value of the total number of votes has been proposed.

  For the search for the corresponding points, for example, features such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), BREF (Binary Robust Independent Elementary Features) are used. When SIFT or SURF is used, the corresponding points can be detected with high accuracy. However, the local feature amount of SIFT or SURF has a large data size, and the use of SIFT or SURF increases the calculation load during search.

  BRIEF is expressed by a local feature amount based on a luminance difference between pixels calculated for each of a plurality of pixel pairs set in a feature region around each feature point. For example, a set of bit values corresponding to the sign (positive / negative) of the luminance difference is the local feature amount. Therefore, BRIEF can be extracted at a high speed with a small amount of calculation compared to SIFT and SURF. In other words, if BREF is used, a local feature amount can be extracted from an input image at high speed even with a portable terminal having a lower calculation capability than a server device or the like.

  Other features that express the features of an image using local feature amounts based on the luminance difference between pixels as in BREF include, for example, ORB (Oriented Fast and Rotated BRIEF) and FRAK (Fast Retina Keypoint). The ORB uses learning based on a large number of images when determining the arrangement of pixel pairs. FREEK uses information about the human retina pattern when determining the arrangement of pixel pairs. Even when ORB or FREEK is used, a local feature can be extracted from an input image at high speed.

  Although it is not a method that uses local feature amounts based on pixel pairs, a feature region identification method that determines whether each small region is a feature region from the feature amount of a pixel included in a small region in an image has been proposed. . This method sets a small area in the image so that the small areas overlap each other, stores the number of determinations for each pixel and the intermediate result of the feature amount calculation, and eliminates the feature amount calculation of the overlapping portion Is adopted.

Japanese Unexamined Patent Publication No. 2011-210054 JP-A-61-217884

David G. Lowe, "Distinctive image features from scale-invariant keypoints", International Journal of Computer Vision, 60, 2 (2004), pp.91-110. H.Bay, T.Tuytelaars, and L.V.Gool, "SURF: Speed Up Robust Features", Proc. Of Int. Conf. Of ECCV, 2006. M. Calonder, V. Lepetit, C. Strecha, and P. Fua., "BRIEF: Binary Robust Independent Elementary Features.", In Proceedings of the European Conference on Computer Vision (ECCV), 2010. Alexandre Alahi, Raphael Ortiz, Pierre Vandergheynst, "FREAK: Fast Retina Keypoint", In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: An efficient alternative to SIFT or SURF", In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2011.

  As described above, the local feature amount based on the pixel pair has a low processing load for extraction, and contributes to a reduction in matching processing load. On the other hand, since the number of combinations of each feature point of the input image and each feature point of the reference image is large, the processing load for searching for corresponding points is high. The number of combinations increases as more feature points are arranged. On the other hand, by arranging many feature points, it becomes possible to search for more precise corresponding points, and there is a possibility that the determination accuracy by statistical processing may be improved.

  Therefore, according to one aspect, an object of the present invention is to provide an information processing apparatus, an image determination method, and a program that can reduce a processing load for searching for a corresponding feature point.

  According to one aspect of the present disclosure, among a plurality of pixel pairs arranged in a common arrangement rule in each of a plurality of image areas that partially overlap, the image areas are arranged in each of the image areas in the overlap area. A storage unit for storing arrangement information indicating an arrangement rule set so that the positions of the respective pixel pairs coincide with each other, and a feature amount set of each pixel pair arranged in the image area according to the arrangement rule. The first image and the second image based on the distance between the feature amount of each image region set in the first image and the feature amount of each image region set in the second image. A calculation unit that determines the similarity of the image pair, the calculation unit stores a calculation result related to the feature amount of the pixel pair located in the overlapping region in the storage unit, and another image region at the same position as the pixel pair. When calculating the feature value of a pixel pair in The information processing apparatus is provided that utilizes a calculation result of the.

  In addition, according to another aspect of the present disclosure, the computer overlaps the image regions of a plurality of pixel pairs that are arranged in a common arrangement rule in each of the plurality of image regions that are partially overlapped. In the area, obtain arrangement information indicating an arrangement rule set so that the positions of the pixel pairs arranged in each image area coincide with each other, and set a feature amount set of each pixel pair arranged in the image area according to the arrangement rule. The first image and the first image are used based on the distance between the feature amount of each image region set in the first image and the feature amount of each image region set in the second image. In the process of determining the similarity with the image of 2 and calculating the distance, the calculation result relating to the feature amount of the pixel pair located in the overlapping region is stored in the storage unit and is located at the same position as the pixel pair. Special characteristics of pixel pairs in other image areas Image judging method utilizing the calculation results stored in the storage unit in the calculation for the amount is provided.

  In addition, according to another aspect of the present disclosure, overlapping of overlapping image areas among a plurality of pixel pairs arranged in a common arrangement rule in each of a plurality of overlapping image areas on the computer. In the area, obtain arrangement information indicating an arrangement rule set so that the positions of the pixel pairs arranged in each image area coincide with each other, and set a feature amount set of each pixel pair arranged in the image area according to the arrangement rule. The first image and the first image are used based on the distance between the feature amount of each image region set in the first image and the feature amount of each image region set in the second image. In the process of determining the similarity with the image of 2 and calculating the distance, the calculation result related to the feature amount of the pixel pair located in the overlapping region is stored in the storage unit, and is in the same position as the pixel pair. Of pixel pairs in other image areas Program for executing a process to utilize the calculation results stored in the storage unit in the calculation regarding symptoms amount is provided.

  According to the present invention, it is possible to reduce the processing load for searching for a corresponding feature point.

It is the figure which showed an example of the information processing apparatus which concerns on 1st Embodiment. It is the figure which showed an example of the system which concerns on 2nd Embodiment. It is the figure which showed an example of the hardware which can implement | achieve the function which the terminal device which concerns on 2nd Embodiment has. It is the block diagram which showed an example of the function which the terminal device which concerns on 2nd Embodiment has. It is the figure which showed an example of the pixel pair information which concerns on 2nd Embodiment. It is a figure for demonstrating the feature point and feature area of the input image which concern on 2nd Embodiment. It is a figure for demonstrating calculation of the feature-value which concerns on 2nd Embodiment. It is a 1st figure for demonstrating efficiency improvement of the feature point calculation which concerns on 2nd Embodiment. It is a 2nd figure for demonstrating efficiency improvement of the feature point calculation which concerns on 2nd Embodiment. It is a 3rd figure for demonstrating efficiency improvement of the feature point calculation which concerns on 2nd Embodiment. It is a 4th figure for demonstrating efficiency improvement of the feature point calculation which concerns on 2nd Embodiment. It is the block diagram which showed an example of the function which the server apparatus which concerns on 2nd Embodiment has. It is the figure which showed an example of the reference image information (reference image management table) which concerns on 2nd Embodiment. It is the figure which showed an example of the reference image information (feature quantity management table) which concerns on 2nd Embodiment. It is the figure which showed an example of the distance information (distance recording table) which concerns on 2nd Embodiment. It is a figure for demonstrating the feature point and feature area of a reference image which concern on 2nd Embodiment. It is a figure for demonstrating the voting process which concerns on 2nd Embodiment. It is a figure for demonstrating the determination which concerns on 2nd Embodiment. It is a 1st figure for demonstrating efficiency improvement of the corresponding point search (distance calculation) which concerns on 2nd Embodiment. It is a 2nd figure for demonstrating efficiency improvement of the corresponding point search (distance calculation) which concerns on 2nd Embodiment. It is a flowchart for demonstrating operation | movement of the terminal device which concerns on 2nd Embodiment. It is a flowchart for demonstrating the operation | movement regarding calculation of a feature-value among operation | movement of the terminal device which concerns on 2nd Embodiment. It is a flowchart for demonstrating operation | movement of the server apparatus which concerns on 2nd Embodiment. It is a flowchart for demonstrating the operation | movement regarding the search of a reference image among the operation | movement of the server apparatus which concerns on 2nd Embodiment. It is a flowchart for demonstrating operation | movement regarding distance calculation among operation | movement of the server apparatus which concerns on 2nd Embodiment. It is a figure for demonstrating arrangement | positioning of the feature point which concerns on the modification (modification # 1) of 2nd Embodiment. It is a figure for demonstrating the arrangement | positioning of the pixel pair and the structure of a local feature-value which concern on the modification (modification example # 1) of 2nd Embodiment. It is a figure for demonstrating arrangement | positioning of the feature point which concerns on the modification (modification example # 2) of 2nd Embodiment. It is a figure for demonstrating arrangement | positioning of a pixel pair and the structure of a local feature-value which concern on the modification (modification example # 2) of 2nd Embodiment. It is the figure which showed an example of the distance information which concerns on the modification (modification # 2) of 2nd Embodiment. It is a 1st flowchart for demonstrating operation | movement regarding distance calculation among operation | movement of the server apparatus which concerns on the modification (modification example # 2) of 2nd Embodiment. It is a 2nd flowchart for demonstrating operation | movement regarding distance calculation among operation | movement of the server apparatus which concerns on the modification (modification example # 2) of 2nd Embodiment. It is a 3rd flowchart for demonstrating operation | movement regarding distance calculation among operation | movement of the server apparatus which concerns on the modification (modification example # 2) of 2nd Embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, about the element which has the substantially same function in this specification and drawing, duplication description may be abbreviate | omitted by attaching | subjecting the same code | symbol.

<1. First Embodiment>
The first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of an information processing apparatus according to the first embodiment. An information processing apparatus 10 illustrated in FIG. 1 is an example of an information processing apparatus according to the first embodiment.

As illustrated in FIG. 1, the information processing apparatus 10 includes a storage unit 11 and a calculation unit 12.
The storage unit 11 is a volatile storage device such as a RAM (Random Access Memory) or a nonvolatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The calculation unit 12 is a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). However, the calculation unit 12 may be an electronic circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). For example, the calculation unit 12 executes a program stored in the storage unit 11 or another memory.

  The storage unit 11 includes a plurality of pixel pairs arranged in a common arrangement rule in each of a plurality of overlapping image areas, and each pixel pair arranged in each image area in the overlapping area where the image areas overlap. The arrangement information indicating the arrangement rule set so that the positions of the two coincide with each other is stored.

  For example, when the right half area (right area R2) of the image area Ain1 and the left half area (left area R1) of the image area Ain2 are set to overlap, the right area R2 of the image area Ain1 and The left region R1 of the image region Ain2 corresponds to the overlapping region (the hatched region in FIG. 1). When the right region R2 of the image region Ar1 and the left region R1 of the image region Ar2 are set to overlap, the right region R2 of the image region Ar1 and the left region R1 of the image region Ar2 are the above overlapping regions. It corresponds to.

  In the example of FIG. 1, the storage unit 11 includes information on a pixel pair in which each pixel is arranged in the right region R2, information on a pixel pair in which each pixel is arranged in the left region R1, and 1 in each of the regions R1 and R2. Information on pixel pairs in which pixels are arranged one by one is stored as arrangement information. Hereinafter, a set of pixel pairs in which one pixel is arranged in each of the regions R1 and R2, a group A, a set of pixel pairs in which each pixel is arranged in the left region R1, a group B, and each pixel in the right region R2 A set of pixel pairs to be processed is expressed as a group C.

  The arrangement rule indicated by the arrangement information is set so that the B group pixel pair and the C group pixel pair have the same arrangement. For example, when pixel pairs are arranged in the image areas Ain1 and Ain2 in accordance with this arrangement rule, the positions of the pixel pairs arranged in the right area R2 of the image area Ain1 and the pixel pairs arranged in the left area R1 of the image area Ain2 The position matches. The same applies when pixel pairs are arranged in the image areas Ar1 and Ar2.

  The calculation unit 12 uses a set of feature quantities of each pixel pair arranged in the image area according to the arrangement rule as a feature quantity of the image area. The calculation unit 12 also determines the first image and the second image based on the distance between the feature amount of each image region set in the first image and the feature amount of each image region set in the second image. The similarity with the image is determined.

  For example, BREF, SIFT, SURF, CARD (Compact And Realtime Descriptors), FRAK, BRISK (Binary Robust Invariant Scalable Keypoints) and the like can be used as the feature amount. For example, when BRIEF is used, the bit value based on the sign (positive / negative) of the luminance difference between the two pixels of the pixel pair becomes the feature amount of the pixel pair. The sign (positive / negative) of the luminance difference is associated with the bit value serving as the feature amount in advance. In addition, a bit string (a set of feature amounts) in which bit values representing the features of each pixel pair are arranged is a feature amount representing the features of the image area in which each pixel pair is arranged.

  In the example of FIG. 1, the feature quantity Qin1 of the image area Ain1 is a bit string in which the feature quantities of the A group pixel pairs are arranged, a bit string in which the feature quantities of the B group pixel pairs are arranged, and the characteristics of the C group pixel pairs. It is expressed as a bit string obtained by combining a bit string in which quantities are arranged. The feature amount Qin2 of the image area Ain2 is a bit string in which the feature amounts of the pixel pairs in the A group are arranged, a bit string in which the feature amounts of the B group pixel pairs are arranged, and the feature amounts of the C group pixel pairs are arranged. It is expressed as a bit string combined with a bit string.

  If the image area Ain1 is an image area set as the first image and the image area Ar1 is an image area set as the second image, the calculation unit 12 calculates the distance between the feature quantities Qin1 and Qr1. To do. At this time, the calculation unit 12 calculates the distance dA11 between the bit strings in the A group, the distance dB11 between the bit strings in the B group, and the distance dC11 between the bit strings in the C group, and calculates the sum of the distances dA11, dB11, and dC11 as the feature amount Qin1. , Qr1 distance.

  Similarly, the calculation unit 12 calculates a distance for a set of other image regions (for example, image regions Ain2, Ar2). For example, the calculation unit 12 calculates the distance between the feature amounts Qin2 and Qr2. At this time, the calculation unit 12 calculates the distance dA22 between the bit strings of the A group, the distance dB22 between the bit strings of the B group, and the distance dC22 between the bit strings of the C group, and the sum of the distances dA22, dB22, and dC22 is the feature amount Qin2. , Qr2 distance. Then, the calculation unit 12 determines the similarity between the first image and the second image based on the calculated distance.

  However, the calculation unit 12 stores the calculation result regarding the feature amount of the pixel pair located in the overlapping region in the storage unit 11, and performs the calculation regarding the feature amount of the pixel pair in the other image region at the same position as the pixel pair. The calculation result stored in the storage unit 11 is used.

  For example, when calculating the distance between the image areas Ain1 and Ar1 (comparison # 1), the calculation unit 12 calculates the distance dC11 related to the pixel pair of the C group. Thereafter, when calculating the distance between the image areas Ain2 and Ar2 (comparison # 2), the calculation unit 12 obtains the distance dB22 related to the group B pixel pair.

  The group C pixel pair arranged in the image area Ain1 and the group B pixel pair arranged in the image area Ain2 are set at the same position. Therefore, the bit string representing the characteristics of the C group pixel pairs arranged in the image area Ain1 and the bit string representing the characteristics of the B group pixel pairs arranged in the image area Ain2 are the same bit string. Similarly, the bit string representing the characteristics of the group C pixel pair arranged in the image area Ar1 and the bit string representing the characteristics of the group B pixel pair arranged in the image area Ar2 are the same bit string.

  Therefore, the distance dC11 and the distance dB22 have the same value. Therefore, when the calculation unit 12 calculates the distance dC11, the calculation unit 12 stores the calculation result in the storage unit 11. When calculating the distance dB22, the calculation is omitted and the distance dC11 stored in the storage unit 11 is used. That is, when calculating the distance between the image areas Ain2 and Ar2, the calculation unit 12 calculates the distances dA22 and dC22, and sets the sum of dA22, dC22, and dC11 as the distance between the feature amounts Qin2 and Qr2. As described above, it is possible to reduce the calculation amount of the distance calculation by storing a part of the calculation result regarding the overlapping area in the storage unit 11 and using it for the subsequent calculation.

  By reducing the load on the distance calculation between the image areas set in the first and second images, a set of image areas with the minimum distance can be extracted at a high speed with a small amount of calculation, and corresponding features It becomes possible to reduce the processing load for the point search.

The first embodiment has been described above.
<2. Second Embodiment>
Next, a second embodiment will be described.

[2-1. system]
A system according to the second embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a system according to the second embodiment.

  As shown in FIG. 2, the system according to the second embodiment includes a terminal device 100 and a server device 200. Devices such as a mobile phone, a smartphone, a digital camera, and a personal computer are examples of the terminal device 100. An application server, a content management system, and the like are examples of the server device 200.

  The terminal device 100 can access the server device 200 via the radio base station BS and the network NW. The radio base station BS is, for example, a mobile communication base station or a wireless LAN (Local Area Network) base station. The network NW is a backbone network for mobile communication, a WAN (Wide Area Network), or the like.

  The system in FIG. 2 can be applied to, for example, a photo management service that enables the server device 200 to manage a captured image captured by the terminal device 100. This system can also be applied to social media services such as SNS (Social Networking Service) having a photo management function. Hereinafter, an example of the photo management service will be described, but the application range of the system according to the second embodiment is not limited to this.

  In many cases, time information and position information are used for image management. For example, there are a method of grouping and managing images using shooting date and time, and a method of grouping and managing images using shooting location information acquired by GPS (Global Positioning System). If the images are grouped, it becomes easy to search for images with hints of shooting time and shooting location. In addition, there are methods of tagging images such as “family trip” and “wedding” and managing the images using the tags.

  The second embodiment relates to a mechanism for automatically recognizing a specific object (hereinafter referred to as a landmark) from an image and tagging the recognition result to the image. For example, the system of FIG. 2 can provide a mechanism in which the server apparatus 200 automatically recognizes a landmark and attaches a tag when information of an image taken at a travel destination by the terminal apparatus 100 is transmitted to the server apparatus 200. With this mechanism, it is easy to realize image search specifying landmarks and automatic image organization.

  Landmark recognition is realized by a process of determining whether a portion similar to a landmark image (hereinafter referred to as a reference image) is included in the input image. A plurality of reference images are used. The reference image is an image of the landmark taken from the front, an image taken from another direction, an image taken from the same direction with a different angle of view or brightness, and the like. By using a plurality of reference images for the same landmark, the occurrence of recognition failure or erroneous recognition is reduced.

The landmark is recognized, for example, by the following procedure.
First, the terminal device 100 sets a plurality of feature points for an input image that is a landmark recognition target. However, the position of the feature point is set in advance. The arrangement of the feature points is determined by, for example, a method of arranging the feature points at an equal interval at a predetermined interval or a method of arranging an irregular arrangement pattern determined at random. However, the arrangement of the feature points in the reference image is the same as the arrangement of the feature points set in the input image.

  The terminal device 100 that has set a feature point extracts a feature amount from a peripheral region (hereinafter referred to as a feature region) around the feature point set in the input image. The feature area is, for example, a rectangular area having a feature point as the center (for example, a rectangular area having 48 pixels on one side). The feature amount is, for example, BRIEF, SIFT, SURF, CARD, FREEK, BRISK, or the like. In the following, an example using BRIEF will be described.

  The feature amount extracted by the terminal device 100 is transmitted to the server device 200. The server device 200 specifies a feature region corresponding to the feature amount received from the terminal device 100 using the feature amount extracted from each feature region of the reference image. For example, the server device 200 calculates the distance between feature amounts for a set of each feature area of the input image and each feature area of the reference image, and specifies a set of feature areas that minimizes the distance. That is, the server device 200 extracts a feature point pair.

  Next, the server apparatus 200 performs statistical processing on the extracted feature point pair and determines whether or not a reference image is included in the input image. The server device 200 executes the above determination process for all reference images, and identifies the reference image included in the input image. Then, the server device 200 transmits landmark information corresponding to the identified reference image to the terminal device 100. The terminal device 100 executes processing for displaying the landmark information received from the server device 200 on the screen and processing for adding the information to the image.

  As described above, the system in FIG. 2 is configured to send only the feature amount without sending the input image from the terminal device 100 to the server device 200. For this reason, there is less concern that an input image, which is personal information, leaks outside the terminal device 100 via the network NW or the like. Further, the system of FIG. 2 uses a feature amount based on a pixel pair such as BRIEF, and can provide a service without imposing an excessive load on the terminal device 100.

The system according to the second embodiment has been described above. Hereinafter, the terminal device 100 and the server device 200 included in this system will be further described.
[2-2. hardware]
With reference to FIG. 3, hardware capable of realizing the functions of the terminal device 100 will be described. FIG. 3 is a diagram illustrating an example of hardware capable of realizing the functions of the terminal device according to the second embodiment.

  The functions of the terminal device 100 can be realized using, for example, hardware resources of the information processing apparatus illustrated in FIG. That is, the functions of the terminal device 100 are realized by controlling the hardware shown in FIG. 3 using a computer program.

  As shown in FIG. 3, this hardware mainly includes a CPU 902, a ROM (Read Only Memory) 904, a RAM 906, a host bus 908, and a bridge 910. Further, this hardware includes an external bus 912, an interface 914, an input unit 916, an output unit 918, a storage unit 920, a drive 922, a connection port 924, and a communication unit 926.

  The CPU 902 functions as, for example, an arithmetic processing unit or a control unit, and controls the overall operation of each component or a part thereof based on various programs recorded in the ROM 904, the RAM 906, the storage unit 920, or the removable recording medium 928. . The ROM 904 is an example of a storage device that stores a program read by the CPU 902, data used for calculation, and the like. The RAM 906 temporarily or permanently stores, for example, a program read by the CPU 902 and various parameters that change when the program is executed.

  These elements are connected to each other via, for example, a host bus 908 capable of high-speed data transmission. On the other hand, the host bus 908 is connected to an external bus 912 having a relatively low data transmission speed via a bridge 910, for example. As the input unit 916, for example, a mouse, a keyboard, a touch panel, a touch pad, a button, a switch, a lever, or the like is used. Furthermore, as the input unit 916, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used.

  As the output unit 918, for example, a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), or an ELD (Electro-Luminescence Display) is used. As the output unit 918, an audio output device such as a speaker or headphones, or a printer may be used. In other words, the output unit 918 is a device that can output information visually or audibly.

  The storage unit 920 is a device for storing various data. As the storage unit 920, for example, a magnetic storage device such as an HDD is used. Further, as the storage unit 920, a semiconductor storage device such as an SSD (Solid State Drive) or a RAM disk, an optical storage device, a magneto-optical storage device, or the like may be used.

  The drive 922 is a device that reads information recorded on a removable recording medium 928 that is a removable recording medium or writes information on the removable recording medium 928. As the removable recording medium 928, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is used.

  The connection port 924 is a port for connecting an external connection device 930 such as a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or an optical audio terminal. For example, a printer or a camera is used as the external connection device 930.

  The communication unit 926 is a communication device for connecting to the network 932. As the communication unit 926, for example, a communication circuit for wired or wireless LAN (Local Area Network), a communication circuit for WUSB (Wireless USB), a communication circuit or router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) Communication circuits, routers, communication circuits for mobile phone networks, and the like are used. A network 932 connected to the communication unit 926 is a wired or wireless network, and includes, for example, the Internet, a LAN, a broadcast network, a satellite communication line, and the like.

  The hardware of the terminal device 100 has been described above. Note that the above hardware can also realize the functions of the server device 200. Therefore, description of the hardware of the server device 200 is omitted.

[2-3. Terminal device functions]
Next, functions of the terminal device 100 will be described with reference to FIGS.
FIG. 4 is a block diagram illustrating an example of functions of the terminal device according to the second embodiment. FIG. 5 is a diagram illustrating an example of pixel pair information according to the second embodiment. FIG. 6 is a diagram for explaining feature points and feature regions of an input image according to the second embodiment. FIG. 7 is a diagram for explaining calculation of feature amounts according to the second embodiment.

  FIG. 8 is a first diagram for explaining the efficiency of the feature point calculation according to the second embodiment. FIG. 9 is a second diagram for explaining the efficiency of the feature point calculation according to the second embodiment. FIG. 10 is a third diagram for explaining the efficiency of the feature point calculation according to the second embodiment. FIG. 11 is a fourth diagram for explaining the efficiency of the feature point calculation according to the second embodiment.

As illustrated in FIG. 4, the terminal device 100 includes a storage unit 101, an image acquisition unit 102, a feature point setting unit 103, a feature amount calculation unit 104, and a transmission unit 105.
Note that the function of the storage unit 101 can be realized by using the above-described RAM 906, the storage unit 920, or the like. The functions of the image acquisition unit 102, the feature point setting unit 103, and the feature amount calculation unit 104 can be realized using the above-described CPU 902 or the like. The function of the transmission unit 105 can be realized by using the connection port 924, the communication unit 926, or the like described above.

(Regarding pixel pair information 101a)
The storage unit 101 stores pixel pair information 101a. The pixel pair information 101a is information representing a pixel pair (hereinafter referred to as a pixel pair) used when calculating a feature amount of a feature region set for each feature point of an image. For example, the pixel pair information 101a includes position information indicating the positions of two pixels that form a pixel pair. The feature region is, for example, a rectangular region centered on the feature point, and is set to a size of 32 pixels × 32 pixels, 48 pixels × 48 pixels, or the like.

  An example of the pixel pair information 101a is shown in FIG. As shown in FIG. 5, the pixel pair information 101a includes the coordinates (X1, Y1) and (X2, Y2) of two pixels forming the pixel pair Lk (k = 1, 2,...). The coordinates of the pixel are expressed by coordinate values with the upper left corner of the feature area as the origin (0, 0). Also, the number of pixel pairs is set to 128, 256, 512, for example. For example, when the number of pixel pairs is set to 256, 256 pixel pairs are set for each feature region.

(About feature point settings)
Refer to FIG. 4 again. The image acquisition unit 102 acquires an image (hereinafter referred to as an input image Pin) that is a landmark recognition target. For example, the image acquisition unit 102 acquires an input image Pin from an external storage device (not shown) or a camera (not shown) connected to the terminal device 100. In addition, the image acquisition unit 102 may acquire the input image Pin using a camera function of the terminal device 100. The input image Pin acquired by the image acquisition unit 102 in this way is stored in the storage unit 101.

  The feature point setting unit 103 reads the input image Pin stored in the storage unit 101, and sets a plurality of feature points in the read input image Pin. For example, as shown in FIG. 6, the feature point setting unit 103 sets feature points in the input image Pin by using a method (Dens Sampling) of extracting a plurality of feature points arranged at equal intervals. However, a method such as FAST (Features from Accelerated Segment Test) can also be applied for setting the feature points.

  As shown in FIG. 6, a feature area Ain is set with reference to the feature point Win. Note that the feature point set in the input image Pin is denoted as Win. In addition, the m-th feature point is represented as Win (m), and the m + 1-th feature point is represented as Win (m + 1). Further, a feature region set with the m-th feature point Win (m) as a reference is denoted as Ain (m). Information on the feature points Win and the feature area Ain set by the feature point setting unit 103 is stored in the storage unit 101 and input to the feature amount calculation unit 104.

(About feature amount calculation and transmission)
Refer to FIG. 4 again. The feature amount calculation unit 104 refers to the pixel pair information 101a and calculates the feature amount of each feature region Ain set by the feature point setting unit 103.

  For example, as shown in FIG. 7, the feature amount calculation unit 104 extracts the luminance of each pixel included in the pixel pair set in the feature area Ain of the input image Pin (C1). Next, the feature amount calculation unit 104 obtains a difference between the two extracted luminances, and calculates a luminance difference between each pixel pair (C2). However, the feature amount calculation unit 104 may omit the calculation of the luminance difference for some pixel pairs by an efficiency method described later. Next, the feature amount calculation unit 104 generates a bit string (hereinafter referred to as a local feature amount) in which bit values representing the sign (positive / negative) of the calculated luminance difference are arranged (C3).

  The sign of the luminance difference and the bit value are associated in advance. For example, a positive sign is associated with the bit value “1”, and a negative sign is associated with the bit value “0”. In addition, when calculating the luminance difference, it is also set in advance which one of the two pixels forming the pixel pair is subtracted from the luminance of the other pixel. For example, it is preset that the sign of the luminance difference obtained by subtracting the luminance of the pixel located at the coordinates (X2, Y2) from the luminance of the pixel located at the coordinates (X1, Y1) is used for generating the local feature amount. . The order in which the bit values are arranged in the step C3 is also set in advance.

  When 256 pixel pairs are set, a local feature amount expressed by a 256-bit bit string is obtained. The local feature amount is calculated for all feature regions Ain included in the input image Pin. A set of local feature amounts calculated by the feature amount calculation unit 104 for the input image Pin is input to the transmission unit 105. The transmission unit 105 transmits the local feature amount input from the feature amount calculation unit 104 to the server device 200. In this way, since the input image Pin itself is not transmitted, it is possible to reduce the risk of leaking personal information of the user to the outside.

(About efficient feature calculation)
Here, the above-described technique for improving the amount of feature calculation will be described with reference to FIGS. 6 and 8 to 11.

  As already described, the arrangement of the feature points Win and the size of the feature area Ain are set in advance. As an example of the setting, as shown in FIG. 6, each feature region Ain is set so that adjacent feature regions Ain are in contact with each other, or the feature regions Ain are arranged so that the adjacent feature regions Ain partially overlap each other. There is a setting to arrange.

  The example of FIG. 6 shows a case where the characteristic areas Ain arranged side by side in the left-right direction (X direction) are set so as to overlap only a half of the characteristic area Ain. In this example, the right half portion of the feature area Ain (m) set with the mth feature point Win (m) as a reference is a feature set with the m + 1th feature point Win (m + 1) as a reference. It overlaps with the left half of the area Ain (m + 1). Further, in this example, the feature areas Ain that are vertically adjacent to each other are set to contact each other.

  When calculating the local feature quantities of the feature areas Ain (m) and Ain (m + 1), the feature quantity calculation unit 104 adds pixel pairs Lk (k = 1, 2, 2, respectively) to the feature areas Ain (m) and Ain (m + 1). ...) And a bit value based on the luminance difference of each pixel pair Lk is calculated.

  The bit values may be calculated independently for the feature areas Ain (m) and Ain (m + 1), but the calculation of the bit values regarding the overlapping portions of the feature areas Ain (m) and Ain (m + 1) is redundant. . When calculating the bit value for the left half of the feature area Ain (m + 1), if the bit value calculated for the right half of the feature area Ain (m) can be used, the amount of calculation of the local feature amount can be reduced.

  In order to use the bit value, the pixel pair arrangement in the right half area (right area) of the feature area Ain (m) and the pixel pair in the left half area (left area) of the feature area Ain (m + 1) It is assumed that the arrangement is the same. However, even if only a part of the pixel pair in the right region of the feature region Ain (m) has the same arrangement as the pixel pair in the left region of the feature region Ain (m + 1), the calculation amount due to the diversion of the bit value is reduced. A reduction effect is obtained.

  Therefore, in each feature area Ain, the pixel pair information 101a is set so as to include a pixel pair (a divertable pixel pair) having the same arrangement in the right area and the left area. In this case, as shown in FIG. 8, the feature amount calculation unit 104 calculates the local feature amount of the feature region Ain (m), and then holds the bit value corresponding to the divertable pixel pair.

  Next, when calculating the local feature amount of the feature region Ain (m + 1), the feature amount calculation unit 104 omits the calculation of the bit value corresponding to the divertable pixel pair in the left region of the feature region Ain (m + 1). Then, the local feature amount is obtained by using the held bit value. In addition, after calculating the local feature amount of the feature region Ain (m + 1), the feature amount calculation unit 104 holds bit values corresponding to pixel pairs that can be used for calculation of the feature region Ain (m + 2).

As described above, the calculation amount of the local feature amount can be reduced by diverting the already calculated bit value when calculating the local feature amount for a part of the pixel pairs set in the overlapping region.
Here, the arrangement of the pixel pair that allows the above calculation method to be applied will be further described.

  As shown in FIG. 9, in the pixel pair, two pixels are arranged so as to straddle the boundary between the right region and the left region of the feature region Ain (Group A), and each pixel is arranged in the left region. It can be classified into one (group B) and one in which each pixel is arranged in the right region (group C). The arrangement of the pixel pairs belonging to the B group in the left region and the arrangement of the pixel pairs belonging to the C group in the right region are the same. That is, when the left area and the right area are overlapped, the position of each pixel pair set in the left area matches the position of each pixel pair set in the right area.

  As described above, the arrangement of the pixel pairs belonging to the A group, the B group, and the C group is determined, and the whole arrangement of all the pixel pairs is recorded in the pixel pair information 101a and used for the calculation of the local feature amount. The The pixel pair information 101a may include attribute information for specifying the A group, the B group, and the C group for each pixel pair. For example, as shown in FIG. 10, when the pixel pairs L1, L2,..., L6 are arranged in the feature areas Ain (m), Ain (m + 1), the positions of the pixel pairs L5, L6 coincide in the overlapping area. Therefore, efficient calculation of the local feature amount shown in FIG. 8 can be realized.

  In order to facilitate the diversion of the bit values representing the characteristics of the pixel pairs belonging to the A group, the B group, and the C group, the bit string of the A group, the bit string of the B group, and the bit string of the C group are divided as shown in FIG. It is preferable to adopt a structure of local feature amounts. When this structure is adopted, for example, as shown in FIG. 11, a group C bit string is extracted from the local feature quantity of the feature area Ain (m), and the B group of the local feature quantity of the feature area Ain (m + 1) is extracted. By inserting it in the bit string portion, the calculation result can be diverted.

  As described above, by combining the calculation result shown in FIG. 8, the arrangement of the pixel pairs shown in FIG. 9, and the structure of the local feature shown in FIG. 11, the calculation of the local feature is made efficient. It becomes possible. Further, as will be described later, the processing of the server apparatus 200 is also made efficient by adopting the arrangement of the pixel pairs shown in FIG. 9 and the structure of the local feature quantity shown in FIG.

The function of the terminal device has been described above.
[2-4. Server device functions]
Next, functions of the server device 200 will be described with reference to FIGS.

  FIG. 12 is a block diagram illustrating an example of functions of the server device according to the second embodiment. FIG. 13 is a diagram illustrating an example of reference image information (reference image management table) according to the second embodiment. FIG. 14 is a diagram illustrating an example of reference image information (feature amount management table) according to the second embodiment. FIG. 15 is a diagram illustrating an example of distance information (distance recording table) according to the second embodiment.

  FIG. 16 is a diagram for explaining feature points and feature regions of a reference image according to the second embodiment. FIG. 17 is a diagram for explaining voting processing according to the second embodiment. FIG. 18 is a diagram for describing determination according to the second embodiment. FIG. 19 is a first diagram for explaining the efficiency of the corresponding point search (distance calculation) according to the second embodiment. FIG. 20 is a second diagram for explaining the efficiency of the corresponding point search (distance calculation) according to the second embodiment.

As illustrated in FIG. 12, the server device 200 includes a storage unit 201, a reception unit 202, a distance calculation unit 203, a voting processing unit 204, and a determination unit 205.
The function of the storage unit 201 can be realized using the above-described RAM 906, the storage unit 920, or the like. The function of the reception unit 202 can be realized by using the connection port 924, the communication unit 926, and the like described above. The functions of the distance calculation unit 203, the voting processing unit 204, and the determination unit 205 can be realized using the above-described CPU 902 or the like.

(Regarding pixel pair information 201a, reference image information 201b, distance information 201c)
The storage unit 201 stores a reference image Pr. The storage unit 201 stores pixel pair information 201a, reference image information 201b, and distance information 201c.

  The pixel pair information 201a is information related to the same pixel pair as the pixel pair information 101a stored in the storage unit 101 of the terminal device 100. The reference image information 201b is information related to the reference image Pr. The distance information 201c is information including a distance record table in which a part of calculation results is stored when calculating the distance between local feature points.

  The reference image information 201b includes a reference image management table and a feature amount management table. The storage unit 201 stores reference images Pr corresponding to various landmarks. In addition, a plurality of reference images Pr corresponding to a plurality of sizes and a plurality of photographing directions for one landmark are stored in the storage unit 201. Information that associates the data of the reference image Pr with the landmark is a reference image management table.

  As shown in FIG. 13, the reference image management table is a database that associates landmark names with image data (or links to image data) of the reference image Pr. Each record in the reference image management table is specified by the identification number ID # 1.

  As shown in FIG. 14, the feature amount management table is a database that associates each feature point included in the reference image Pr with a local feature amount calculated for each feature point. The feature amount management table is generated for each reference image Pr specified by the identification number ID # 1. Each record of the feature quantity management table is specified by the identification number ID # 2.

  As shown in FIG. 15, the distance recording table includes a feature area Ain (m) (m = 1, 2,..., M) of the input image Pin and a feature area Ar (n) (n = 1, 1) of the reference image Pr. 2,.., N) is a database in which a part of the calculation result (distance d) is recorded when the distance D is calculated. The distance d will be described in detail in the description of the method for improving the characteristic amount calculation described later.

(About feature point settings)
The feature points of the reference image Pr are set by a method such as Dense Sampling or FAST as in the case of setting feature points in the input image Pin.

  As shown in FIG. 16, a feature area Ar is set based on the feature point Wr. A feature point set in the reference image Pr is denoted as Wr. In addition, the nth feature point is denoted as Wr (n), and the (n + 1) th feature point is denoted as Wr (n + 1). Further, a feature region set based on the nth feature point Wr (n) is denoted as Ar (n). Local feature amounts of the feature region Ar (n) (n = 1, 2,..., N) are calculated in advance and stored in the feature amount management table. The calculation of the local feature amount is the same as the method of calculating the local feature amount of the feature region Ain.

(About distance calculation)
Refer to FIG. 12 again. The receiving unit 202 receives a local feature amount of the input image Pin from the terminal device 100. The local feature amount of the input image Pin received by the reception unit 202 is input to the distance calculation unit 203. The distance calculation unit 203 calculates the distance D between each feature area Ain of the input image Pin and each feature area Ar of the reference image Pr based on the local feature amount. The distance D represents the difference in features between the feature regions Ain and Ar.

Here, a method for calculating the distance D between the feature regions Ain and Ar will be further described.
Hereinafter, the local feature amount of the feature region Ain is denoted as LFin, and the local feature amount of the feature region Ar is denoted as LFr. The distance calculation unit 203 calculates a bit string HV from the local feature values LFin and LFr by the following equation (1). Next, the distance calculation unit 203 calculates the length of the bit string HV (the number of bit values 1 included in the HV). The length of the bit string HV is the distance D, and represents the difference in features between the feature regions Ain and Ar.

HV = {LFin XOR LFr}
... (1)
XOR included in the above equation (1) represents an exclusive OR operation. This calculation is an operation in which the bit values of the local feature values LFin and LFr are set to 1 when they match and 0 when they do not match. Therefore, by summing the number of bit values that are 1 by this calculation, the distance D (feature difference) between the feature regions Ain and Ar based on the local feature amount is obtained.

  The distance calculation unit 203 calculates the distance D between each feature region Ain of the input image Pin and each feature region Ar of the reference image Pr for all the reference images Pr by the above method. However, the distance calculation unit 203 omits a part of the calculation by an efficiency method described later, and efficiently calculates the distance D. Information on the distance D calculated by the distance calculation unit 203 is input to the voting processing unit 204.

(About voting process)
The voting processing unit 204 extracts a pair of feature areas Ar and Ain having the highest similarity (the distance D is the smallest) based on the distance D calculated for each pair of feature areas Ar and Ain. For example, as shown in FIG. 17, the voting processing unit 204, for the feature region Ar (n), the feature region Ain (m) having the smallest distance D from the feature regions Ain (1),. ) Are extracted (corresponding point search).

  The voting processing unit 204 uses the positional relationship (for example, a vector) between the feature point Wr of the feature area Ar and the center of the reference image Pr, and uses the extracted feature point Win of the feature area Ain as a reference. A point corresponding to the center (hereinafter referred to as a specific point) is specified (center estimation). For example, the voting processing unit 204 has moved from the feature point of the corresponding input image Pin to the same direction and distance based on the direction and distance (vector) from the known feature point of the feature region Ar to the center of the reference image Pr. A point is a specific point.

  In the case of FIG. 17, the voting processing unit 204 uses the positional relationship between the feature point Wr (n) of the feature area Ar (n) and the center of the reference image Pr, and uses the feature point Win (m) of the feature area Ain (m). ) To identify a specific point (white rectangular point). This specifying process is performed when the reference image Pr is superimposed on the input image Pin so that the feature point of the input image Pin matches the feature point of the corresponding reference image Pr, and the center of the reference image Pr in the input image Pin is determined. This corresponds to the process to be specified.

  When the horizontal width of the input image Pin is wi, the height is hi, the horizontal width of the reference image Pr is wr, and the height is hr, the coordinates (Xv, Yv) of the specific point are expressed by the following equations (2) and (3). Can be calculated by In the following equations (2) and (3), the coordinates of the feature point Wr are expressed as (Xr, Yr), and the coordinates of the feature point Win corresponding to the feature point Wr are expressed as (Xi, Yi). .

Xv = Xi-Xr + wr / 2
... (2)
Yv = Yi-Yr + hr / 2
... (3)
The voting processing unit 204 that has calculated the coordinates of the specific point determines whether or not the reference image Pr is included in the input image Pin by a statistical method (for example, voting). As shown in FIG. 17, the voting is performed using a voting map in which a grid is provided in an area having the same size as the input image Pin. A counter is associated with each cell of the voting map. The voting processing unit 204 identifies a cell including the coordinates of the specific point, and counts up the count value (the number of votes) of the counter corresponding to the identified cell. The process related to this count-up is voting.

  The voting processing unit 204 votes for each feature point of the reference image Pr. As shown in FIG. 18, a distribution of the number of votes is obtained from the vote map after the vote is completed. When the voting ends, the determination unit 205 extracts the maximum number of votes. The determination unit 205 compares the extracted maximum number with a preset threshold Th. As a modification, the determination unit 205 may compare a normalized value obtained by dividing the maximum number by the total number of votes in the vote map with a threshold value. Note that the threshold value is set so that an evaluation value such as a correct answer probability falls within an allowable range by performing experiments using various images.

  When the maximum number is larger than the threshold Th, the determination unit 205 determines that the reference image Pr exists in the input image Pin. In this case, the determination unit 205 determines the position of the cell corresponding to the maximum number as the center position of the reference image Pr. On the other hand, when the maximum number is smaller than the threshold Th, the determination unit 205 determines that the reference image Pr does not exist in the input image Pin. By performing the determination using the voting map, it is possible to absorb a slight error existing between the input image Pin and the reference image Pr and perform a robust determination.

  If it is determined that the reference image Pr exists in the input image Pin, the determination unit 205 reads the reference image management table corresponding to the reference image Pr from the storage unit 201, and obtains information on the landmark corresponding to the reference image Pr. It transmits to the terminal device 100. The landmark information includes, for example, a landmark name and position, an ID of a reference image management table, and the like.

  Tag information may be prepared for each landmark, and tag information corresponding to the landmark may be transmitted to the terminal device 100 according to the determination result. In this case, the terminal device 100 may be configured to automatically add tag information to the input image Pin.

  When it is determined that there are a plurality of landmarks, the determination unit 205 may transmit information on all the landmarks to the terminal device 100, or the landmark having the largest total number or maximum number of votes. Information may be transmitted to the terminal device 100.

  When it is determined that the reference image Pr does not exist in the input image Pin for all the reference images Pr included in the reference image management table, the determination unit 205 displays error information indicating that no landmark is present in the terminal device 100. Send to.

(About efficient search method for corresponding points)
Here, a method for improving the efficiency of the corresponding point search will be described with reference to FIGS. 9, 17, 19, and 20.

  As already described, the distance calculation unit 203 calculates the distance D of the local feature amount between each feature point Wr of the reference image Pr and each feature point Win of the input image Pin for corresponding point search. . For example, as shown in FIG. 17, the distance calculation unit 203 includes a feature area Ar (n) set on the basis of feature points Wr (n) (n = 1, 2,..., N), and feature points Win ( m) The distance D is calculated with respect to the feature area Ain (m) set with reference to m = 1, 2,..., M. As the number of feature points Wr and Win increases, the load for calculating the distance D increases. Therefore, if the load for calculating the distance D can be reduced, the corresponding point search process can be made more efficient.

  For example, when calculating the distance D between the feature points Win (m) and Win (m + 1) and the feature points Wr (n) and Wr (n + 1), as shown in FIG. Calculations S1,..., S4 are executed.

  The distance calculation unit 203 calculates (S1) the distance D based on the local feature amount of the feature region Ain (m) and the local feature amount of the feature region Ar (n), and (S2) the local feature amount of the feature region Ain (m). The calculation of the distance D based on the local feature amount of the feature area Ar (n + 1) is executed. The distance calculation unit 203 calculates (S3) the distance D based on the local feature amount of the feature region Ain (m + 1) and the local feature amount of the feature region Ar (n), and (S4) the local feature amount of the feature region Ain (m + 1). Calculation of the distance D based on the feature amount and the local feature amount of the feature region Ar (n + 1) is executed.

  The right area of the feature area Ain (m) and the left area of the feature area Ain (m + 1) are overlapping areas. Further, the right region of the feature region Ar (n) and the left region of the feature region Ain (n + 1) are overlapping regions. Further, as shown in FIG. 9, the group B pixel pair arranged in the left region and the group C pixel pair arranged in the right region are the same arrangement.

  That is, the group C bit string of the local feature quantities of the feature area Ain (m) is the same as the group B bit string of the local feature quantities of the feature area Ain (m + 1). Further, the bit string of group C among the local feature values of the feature area Ar (n) is the same as the bit string of group B of the local feature values of the feature area Ar (n + 1). Therefore, the calculation result (distance d) regarding the bit string of the C group in the distance D calculated in S1 coincides with the calculation result regarding the bit string of the B group among the distance D calculated in S4. Therefore, if a part of the calculation result (distance d) calculated in S1 is diverted to the calculation of S4, a part of the calculation of D4 can be omitted and the load on the calculation of the distance D can be reduced.

  As shown in FIG. 20, the distance calculation unit 203 performs an exclusive OR operation between the bit strings of the A group, the bit strings of the B group, and the bit strings of the C group. Distance DB, distance C of group C is calculated. The distance calculation unit 203 calculates the distance D by adding up the distance DA of the A group, the distance DB of the B group, and the distance DC of the C group. For example, the distance D (m, n) between the feature areas Ain (m) and Ar (n) is the distance DA (m, n) of the A group, the distance DB (m, n) of the B group, and the distance DC of the C group. It is given as the sum of (m, n).

  The distance calculation unit 203 that has calculated the distance D (m, n) is used when calculating the distance D (m + 1, n + 1) of the feature areas Ain (m + 1) and Ar (n + 1). m, n). At this time, the distance calculation unit 203 stores the distance DC (m, n) in the column of d (m, n) of the distance recording table (see FIG. 15).

  When calculating the distance D (m + 1, n + 1) between the feature areas Ain (m + 1) and Ar (n + 1), the distance calculation unit 203 omits the calculation of the distance DB (m + 1, n + 1) of the B group, The value stored in the column of d (m, n) is used as the group B distance DB (m + 1, n + 1). That is, the distance calculation unit 203 calculates the sum of the distance DA (m + 1, n + 1) in the A group, the value in the d (m, n) column (distance DC (m, n)), and the distance DC (m + 1, n + 1) in the C group. Is a distance D (m + 1, n + 1).

  In the above example, among the plurality of feature points arranged in the image, the feature point arranged at the upper left is used as the starting point, the processing target is moved in order to the right side, and the lowermost feature point is followed by the lower row. The case is shown in which the leftmost feature point is the target of processing, and processing is performed sequentially up to the lower right feature point. In this case, since there is no divertable distance d for the feature point located at the left end, the distance calculation unit 203 does not hold the distance DC of the C group when calculating the distance for the feature point located at the right end. As described above, when executing the process related to the feature point located at the end, the distance calculation unit 203 determines whether or not to execute the process illustrated in FIG. 20 according to the presence or absence of the divertable distance d.

  When the above method is applied, the calculation amount is reduced to 3/4 in the example of FIG. In this way, by reducing the calculation load of the distance D, the processing for searching for corresponding points is made efficient, and the processing for matching images is accelerated.

The function of the server device has been described above.
[2-5. Process flow]
Next, the flow of processing executed by the terminal device 100 and the server device 200 will be described with reference to FIGS.

(2-5-1. Operation of terminal device)
First, the flow of processing related to the operation of the terminal device 100 will be described with reference to FIGS. 21 and 22.

  FIG. 21 is a flowchart for explaining the operation of the terminal device according to the second embodiment. FIG. 22 is a flowchart for explaining operations related to feature quantity calculation among the operations of the terminal device according to the second embodiment.

(Operation of terminal equipment)
(S101) The image acquisition unit 102 acquires an image (input image Pin) that is a landmark recognition target.

  For example, the image acquisition unit 102 acquires the input image Pin from an external storage device or a camera connected to the terminal device 100. In addition, the image acquisition unit 102 may acquire the input image Pin using a camera function of the terminal device 100. The input image Pin acquired by the image acquisition unit 102 is stored in the storage unit 101.

(S102) The feature point setting unit 103 reads the input image Pin stored in the storage unit 101, and sets a plurality of feature points Win in the read input image Pin.
For example, the feature point setting unit 103 sets the feature points Win in the input image Pin using a method (Dens Sampling) of extracting a plurality of feature points Win arranged at equal intervals. However, a method such as FAST can also be applied to set the feature point Win. Information on the feature points Win and feature areas Ain set by the feature point setting unit 103 is stored in the storage unit 101. Information about the feature point Win and the feature area Ain is input to the feature amount calculation unit 104.

(S103) The feature amount calculation unit 104 selects one of the feature points Win set in S102.
(S104) The feature amount calculation unit 104 refers to the pixel pair information 101a, and calculates the local feature amount of the feature region Ain corresponding to the feature point Win selected in S103.

  For example, the feature amount calculation unit 104 extracts the luminance of each pixel for each pixel pair from the feature region Ain of the input image Pin (C1). Next, the feature amount calculation unit 104 calculates a luminance difference between each pixel pair by obtaining a difference between the two luminances extracted for each pixel pair (C2). Next, the feature amount calculation unit 104 generates a bit string (local feature amount) in which bit values representing the sign (positive / negative) of the calculated luminance difference are arranged (C3).

  (S105) The feature amount calculation unit 104 determines whether or not the feature amount calculation has been completed for all feature points Win included in the input image Pin. When the feature amount calculation has been completed for all feature points Win included in the input image Pin, the process proceeds to S106. On the other hand, if the feature amount calculation has not been completed for all feature points Win included in the input image Pin, the process proceeds to S103.

  (S106) The feature amount calculation unit 104 inputs a set of local feature amounts calculated for the input image Pin to the transmission unit 105. Then, the transmission unit 105 transmits the local feature amount input from the feature amount calculation unit 104 to the server device 200. When the process of S106 is completed, the series of processes shown in FIG.

Here, the process of S104 will be further described.
(S111) The feature amount calculation unit 104 refers to the pixel pair information 101a, and sets a plurality of pixel pairs in the feature region Ain set around the feature point Win.

  After the process of S111, the processes of S112 to S116 are sequentially executed for a set of pixel pairs belonging to each group in the order of the A group, the B group, and the C group. The group A is a set of pixel pairs in which one pixel is included in the left half range of the feature area Ain and the other pixel is included in the right half range. Group B is a set of pixel pairs in which both pixels are included in the left half of the feature area Ain. Group C is a set of pixel pairs in which both pixels are included in the right half of the feature area Ain.

  (S112) The feature amount calculation unit 104 determines whether there is a divertable bit string. The group B pixel pair and the group C pixel pair have the same arrangement. Therefore, for example, when the local feature amount of the feature area Ain (m−1) located to the left of the feature area Ain (m) has been calculated, the bit string of group C of the feature area A (m−1) is used as the feature area. It can be used for the bit string of group B of A (m).

  When the processing target is the A group or the C group, there is no divertable bit string, so the process proceeds to S113. When the processing target is the group B and there is a divertable bit string, the feature quantity calculation unit 104 sets the diversion source C group bit string as the diversion destination B group bit string. And the process of S113-S116 is abbreviate | omitted, and a process target switches to C group. If the processing target is group B and there is no divertable bit string, the process proceeds to S113.

(S113) The feature amount calculation unit 104 selects one pixel pair belonging to the group to be processed.
(S114) The feature amount calculation unit 104 calculates the luminance difference of the pixel pair selected in S113. For example, the feature amount calculation unit 104 extracts the luminance of each pixel of the pixel pair from the feature area Ain of the input image Pin. Then, the feature amount calculation unit 104 calculates the luminance difference between the pixel pairs by obtaining the difference between the two extracted luminances.

(S115) The feature amount calculation unit 104 selects a bit value corresponding to the sign (positive / negative) of the luminance difference calculated in S114, and sets the selected bit value as the bit value of the local feature amount.
The sign of the luminance difference and the bit value are associated in advance. For example, a positive sign is associated with the bit value “1”, and a negative sign is associated with the bit value “0”. In addition, when calculating the luminance difference, it is also set in advance which one of the two pixels forming the pixel pair is subtracted from the luminance of the other pixel. For example, it is preset that the sign of the luminance difference obtained by subtracting the luminance of the pixel located at the coordinates (X2, Y2) from the luminance of the pixel located at the coordinates (X1, Y1) is used for generating the local feature amount. .

  (S116) The feature amount calculation unit 104 determines whether all pixel pairs belonging to the group to be processed have been selected. If all pixel pairs have not been selected, the process proceeds to S113. On the other hand, when all the pixel pairs have been selected, the process proceeds to S117.

  (S117) The feature amount calculation unit 104 combines the bit strings of each group into a local feature amount. In addition, the feature amount calculation unit 104 stores the bit string of the C group in the storage unit 101 and saves it. When the process of S117 is completed, the series of processes shown in FIG.

The processing flow related to the operation of the terminal device 100 has been described above.
(2-5-2. Operation of server device)
Next, the flow of processing related to the operation of the server device 200 will be described with reference to FIGS.

  FIG. 23 is a flowchart for explaining the operation of the server apparatus according to the second embodiment. FIG. 24 is a flowchart for explaining the operation related to the search for the reference image among the operations of the server apparatus according to the second embodiment. FIG. 25 is a flowchart for explaining operations related to distance calculation among the operations of the server device according to the second embodiment.

  (S201) The receiving unit 202 receives the local feature amount of the input image Pin from the terminal device 100. The local feature amount of the input image Pin received by the reception unit 202 is input to the distance calculation unit 203.

(S202) The distance calculation unit 203 selects one reference image Pr included in the reference image management table.
(S203) The distance calculation unit 203, the voting processing unit 204, and the determination unit 205 execute search processing for the reference image Pr selected in S202.

  For example, the distance calculation unit 203 calculates a distance D representing the degree of difference between the feature area Ar of the reference image Pr and the feature area Ain of the input image Pin. The voting processing unit 204 generates a voting map MP based on the distance D calculated by the distance calculating unit 203. The determination unit 205 uses the voting map MP generated by the voting processing unit 204 to determine whether or not the reference image Pr exists in the input image Pin based on the comparison result between the maximum number of votes and the threshold Th.

  (S204) The determination unit 205 determines whether all the reference images Pr included in the reference image management table have been selected. If all the reference images Pr included in the reference image management table have not been selected, the process proceeds to S202. On the other hand, when all the reference images Pr included in the reference image management table have been selected, the process proceeds to S205.

  (S205) The determination unit 205 determines whether or not at least one reference image Pr exists in the input image Pin based on the result of S203. If at least one reference image Pr exists in the input image Pin, the process proceeds to S206. On the other hand, if it is not determined from the result of S203 that at least one reference image Pr exists in the input image Pin, the process proceeds to S207.

  (S206) The determination unit 205 reads the reference image management table of the corresponding reference image Pr from the storage unit 201, and transmits landmark information included in the reference image Pr to the terminal device 100. When the process of S206 is completed, the series of processes illustrated in FIG.

  (S207) The determination unit 205 transmits error information indicating that the landmark detection has failed to the terminal device 100. When the process of S207 is completed, the series of processes illustrated in FIG.

Here, the process of S203 will be further described.
(S221) The distance calculation unit 203 selects one feature point Wr of the reference image Pr.
(S222) The distance calculation unit 203, based on the local feature amount of the input image Pin received by the receiving unit 202, the distance D (feature of each feature region Ain of the input image Pin and each feature region Ar of the reference image Pr). Difference). For example, the distance calculation unit 203 calculates a bit string HV from the local feature amounts LFin and LFr according to the above equation (1). However, LFin is a local feature amount of the feature region Ain, and LFr is a local feature amount of the feature region Ar.

  The distance calculation unit 203 calculates the length of the bit string HV (the number of bit values 1 included in the HV). The length of the bit string HV is a distance D representing the degree of feature difference between the feature areas Ain and Ar. That is, as the distance D is larger, the features of the feature regions Ain and Ar are greatly different, and as the distance D is smaller, the features of the feature regions Ain and Ar are similar. The distance D calculated by the distance calculation unit 203 is input to the voting processing unit 204.

  (S223) The voting processing unit 204 has the highest similarity (the distance D is the smallest) based on the distance D calculated for the set of the feature area Ar of the reference image Pr and each feature area Ain of the input image Pin. ) A set of feature areas Ar and Ain is extracted. Further, the voting processing unit 204 identifies a set of feature points Win and Wr corresponding to the extracted set. The voting processing unit 204 detects (specifies) the position (specific point) in the input image Pin where the center of the reference image Pr is located based on the positional relationship between the feature point Wr of the feature area Ar and the center of the reference image Pr. .

  (S224) The voting processing unit 204 votes for the position (the grid) of the voting map MP corresponding to the specific point specified in S223. That is, the voting processing unit 204 extracts a cell of the voting map MP including the coordinates of the specific point, and counts up the count value (the number of votes) of the counter corresponding to the extracted cell.

  (S225) The voting processing unit 204 determines whether or not all feature points included in the reference image Pr have been selected. If all feature points included in the reference image Pr have not been selected, the process proceeds to S221. On the other hand, if all feature points included in the reference image Pr have been selected, the process proceeds to S226.

(S226) The determination unit 205 extracts the maximum number of votes from the vote map MP.
(S227) The determination unit 205 determines whether or not the maximum value of the number of votes extracted in S226 is larger than the threshold value Th. When the maximum value of the number of votes is larger than the threshold value Th, the process proceeds to S229. On the other hand, when the maximum value of the number of votes is not larger than the threshold value Th, the process proceeds to S228.

  (S228) The determination unit 205 determines that the reference image Pr does not exist in the input image Pin as the processing result of S203. When the process of S228 is completed, the series of processes shown in FIG. 24 ends.

  (S229) The determination unit 205 determines that the reference image Pr exists in the input image Pin as the processing result of S203. When the process of S229 is completed, the series of processes shown in FIG.

Here, the process of S222 will be further described.
(S231) The distance calculation unit 203, for a plurality of feature points Win (m) (m = 1, 2,..., M) arranged in the input image Pin, feature points having an upper left as a start point and a lower right as an end point. An index j indicating the position of Win is initialized to 1. That is, with Win (1) as the upper left feature point, the position of the feature point moves to the right as j increases, the position of the feature point located at the right end moves to the feature point at the lower left end, The position of the feature point Win (j) is specified according to j so that Win (M) is the lower right feature point.

  (S232) The distance calculation unit 203 determines the distance DA between the group A bit string of the local feature values of the feature area Ar (n) to be processed and the group A bit string of the local feature values of the feature area Ain (j). Calculate That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the A group as the distance DA.

(S233) The distance calculation unit 203 determines whether there is a divertable calculation result for the group B.
(Decision # 1) The distance calculation unit 203 determines whether or not there is an overlapping region of the feature regions Ain (j) and Ain (j−1). For example, when j is 1, since there is no Ain (j−1), the distance calculation unit 203 determines that there is no overlapping region. When the feature area Ain (j) is located at the left end, the feature areas Ain (j) and Ain (j−1) do not overlap, and the distance calculation unit 203 determines that there is no overlap area.

  (Decision # 2) When it is determined in the determination # 1 that there is an overlapping area, the distance calculation unit 203 determines whether there is an overlapping area of the feature areas Ar (n) and Ar (n−1). For example, when Ar (n) is located at the upper left, there is no Ar (n−1), and therefore the distance calculation unit 203 determines that there is no overlapping region. When the feature area Ar (n) is located at the left end, the feature areas Ar (n) and Ar (n−1) do not overlap, and the distance calculation unit 203 determines that there is no overlap area.

  When it is determined in the determination # 1 that there is no overlapping area of the feature areas Ain (j) and Ain (j−1), the distance calculation unit 203 determines that there is no divertable calculation result. Also, when it is determined in the determination # 2 that there is no overlapping area of the feature areas Ar (n) and Ar (n−1), the distance calculation unit 203 determines that there is no calculation result that can be used. On the other hand, when it is determined in the determination # 2 that there is an overlapping area of the feature areas Ar (n) and Ar (n−1), the distance calculation unit 203 determines that there is a divertable calculation result.

If it is determined that there is a divertable calculation result, the process proceeds to S235. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S234.
(S234) The distance calculation unit 203 calculates the distance DB between the bit string of the B group among the local feature values of the feature area Ar (n) to be processed and the bit string of the B group among the local feature values of the feature area Ain (j). Calculate (j, n). That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the group B as the distance DB (j, n).

  (S235) The distance calculation unit 203 is a distance DB between the bit string of the B group among the local feature values of the feature area Ar (n) to be processed and the bit string of the B group among the local feature values of the feature area Ain (j). A divertable calculation result calculated in the past is used as (j, n). That is, the distance calculation unit 203 determines the distance DC () between the bit string of the C group of the local feature amount of the feature area Ar (n−1) and the bit string of the C group of the local feature quantity of the feature area Ain (j−1). j-1, n-1) is defined as a distance DB (j, n). The distance DC (j-1, n-1) is stored in the distance recording table.

  (S236) The distance calculation unit 203 calculates the distance DC between the C group bit string of the local feature values of the feature area Ar (n) to be processed and the C group bit string of the local feature values of the feature area Ain (j). Calculate (j, n). That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the group C as the distance DC (j, n).

  (S237) The distance calculation unit 203 stores the distance DC (j, n) calculated in S236 in the distance recording table. At this time, the distance calculation unit 203 stores the distance DC (j, n) in the d (j, n) column of the distance recording table.

  (S238) The distance calculation unit 203 adds the distance DA (j, n), the distance DB (j, n), and the distance DC (j, n) to calculate the feature regions Ain (j) and Ar (n). A distance D (j, n) is obtained.

(S239) The distance calculation unit 203 increments the index j (j = j + 1). That is, the distance calculation unit 203 moves the feature area Ain to be processed.
(S240) The distance calculation unit 203 determines whether j is greater than N (j> N). When j> N, the series of processes shown in FIG. 25 ends. On the other hand, if j> N is not true, the process proceeds to S232.

  As described above, when there is a calculation result that can be used for the overlapping region, the calculation result can be used (S235) and a part of the calculation (S234) can be omitted to reduce the load of distance calculation. As a result, it is possible to reduce the processing load related to the corresponding point search and to speed up the processing.

The flow of processing related to the operation of the server device 200 has been described above.
[2-6. Modification # 1 (Example of Enlarging Overlapping Area)]
Here, a modification (modification # 1) according to the second embodiment will be described. Modification # 1 relates to a method for increasing the ratio of overlapping regions to feature regions. For example, the technique of the modified example # 1 is applied when the feature points are arranged more densely than the feature point arrangement illustrated in FIG. 6 or when the size of the feature region is increased without changing the feature point density. it can. Here, a case where feature points are densely arranged will be described as an example with reference to FIGS. 26 and 27.

  FIG. 26 is a diagram for explaining the arrangement of feature points according to a modification (Modification # 1) of the second embodiment. FIG. 27 is a diagram for explaining the arrangement of pixel pairs and the structure of local feature amounts according to a modified example (modified example # 1) of the second embodiment.

  In the example of FIG. 6 which has already been described, the overlapping region between adjacent feature regions Ain is set to a range of 1/2 of the feature region Ain. On the other hand, in the example of FIG. 26, the overlapping area between adjacent characteristic areas Ain is set to a range of 3/4 of the characteristic area Ain. In this case, a part of the feature areas Ain (2), Ain (3), and Ain (4) overlaps the feature area Ain (1). Therefore, the arrangement of the pixel pairs is set in consideration of which of the small regions obtained by dividing the feature region Ain into four as shown in FIG.

  In the example of FIG. 9 already described, one pixel is in the right region, the other pixel is in the left region, the pixel pair is A group, both pixels are in the left region, the pixel pair is B group, and both pixels are in the right region. The pixel pair classification in which the pixel pair located at C is the C group is shown. On the other hand, the example of FIG. 27 classifies the A group, the B group, the C group, and the D group by the number of lines connecting the pixel pairs across the boundaries of the small regions, and further sets the B group to 2 according to the pixel position. The grouping is divided into three groups, group C is divided into three groups, and group D is divided into four groups. Hereinafter, the small areas are referred to as small area # 1, small area # 2, small area # 3, and small area # 4 in order from the left.

(About Group A)
In the pixel pair belonging to the A group, one pixel is located in the small area # 1, and the other pixel is located in the small area # 4. Therefore, a line connecting pixel pairs belonging to the A group straddles three boundaries.

(About Group B)
The B group is divided into a B1 group and a B2 group. In the pixel pair belonging to the group B1, one pixel is located in the small area # 1, and the other pixel is located in the small area # 3. In the pixel pair belonging to the group B2, one pixel is located in the small area # 2, and the other pixel is located in the small area # 4. Therefore, the line connecting the pixel pairs belonging to the groups B1 and B2 straddles two boundaries.

  The pixel pairs in the B1 group and the pixel pairs in the B2 group are set to have the same arrangement. Therefore, the B2 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D regarding the B1 group.

  For example, the bit string of the B2 group in the feature area Ain (1) matches the bit string of the B1 group in the feature area Ain (2). When calculating the local feature amount of the feature region Ain (2), the bit string of the group B2 of the feature region Ain (1) can be used. In addition, among the distances D (1,1) between the feature regions Ar (1) and Ain (1), the distance DB2 (1,1) of the B2 group is the distance D (between the feature regions Ar (2) and Ain (2). 2, 2) coincides with the distance DB1 (2, 2) of the B1 group. Therefore, the distance DB2 (1, 1) can be used when calculating the distance D (2, 2).

(About Group C)
Group C is divided into group C1, group C2, and group C3. In the pixel pair belonging to the group C1, one pixel is located in the small area # 1, and the other pixel is located in the small area # 2. In the pixel pair belonging to the group C2, one pixel is located in the small area # 2, and the other pixel is located in the small area # 3. In the pixel pair belonging to the group C3, one pixel is located in the small region # 3 and the other pixel is located in the small region # 4. Therefore, the line connecting the pixel pairs belonging to the groups C1, C2, and C3 straddles one boundary.

  The pixel pair of the C1 group, the pixel pair of the C2 group, and the pixel pair of the C3 group are set to have the same arrangement. Therefore, the C2 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D related to the C1 group. The C3 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D related to the C1 group or the C2 group.

  For example, the bit string of the C2 group of the feature area Ain (1) matches the bit string of the C1 group of the feature area Ain (2). The bit string of the C3 group of the feature area Ain (1) matches the bit string of the C2 group of the feature area Ain (2) and the bit string of the C1 group of the feature area Ain (3). Therefore, when calculating the local feature amount of the feature region Ain (2), the bit strings of the C2 group and the C3 group of the feature region Ain (1) can be used. Further, when calculating the local feature amount of the feature region Ain (3), the bit string of the C3 group of the feature region Ain (1) can be diverted.

  Further, among the distances D (1, 1) between the feature regions Ar (1) and Ain (1), the distance DC2 (1, 1) of the C2 group is the distance D () between the feature regions Ar (2) and Ain (2). 2, 2) which is the same as the distance DC1 (2, 2) of the C1 group. Among the distances D (1,1) between the feature areas Ar (1) and Ain (1), the distance DC3 (1,1) of the C3 group is the distance D (2,2) between the feature areas Ar (2) and Ain (2). It corresponds to the distance DC2 (2, 2) of the C2 group in 2). Therefore, the distances DC2 (1,1) and DC3 (1,1) can be used when calculating the distance D (2,2).

  In addition, among the distances D (1,1) between the characteristic regions Ar (1) and Ain (1), the distance DC3 (1,1) of the C3 group is the distance D (between the characteristic regions Ar (3) and Ain (3). 3, 3) which is the same as the distance DC1 (3, 3) of the C1 group. Therefore, the distance DC3 (1, 1) can be used when calculating the distance D (3, 3).

(About Group D)
The D group is divided into a D1, D2, D3, and D4 groups. In the pixel pair belonging to the D1 group, both pixels are located in the small region # 1. In the pixel pair belonging to the group D2, both pixels are located in the small area # 2. In the pixel pair belonging to the group D3, both pixels are located in the small region # 3. In the pixel pair belonging to the group D4, both pixels are located in the small region # 4. That is, a line connecting pixel pairs belonging to the D1, D2, D3, and D4 groups does not straddle the boundary.

  The pixel pair of the D1 group, the pixel pair of the D2 group, the pixel pair of the D3 group, and the pixel pair of the D4 group are set to have the same arrangement. Therefore, the D2 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D regarding the D1 group. The D3 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D regarding the D1 group or the D2 group. The D4 group gives a bit string or a part of the distance D (distance d) that can be used when calculating the local feature amount or the distance D regarding the D1, D2, and D3 groups.

  For example, the bit string of the D2 group in the feature area Ain (1) matches the bit string of the D1 group in the feature area Ain (2). The bit string of the D3 group in the feature area Ain (1) matches the bit string of the D2 group in the feature area Ain (2) and the bit string of the D1 group in the feature area Ain (3). The bit string of the D4 group of the feature area Ain (1) matches the bit string of the D3 group of the feature area Ain (2), the bit string of the D2 group of the feature area Ain (3), and the bit string of the D1 group of the feature area Ain (4) To do.

  Therefore, when calculating the local feature amount of the feature region Ain (2), the bit strings of the D2, D3, and D4 groups of the feature region Ain (1) can be used. Further, when calculating the local feature amount of the feature region Ain (3), the bit strings of the D3 group and the D4 group of the feature region Ain (1) can be used. In addition, when calculating the local feature amount of the feature region Ain (4), the bit string of the D4 group of the feature region Ain (1) can be used.

  Further, among the distances D (1, 1) of the feature areas Ar (1) and Ain (1), the distance DD2 (1, 1) of the D2 group is the distance D () of the feature areas Ar (2) and Ain (2). 2, 2) coincides with the distance DD1 (2, 2) of the D1 group. Further, among the distances D (1, 1) of the feature areas Ar (1) and Ain (1), the distance DD3 (1, 1) of the group D3 is the distance D () of the feature areas Ar (2) and Ain (2). 2, 2), which matches the distance DD2 (2, 2) of the D2 group. Further, the distance DD3 (1, 1) matches the distance DD1 (3, 3) of the D1 group among the distances D (3, 3) of the feature regions Ar (3), Ain (3).

  The distance DD4 (1,1) of the D4 group out of the distance D (1,1) between the feature regions Ar (1) and Ain (1) is the distance D () between the feature regions Ar (2) and Ain (2). 2, 2), which coincides with the distance DD3 (2, 2) of the D3 group. Furthermore, the distance DD4 (1, 1) matches the distance DD2 (3, 3) of the D2 group among the distances D (3, 3) of the feature regions Ar (3), Ain (3). The distance DD4 (1, 1) matches the distance DD1 (4, 4) of the D1 group among the distances D (4, 4) of the feature regions Ar (4), Ain (4).

  Therefore, when calculating the distance D (2, 2), the distances DD2 (1, 1), DD3 (1, 1), and DD4 (1, 1) can be used. Further, when calculating the distance D (3, 3), the distances DD3 (1, 1) and DD4 (1, 1) can be used. Furthermore, the distance DD4 (1, 1) can be used when calculating the distance D (4, 4).

  In order to facilitate the above-mentioned diversion, as shown in FIG. 27, the local feature amounts are obtained by combining bit groups of A group, B1 group, B2 group, C1 group,..., C3 group, D1 group,. It is expressed as a bit string. The distance D is expressed by a value obtained by adding the distances between bit strings of the same group. In the case of the modified example # 1, the divertable bit string and the type of the distance d are changed as described above, but the basic flow of the processing flow described in FIGS. 22 and 25 is the same.

The modification # 1 has been described above.
[2-7. Modification # 2 (Arrangement in which part of the feature region overlaps two-dimensionally)]
Next, a modification (modification # 2) according to the second embodiment will be described. The modified example # 2 relates to a method of arranging the feature regions adjacent vertically and horizontally so as to partially overlap.

(2-7-1. Feature points, pixel pairs, distance recording table)
With reference to FIGS. 28 to 30, the arrangement of feature points, the arrangement of pixel pairs, the setting of a distance recording table, and the like according to the modified example # 2 will be described.

  FIG. 28 is a diagram for describing arrangement of feature points according to a modification (Modification # 2) of the second embodiment. FIG. 29 is a diagram for explaining the arrangement of pixel pairs and the structure of local feature amounts according to a modification (modification # 2) of the second embodiment. FIG. 30 is a diagram illustrating an example of distance information according to a modification (Modification # 2) of the second embodiment.

  The example of FIG. 28 shows a case where the arrangement of the feature points Win and the size of the feature area Ain are set so that the feature areas Ain that are adjacent in the vertical and horizontal directions overlap in the range of ½ of the feature area Ain. In this example, the position of the feature point Win is expressed using a two-dimensional coordinate display. The same applies to the position of the feature region Ain.

  For example, the feature point at the upper left corner is Win (1,1), the feature point right next to Win (1,1) is Win (1,2), and the feature point below Win (1,1) is Win (2 , 1). Further, the right region of the feature region Ain (1, 1) and the left region of the feature region Ain (1, 2) overlap, and the lower half region (lower region) of the feature region Ain (1, 1) The upper half region (upper region) of the feature region Ain (2, 1) overlaps. Furthermore, the lower left quarter area (lower left area) of the feature area Ain (1, 2) and the upper right quarter area (upper right area) of the feature area Ain (2, 1) overlap. As described above, the modified example # 2 considers overlapping regions having various shapes and sizes.

  In the case of the modified example # 2, as shown in FIG. 29, pixel pairs are classified using four small regions obtained by dividing the feature region Ain into four parts in the vertical and horizontal directions. Here, the upper right small area (first quadrant) is small area # 1, the upper left small area (second quadrant) is small area # 2, the lower left small area (third quadrant) is # 3, and the lower right small area is # 3. The region (fourth quadrant) is denoted as # 4.

(About Group A)
Group A is a set of pixel pairs each having a pixel in two small areas arranged on a diagonal line. That is, in the group A, a pixel pair in which one pixel is in the small region # 1 and the other pixel is in the small region # 3, and one pixel is in the small region # 2, and the other pixel is the small region # 4. Includes the pixel pair.

(About Group B)
Group B is a set of pixel pairs in which pixels are respectively located in two small areas arranged above and below. The B group is divided into a B1 group and a B2 group. The B1 group is a set of pixel pairs in which pixels are located in small areas # 2 and # 3 that divide the left area into upper and lower parts. The B2 group is a set of pixel pairs in which pixels are located in small areas # 1 and # 4 that divide the right area into upper and lower parts. The pixel pairs in the B1 group and the pixel pairs in the B2 group are set to have the same arrangement.

(About Group C)
Group C is a set of pixel pairs in which pixels are respectively located in two small regions arranged on the left and right. Group C is divided into group C1 and group C2. Group C1 is a set of pixel pairs in which pixels are located in small areas # 1 and # 2 that divide the upper area into left and right. Group C2 is a set of pixel pairs in which pixels are located in small regions # 3 and # 4 that divide the lower region into left and right. The pixel pairs in the C1 group and the pixel pairs in the C2 group are set to have the same arrangement.

(About Group D)
Group D is a set of pixel pairs in which two pixels are located in each small region. The D group is divided into a D1, D2, D3, and D4 groups. The D1 group is a pixel pair in which each pixel is located in the small area # 1. The group D2 is a pixel pair in which each pixel is located in the small area # 2. The group D3 is a pixel pair in which each pixel is located in the small area # 3. The D4 group is a pixel pair in which each pixel is located in the small region # 4. The pixel pairs of the D1, D2, D3, and D4 groups are set to have the same arrangement.

  Also in the modification # 2, the local feature amount is expressed as a bit string obtained by combining the bit strings of the A group, the B1 group, the B2 group, the C1 group, the C2 group, the D1 group,. The distance D is expressed by a value obtained by adding the distances between bit strings of the same group. Further, as shown in FIG. 30, four types of distance recording tables for storing the divertable distance d are prepared: a B2 group table, a C2 group table, a D3 group table, and a D4 group table.

(2-7-2. Operation of server device)
Here, the operation of the server apparatus 200 according to the modified example # 2 will be described with reference to FIGS. However, the part which is mainly deformed in the modification # 2 is in the flow of the processing related to S222 described above. Therefore, this part will be described.

  FIG. 31 is a first flowchart for explaining operations related to distance calculation among the operations of the server device according to the modified example (modified example # 2) of the second embodiment. FIG. 32 is a second flowchart for explaining the operation related to the distance calculation among the operations of the server device according to the modified example (modified example # 2) of the second embodiment. FIG. 33 is a third flowchart for explaining operations related to distance calculation among the operations of the server apparatus according to the modified example (modified example # 2) of the second embodiment.

  (S301) The distance calculation unit 203, for a plurality of feature points Win (m) (m = 1, 2,..., M) arranged in the input image Pin, feature points having the upper left as a start point and the lower right as an end point. Indexes jx and gy indicating the position of Win are initialized to 1. The coordinates (jx, jy) represented by the indexes jx and Jy represent the two-dimensional position of the feature point Win (see FIG. 28). However, the coordinate (1, 1) is the position of the feature point at the upper left corner. Similarly, the feature point Wr arranged in the reference image Pr also expresses the position using two-dimensional coordinates (nx, ny).

  (S302) The distance calculation unit 203 includes a group A bit string of the local feature values of the feature area Ar (nx, ny) to be processed and a bit string of the group A of the local feature values of the feature area Ain (jx, jy). And calculate the distance DA. That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the A group as the distance DA.

(S303) The distance calculation unit 203 determines whether there is a divertable calculation result for the B1 group.
The distance calculation unit 203 determines whether or not there are feature regions Ain (jx−1, gy) and Ar (nx−1, ny). For example, when the feature area Ain (jx, gy) is located at the left end (when jx = 1), there is no Ain (jx−1, jy). Similarly, when Ar (nx, ny) is located at the left end (when nx = 1), there is no Ar (nx-1, ny).

  When there are feature regions Ain (jx-1, yy) and Ar (nx-1, ny), the distance calculation unit 203 determines that there is a divertable calculation result for the B1 group. When at least Ain (jx-1, yy) or Ar (nx-1, ny) is absent, the distance calculation unit 203 determines that there is no calculation result applicable to the B1 group.

If it is determined that there is a divertable calculation result, the process proceeds to S305. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S304.
(S304) The distance calculation unit 203 includes a bit string of the B1 group among the local feature values of the feature area Ar (nx, ny) to be processed and a bit string of the B1 group among the local feature values of the feature area Ain (jx, jy). Distance DB1 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates an exclusive OR of the bit strings of the B1 group as the distance DB1 (jx, yy, nx, ny).

  (S305) The distance calculation unit 203 uses a divertable calculation result calculated in the past as the distance DB1 (jx, jy, nx, ny). That is, the distance calculation unit 203 includes a bit string of the group B2 of the local feature amount of the feature area Ar (nx-1, ny) and a bit string of the group B2 of the local feature quantity of the feature area Ain (jx-1, yy). The distance DB2 (jx-1, jy, nx-1, ny) is defined as the distance DB1 (jx, jy, nx, ny). The distance DB2 (jx-1, jy, nx-1, ny) is stored in the column d (jx-1, jy, nx-1, ny) of the B2 group table.

  (S306) The distance calculation unit 203 uses the B2 group bit string of the local feature values of the feature area Ar (nx, ny) to be processed and the B2 group bit string of the local feature values of the feature area Ain (jx, jy). Distance DB2 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the B2 group as the distance DB2 (jx, yy, nx, ny).

  (S307) The distance calculation unit 203 stores the distance DB2 (jx, yy, nx, ny) calculated in S306 in the B2 group table. At this time, the distance calculation unit 203 stores the distance DB2 (jx, jy, nx, ny) in the d (jx, jy, nx, ny) column of the B2 group table.

(S308) The distance calculation unit 203 determines whether there is a divertable calculation result for the C1 group.
The distance calculation unit 203 determines whether or not there are feature regions Ain (jx, gy−1) and Ar (nx, ny−1). For example, when the feature area Ain (jx, jy) is located at the upper end (when jy = 1), there is no Ain (jx, jy-1). Similarly, when Ar (nx, ny) is located at the upper end (when ny = 1), there is no Ar (nx, ny-1).

  When there are feature areas Ain (jx, gy−1) and Ar (nx, ny−1), the distance calculation unit 203 determines that there is a divertable calculation result for the C1 group. When there is no Ain (jx, gy-1) or Ar (nx, ny-1), the distance calculation unit 203 determines that there is no divertable calculation result for the C1 group.

If it is determined that there is a divertable calculation result, the process proceeds to S310. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S309.
(S309) The distance calculation unit 203 includes a bit string of the C1 group among the local feature values of the feature area Ar (nx, ny) to be processed and a bit string of the C1 group among the local feature quantities of the feature area Ain (jx, yy). The distance DC1 (jx, jy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates an exclusive OR of the bit strings of the C1 group as the distance DC1 (jx, yy, nx, ny). When the process of S309 is completed, the process proceeds to S311 in FIG.

  (S310) The distance calculation unit 203 uses a divertable calculation result calculated in the past as the distance DC1 (jx, jy, nx, ny). That is, the distance calculation unit 203 includes the bit string of the C2 group of the local feature amount of the feature area Ar (nx, ny-1) and the bit string of the C2 group of the local feature quantity of the feature area Ain (jx, zy-1). The distance DC2 (jx, jy-1, nx, ny-1) is defined as the distance DC1 (jx, jy, nx, ny). The distance DC2 (jx, jy-1, nx, ny-1) is stored in the d (jx, jy-1, nx, ny-1) column of the C2 group table. When the process of S310 is completed, the process proceeds to S311 in FIG.

  (S311) The distance calculation unit 203 uses the C2 group bit string of the local feature values of the feature area Ar (nx, ny) to be processed and the C2 group bit string of the local feature values of the feature area Ain (jx, jy). The distance DC2 (jx, jy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the C2 group as the distance DC2 (jx, gy, nx, ny).

  (S312) The distance calculation unit 203 stores the distance DC2 (jx, yy, nx, ny) calculated in S311 in the C2 group table. At this time, the distance calculation unit 203 stores the distance DC2 (jx, jy, nx, ny) in the d (jx, jy, nx, ny) column of the C2 group table.

(S313) The distance calculation unit 203 determines whether there is a divertable calculation result for the D1 group.
The distance calculation unit 203 determines whether or not there are feature regions Ain (jx, gy−1) and Ar (nx, ny−1). For example, when the feature area Ain (jx, jy) is located at the upper end (when jy = 1), there is no Ain (jx, jy-1). Similarly, when Ar (nx, ny) is located at the upper end (when ny = 1), there is no Ar (nx, ny-1).

  When there are feature regions Ain (jx, gy−1) and Ar (nx, ny−1), the distance calculation unit 203 determines that there is a divertable calculation result for the D1 group. If there is no Ain (jx, gy-1) or Ar (nx, ny-1), the distance calculation unit 203 determines that there is no divertable calculation result for the D1 group.

If it is determined that there is a divertable calculation result, the process proceeds to S315. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S314.
(S314) The distance calculation unit 203 includes a bit string of the D1 group among the local feature values of the feature area Ar (nx, ny) to be processed and a bit string of the D1 group among the local feature quantities of the feature area Ain (jx, yy). The distance DD1 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates an exclusive OR of the bit strings of the D1 group as the distance DD1 (jx, yy, nx, ny).

  (S315) The distance calculation unit 203 uses a divertable calculation result calculated in the past as the distance DD1 (jx, jy, nx, ny). That is, the distance calculation unit 203 uses the D4 group bit string of the local feature amount of the feature area Ar (nx, ny-1) and the D4 group bit string of the local feature quantity of the feature area Ain (jx, jy-1). The distance DD4 (jx, jy-1, nx, ny-1) is a distance DD1 (jx, gy, nx, ny). The distance DD4 (jx, jy-1, nx, ny-1) is stored in the d (jx, jy-1, nx, ny-1) column of the D4 group table.

  (S316) The distance calculation unit 203 uses the D4 group bit string of the local feature values of the feature area Ar (nx, ny) to be processed and the D4 group bit string of the local feature values of the feature area Ain (jx, jy). The distance DD4 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the D4 group as the distance DD4 (jx, yy, nx, ny).

  (S317) The distance calculation unit 203 stores the distance DD4 (jx, yy, nx, ny) calculated in S316 in the D4 group table. At this time, the distance calculation unit 203 stores the distance DD4 (jx, jy, nx, ny) in the d (jx, jy, nx, ny) column of the D4 group table.

(S318) The distance calculation unit 203 determines whether there is a divertable calculation result for the D2 group.
The distance calculation unit 203 determines whether or not there are feature regions Ain (jx, gy−1) and Ar (nx, ny−1). For example, when the feature area Ain (jx, jy) is located at the upper end (when jy = 1), there is no Ain (jx, jy-1). Similarly, when Ar (nx, ny) is located at the upper end (when ny = 1), there is no Ar (nx, ny-1).

  When there are feature regions Ain (jx, gy−1) and Ar (nx, ny−1), the distance calculation unit 203 determines that there is a divertable calculation result for the D2 group. When there is no Ain (jx, gy-1) or Ar (nx, ny-1), the distance calculation unit 203 determines that there is no divertable calculation result for the D2 group.

If it is determined that there is a divertable calculation result, the process proceeds to S320. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S319.
(S319) The distance calculation unit 203 includes a bit string of the D2 group among the local feature values of the feature area Ar (nx, ny) to be processed and a bit string of the D2 group among the local feature values of the feature area Ain (jx, jy). The distance DD2 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates an exclusive OR between the bit strings of the D2 group as the distance DD2 (jx, yy, nx, ny). When the process of S319 is completed, the process proceeds to S321 in FIG.

  (S320) The distance calculation unit 203 uses a divertable calculation result calculated in the past as the distance DD2 (jx, jy, nx, ny). That is, the distance calculation unit 203 includes a bit string of the D3 group of the local feature amount of the feature area Ar (nx, ny-1) and a bit string of the D3 group of the local feature quantity of the feature area Ain (jx, jy-1). The distance DD3 (jx, gy-1, nx, ny-1) is set as the distance DD2 (jx, gy, nx, ny). The distance DD3 (jx, jy-1, nx, ny-1) is stored in the d (jx, jy-1, nx, ny-1) column of the D3 group table. When the process of S320 is completed, the process proceeds to S321 in FIG.

(S321) The distance calculation unit 203 determines whether there is a divertable calculation result for the D4 group.
The distance calculation unit 203 determines whether or not there are feature regions Ain (jx−1, gy) and Ar (nx−1, ny). For example, when the feature area Ain (jx, gy) is located at the left end (when jx = 1), there is no Ain (jx-1, yy). Similarly, when Ar (nx, ny) is located at the left end (when nx = 1), there is no Ar (nx-1, ny).

  When there are feature regions Ain (jx-1, yy) and Ar (nx-1, ny), the distance calculation unit 203 determines that there is a divertable calculation result for the D4 group. When at least Ain (jx-1, yy) or Ar (nx-1, ny) is absent, the distance calculation unit 203 determines that there is no calculation result applicable to the D4 group.

If it is determined that there is a divertable calculation result, the process proceeds to S323. On the other hand, if it is determined that there is no divertable calculation result, the process proceeds to S322.
(S322) The distance calculation unit 203 uses the D3 group bit string of the local feature values of the feature area Ar (nx, ny) to be processed and the D3 group bit string of the local feature values of the feature area Ain (jx, jy). Distance DD3 (jx, gy, nx, ny) is calculated. That is, the distance calculation unit 203 calculates the exclusive OR of the bit strings of the D3 group as the distance DD3 (jx, yy, nx, ny). When the process of S322 is completed, the process proceeds to S324.

  (S323) The distance calculation unit 203 uses a divertable calculation result calculated in the past as the distance DD3 (jx, jy, nx, ny). That is, the distance calculation unit 203 includes the bit string of the D4 group of the local feature amount of the feature area Ar (nx-1, ny) and the bit string of the D4 group of the local feature quantity of the feature area Ain (jx-1, yy). The distance DD4 (jx-1, yy, nx-1, ny) is defined as the distance DD3 (jx, jy, nx, ny). The distance DD4 (jx-1, jy, nx-1, ny) is stored in the d (jx, jy-1, nx, ny-1) column of the D4 group table.

  (S324) The distance calculation unit 203 adds the distance DA (jx, jy, nx, ny), the distance DB1 (jx, jy, nx, ny), ..., the distance DD4 (jx, jy, nx, ny). Then, the distance calculation unit 203 sets the addition value as the distance D (jx, gy, nx, ny) between the feature areas Ain (jx, yy) and Ar (nx, ny).

  (S325) The distance calculation unit 203 updates indexes jx and jy representing the coordinates (jx, jy) of the feature points. For example, when jx <Mx, the distance calculation unit 203 increments jx (jx = jx + 1). When jx = Mx, the distance calculation unit 203 updates jx to 1 and increments jy (jy = jy + 1). That is, the distance calculation unit 203 moves the feature area Ain to be processed. However, the maximum number of jx is Mx.

  (S326) The distance calculation unit 203 determines whether or not the processing has been completed for all feature points Win. When the processing has been completed for all the feature points Win, the series of processing illustrated in FIGS. 31 to 33 ends. On the other hand, if there is a feature point Win that has not been processed, the process proceeds to S302 in FIG.

  As described above, when there is a calculation result that can be diverted for the overlapping region, the calculation result is diverted (S305, S310, S315, S320, S323), and a part of the calculation (S304, S309, S314, S319, S322) is performed. ) Is omitted, the load of distance calculation is reduced. As a result, it is possible to reduce the processing load related to the corresponding point search and to speed up the processing.

The modification # 2 has been described above.
As described above, by applying the technology according to the second embodiment, it is possible to reduce the processing load for the corresponding point search and to determine the presence or absence of the target object at high speed.

The second embodiment has been described above.
In the above description, the landmark recognition function has been described as an example. However, the present invention can be applied to, for example, an arbitrary application program that involves image similarity determination. Further, the present invention can be applied to an application program for performing face recognition by the server device 200, an application program using AR (Augmented Reality), and the like. Such application examples naturally belong to the technical scope of the second embodiment.

<3. Addendum>
The following additional notes are disclosed with respect to the embodiment described above.
(Supplementary Note 1) Among the plurality of pixel pairs arranged in a common arrangement rule in each of a plurality of image areas partially overlapping, each of the image areas arranged in each image area in the overlapping area where the image areas overlap A storage unit for storing arrangement information representing the arrangement rule set so that the positions of the pixel pairs match;
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount A calculation unit that determines similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
The calculation unit stores a calculation result related to the feature amount of the pixel pair located in the overlap region in the storage unit, and relates to the feature amount of the pixel pair in the other image region at the same position as the pixel pair. An information processing apparatus that uses the calculation result stored in the storage unit during calculation.

(Additional remark 2) The said memory | storage part memorize | stores the feature-value of each said image area | region set to the said 1st image,
The calculation unit calculates the feature amount of each pixel pair arranged in the arrangement rule in each image region set in the second image, and the feature amount of the pixel pair located in the overlap region Is stored in the storage unit, and the feature amount of the pixel pair stored in the storage unit is used while avoiding the calculation of the feature amount of the other pixel pair at the same position as the pixel pair. The information processing apparatus described.

(Supplementary Note 3) The storage unit includes a set of pixel pairs in which both pixels are included in the overlapping region, a set of pixel pairs in which one pixel is included in the overlapping region, and both pixels in the overlapping region. Further storing set information for specifying the set of pixel pairs not included in
The information processing apparatus according to appendix 1 or 2, wherein the calculation unit uses the pixel pair in which both pixels are included in the overlapping area as the pixel pair located in the overlapping area based on the set information.

(Appendix 4) The computer
The position of each pixel pair arranged in each image region in the overlapping region where the image regions overlap among the plurality of pixel pairs arranged in a common arrangement rule in each of the plurality of image regions partially overlapping To obtain the placement information representing the placement rule set to match,
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount Executing a process of determining similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
In the process of calculating the distance, the calculation result relating to the feature amount of the pixel pair located in the overlap region is stored in the storage unit, and the feature of the pixel pair in the other image region at the same position as the pixel pair An image determination method that uses the calculation result stored in the storage unit when calculating a quantity.

(Supplementary note 5)
The position of each pixel pair arranged in each image region in the overlapping region where the image regions overlap among the plurality of pixel pairs arranged in a common arrangement rule in each of the plurality of image regions partially overlapping To obtain the placement information representing the placement rule set to match,
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount A process of determining similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
In the process of calculating the distance, the calculation result relating to the feature amount of the pixel pair located in the overlap region is stored in the storage unit, and the feature of the pixel pair in the other image region at the same position as the pixel pair A program for executing a process of using the calculation result stored in the storage unit when calculating the quantity.

  (Supplementary Note 6) A computer-readable recording medium in which the program according to Supplementary Note 5 is stored.

10 Information processing device 11 Storage unit 12 Calculation unit Ain1, Ain2, Ar1, Ar2 Image region dA11, dB11, dC11, dA22, dB22, dC22 Distance Qin1, Qr1, Qin2, Qr2 Feature amount R1 Left region R2 Right region

Claims (5)

The position of each pixel pair arranged in each image region in the overlapping region where the image regions overlap among the plurality of pixel pairs arranged in a common arrangement rule in each of the plurality of image regions partially overlapping A storage unit for storing arrangement information representing the arrangement rule set to match,
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount A calculation unit that determines similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
The calculation unit stores a calculation result related to the feature amount of the pixel pair located in the overlap region in the storage unit, and relates to the feature amount of the pixel pair in the other image region at the same position as the pixel pair. An information processing apparatus that uses the calculation result stored in the storage unit during calculation. The storage unit stores the feature amount of each image area set in the first image,
The calculation unit calculates the feature amount of each pixel pair arranged in the arrangement rule in each image region set in the second image, and the feature amount of the pixel pair located in the overlap region 2 is stored in the storage unit, and the feature amount of the pixel pair stored in the storage unit is used while avoiding the calculation of the feature amount of the other pixel pair at the same position as the pixel pair. The information processing apparatus described in 1. The storage unit includes a set of pixel pairs in which both pixels are included in the overlap region, a set of pixel pairs in which one pixel is included in the overlap region, and both pixels are not included in the overlap region. Further storing set information for specifying the set of pixel pairs;
The information processing apparatus according to claim 1, wherein the calculation unit uses the pixel pair in which both pixels are included in the overlapping area as the pixel pair located in the overlapping area based on the set information. . Computer
The position of each pixel pair arranged in each image region in the overlapping region where the image regions overlap among the plurality of pixel pairs arranged in a common arrangement rule in each of the plurality of image regions partially overlapping To obtain the placement information representing the placement rule set to match,
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount Executing a process of determining similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
In the process of calculating the distance, the calculation result relating to the feature amount of the pixel pair located in the overlap region is stored in the storage unit, and the feature of the pixel pair in the other image region at the same position as the pixel pair An image determination method that uses the calculation result stored in the storage unit when calculating a quantity. On the computer,
The position of each pixel pair arranged in each image region in the overlapping region where the image regions overlap among the plurality of pixel pairs arranged in a common arrangement rule in each of the plurality of image regions partially overlapping To obtain the placement information representing the placement rule set to match,
A feature amount set of each pixel pair arranged in the image region according to the arrangement rule is used as a feature amount of the image region, and a feature amount of each image region set in the first image and a second amount A process of determining similarity between the first image and the second image based on a distance from the feature amount of each image region set in the image;
In the process of calculating the distance, the calculation result relating to the feature amount of the pixel pair located in the overlap region is stored in the storage unit, and the feature of the pixel pair in the other image region at the same position as the pixel pair A program for executing a process of using the calculation result stored in the storage unit when calculating the quantity.

Images