JP2017037437A - Information processing system, information processor, information processing method and information processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable the execution of image retrieval that more accurately reflects user's intention.SOLUTION: An information processing system includes a terminal device, an image retrieval device, and an information processor. The image retrieval device includes a retrieval part for retrieving an image to be retrieved on the basis of similarity to a retrieval object image from a registration part in which the image to be retrieved is associated with cooperation information for registration, and a transmission part for transmitting the cooperation information associated with the retrieved image to be retrieved to the terminal device. The information processor registers the image to be retrieved in the registration part. The information processing system includes a region setting part for setting an attention region in the image to be retrieved, a weight setting part for setting weight in the attention region, and a change part for changing the similarity on the basis of the weight.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system, an information processing apparatus, an information processing method, and an information processing program.

  For example, an image search technique is known in which feature points are extracted from a captured image obtained by imaging with a camera, and an image having many similar feature points is searched from a previously registered image group. Using this image search technology, images of specific photos and printed materials are registered in a database in association with various electronic data, and similar images are searched from this database using captured images obtained by capturing these photos and printed materials. An image search system that presents cooperative information associated with similar images is known (for example, Patent Document 1).

  In an image search using the similarity of feature points, an image search may be performed based on feature points in a specific area in the search target image, and feature points outside the specific area may not be considered. In this case, if the part that the user wants to search is outside the specific area, it may be difficult to obtain a search result intended by the user.

  On the other hand, a method of excluding an image search target by masking a specific area is also conceivable. However, in this method, since the search is performed without considering the masked region at all, the search accuracy may be reduced.

  The present invention has been made in view of the above, and an object of the present invention is to enable execution of an image search that more accurately reflects the user's intention.

  In order to solve the above-described problems and achieve the object, the search target image is searched based on the similarity with the search target image from the registration unit that registers the search target image and the link information in association with each other. Information processing apparatus including an image search device including a search unit that performs search, a transmission unit that transmits cooperation information associated with the searched image to the terminal device, and an information processing device that registers the search image in the registration unit The system includes an area setting unit that sets a region of interest for a searched image, a weight setting unit that sets a weight for the region of interest, and a changing unit that changes the similarity based on the weight.

  According to the present invention, it is possible to perform an image search that more accurately reflects a user's intention.

FIG. 1 is a diagram schematically illustrating an exemplary configuration of an image search system applicable to each embodiment. FIG. 2 is a diagram schematically illustrating an exemplary configuration of an information processing system according to each embodiment. FIG. 3 is a block diagram illustrating a configuration example of hardware of a registration terminal applicable to each embodiment. FIG. 4 is a block diagram illustrating a configuration example of hardware applicable to the image search server according to each embodiment. FIG. 5 is a block diagram illustrating a configuration example of hardware of a search terminal applicable to each embodiment. FIG. 6 is a functional block diagram illustrating an example of functions of the registration terminal according to each embodiment. FIG. 7 is a functional block diagram of an example for explaining functions of a search terminal applicable to each embodiment. FIG. 8 is a flowchart illustrating an example of image registration processing according to the embodiment. FIG. 9 is a diagram illustrating an example of a login screen according to each embodiment. FIG. 10 is a diagram illustrating an example of a menu screen according to each embodiment. FIG. 11 is a diagram illustrating an example of a registration information list screen according to each embodiment. FIG. 12 is a diagram illustrating an example of a public setting screen according to each embodiment. FIG. 13 is a diagram illustrating an example of a campaign registration screen according to each embodiment. FIG. 14 is a diagram illustrating an example of an edit screen according to each embodiment. FIG. 15 is a diagram illustrating an example of a file selection screen according to each embodiment. FIG. 16 is a diagram illustrating an example of a campaign setting screen according to each embodiment. FIG. 17 is a diagram illustrating an example of a warning screen according to each embodiment. FIG. 18 is a diagram illustrating an example of an XML code in which information indicating a target area is described according to each embodiment. FIG. 19 is a diagram illustrating an example of a search target image. FIG. 20 is a flowchart illustrating an example of attention area setting and weighting processing according to the first embodiment. FIG. 21 is a diagram illustrating an example in which a region of interest and a weight value w are set for a search target image. FIG. 22 is a diagram illustrating an example in which a plurality of attention areas and weight values w are set for a search target image. FIG. 23 is a diagram illustrating an example of an XML code including weighting information according to the first embodiment. FIG. 24 is a diagram illustrating an example of a weight value input screen according to the first embodiment. FIG. 25 is a flowchart illustrating an example of image search processing in the image search server according to the first embodiment. FIG. 26 is a diagram for explaining processing for cutting out an image in a region of interest from a search target image according to the first embodiment. FIG. 27 is a diagram for explaining an example in which each image in the range of each region of interest is stored in advance in the image DB in association with the search target image. FIG. 28 is a diagram for explaining a method for obtaining a changed similarity according to the first embodiment. FIG. 29 is a diagram for explaining a method of outputting a search result based on the changed similarity according to the first embodiment. FIG. 30 is a diagram illustrating an example of masking a search suppression area according to the first embodiment. FIG. 31 is a diagram illustrating an example of a business card image. FIG. 32 is a diagram showing an example of weighting for enabling a search by name based on a business card image to be executed with higher accuracy. FIG. 33 is a diagram for describing an example in which an image partially similar to the check target image is searched as a similar image. FIG. 34 is a flowchart illustrating an example of weighting processing in the similar image check according to the second embodiment. FIG. 35 is a diagram illustrating an example in which a plurality of check areas are specified in the check target image according to the second embodiment. FIG. 36 is a diagram illustrating an example of a result of completion of the similar image check for the check target image according to the second embodiment. FIG. 37 is a flowchart of an example illustrating weighting processing in the similar image check according to the first modification of the second embodiment. FIG. 38 is a diagram for explaining a second modification of the second embodiment. FIG. 39 is a diagram for describing a second modification of the second embodiment.

  Hereinafter, embodiments of an information processing system, an information processing apparatus, an information processing method, and an information processing program will be described in detail with reference to the accompanying drawings.

(System applicable to each embodiment)
FIG. 1 schematically shows a configuration of an example of an image search system as an information processing system applicable to each embodiment. In FIG. 1, the image search system includes an image search server 10 and can communicate with a registration terminal 20 and a search terminal 30 via a network. The image search server 10 may be configured by a single information processing apparatus, or may be configured by distributing functions and control by a plurality of information processing apparatuses connected to each other.

  The image search server 10 associates and registers the image transmitted from the registration terminal 20 and the cooperation information in the database. The image search server 10 searches the image registered in the database based on the captured image obtained by capturing the subject 40 transmitted from the search terminal 30, and transmits cooperation information associated with the searched image to the search terminal 30. To do.

  The image search server 10 includes an analysis unit 12, a search unit 13, a communication unit 16, and an image search database (DB) 11. The image search DB 11 includes an image database (DB) 14 and a linkage information database (DB) 15. The image DB 14 registers images so as to be searchable. In the cooperation information DB 15, cooperation information for linking with an image registered in the image DB 14 is registered in association with the image.

  More specifically, the image search server 10 analyzes an image to be registered by the analysis unit 12 and extracts feature information representing the feature of the image. The image search server 10 stores the extracted feature information in the image DB 14 in association with image identification information for identifying an image. The image search server 10 also stores the cooperation information to be provided to the user in the cooperation information DB 15 in association with the image identification information. The image (feature information) and the link information are associated with the image identification information and stored in the image DB 14 and the link information DB 15, respectively, so that the image is registered in the image search DB 11.

  The registration terminal 20 acquires an image and cooperation information for associating with the image, and transmits the acquired image to the image search server 10. The image search server 10 stores the image and the link information transmitted from the registration terminal 20 in the image DB 14 and the link information DB 15 as described above, and registers the image in the image search DB 11.

  Also, the registration terminal 20 can designate a region of interest for an image registered in the image search DB 11 and can weight the designated region of interest. The registration terminal 20 transmits weighting information including the attention area set to the image and the weight value indicating the weight set to the attention area to the image search server 10 together with the image and the cooperation information. The image search server 10 stores the weighting information transmitted from the registration terminal in the image DB 14 in association with the image, and registers the weighting information in the image search DB 11.

  In the example of FIG. 1, the registration terminal 20 uses a captured image obtained by capturing the subject 40 as an image for registering in the image search DB 11. This is not limited to this example, and the registration terminal 20 may use an image supplied from the outside or an image created inside the registration terminal 20 as an image for registration in the image search DB 11.

  The search terminal 30 has an imaging function, and can transmit an image obtained by imaging the subject 40 to the image search server 10 via the network. For example, the search terminal 30 activates the imaging function in accordance with a user operation to capture an image of the subject 40, and obtains an image obtained by imaging and a search request for requesting a search for an image similar to this image. Send to. The image search server 10 searches for an image similar to the image from the image DB 14 in response to the image transmitted from the search terminal 30 and the search request, and outputs identification information of the searched image as a search result.

  For example, the image transmitted from the search terminal 30 is received by the search unit 13 of the image search server 10. The search unit 13 passes the received image to the analysis unit 12. The analysis unit 12 analyzes the image passed from the search unit 13, extracts feature information from the image, and passes the extracted feature information to the search unit 13. The search unit 13 obtains a similarity between the feature information and the feature information stored in the image DB 14 based on the feature information passed from the analysis unit 12, and searches the image DB 14 for feature information having a high similarity. Here, when the weighting information is associated with the image corresponding to the searched feature information, the search unit 13 changes the similarity according to the weight value indicated in the weighting information, and uses the changed similarity. Then, the feature information is searched from the image DB 14.

  The search unit 13 acquires the link information associated with the image identification information corresponding to the searched feature information from the link information DB 15 and transmits it to the search terminal 30. The link information can include, for example, the URL (Uniform Resource Locator) of the Web site 42a and the Web site 42b on the Internet 41. In this case, the search terminal 30 can access the website 42 a and the website 42 b via the Internet 41 according to the cooperation information searched and transmitted by the image search server 10 based on the captured image obtained by capturing the subject 40. it can.

  FIG. 2 schematically shows a configuration of an example of an information processing system according to each embodiment. In FIG. 2, the same reference numerals are given to the portions common to FIG. 1 described above, and detailed description thereof is omitted.

  In FIG. 2, the information processing system 1 includes an image search server 10, a registration terminal 20, and a registration server 50. The information processing system 1 may further include a user management server 60. In FIG. 2, the image search DB 11 is shown to be connected to the outside of the image search server 10 for explanation. The image search DB 11 further includes a metadata DB that stores metadata related to images registered in the image DB 14.

  Here, in the information processing system 1, image registration is performed in units of campaigns. The campaign includes one or more images and link information associated with each image, and the metadata includes attribute information of the campaign. Each image included in the campaign is described as a page as appropriate. In other words, the campaign includes images on a page basis.

  As described above, the image search server 10 analyzes the image (captured image) transmitted from the search terminal 30 via the Internet 41, extracts the feature information, and extracts the feature information by the search unit 13. The degree of similarity between the information and the feature information stored in the image DB 14 is obtained, and feature information having a high degree of similarity, for example, a degree of similarity equal to or higher than a predetermined value is searched from the image DB 14. Then, the image search server 10 acquires cooperation information corresponding to the image identification information associated with the feature information searched from the image DB 14 from the cooperation information DB 15 and transmits it to the search terminal 30.

  The registration server 50 includes an image search DB 11 ′ corresponding to the image search DB 11, and includes an analysis unit 12 and a search unit 13, similar to the image search server 10. The image search DB 11 ′ further includes a metadata DB that stores metadata related to images included in the image search DB 11 ′.

  The registration terminal 20 includes a registration tool 21. The registration terminal 20 is provided with images and association information for associating with the images from the customers 62a, 62b, 62c,. The registration terminal 20 associates the cooperation information with each image provided from the customers 62a, 62b, 62c,. The process of associating the cooperation information with the images provided by the customers 62a, 62b, 62c,... And transmitting it to the registration server 50 is a process executed by the registration tool 21 in the registration terminal 20.

  Further, the registration tool 21 can designate a region of interest in the images provided by the customers 62a, 62b, 62c,..., Weight the designated region of interest, and set a weight value. The registration tool 21 also transmits weighting information including the attention area and the weight value to the registration server 50.

  The registration server 50 registers the image and linkage information transmitted from the registration terminal 20 and the weighting information in the image search DB 11 ′. That is, the registration server 50 analyzes the image transmitted from the registration terminal 20 by the analysis unit 12 included in the registration server 50, and extracts feature information representing the feature of the image from the image. The registration server 50 stores the image identification information in the image DB 14 included in the image search DB 11 ′ in association with the extracted feature information and the weighting information transmitted from the registration terminal 20. Further, the registration server 50 associates the image identification information with the cooperation information corresponding to the image stored in the image DB 14 and stores the image identification information in the cooperation information DB 15. In this way, the registration server 50 registers the image in the image search DB 11 ′.

  Note that the images, linkage information, and weighting information stored in the image DB 14 and the linkage information DB 15 are further associated with metadata including information (such as a user ID) indicating the registered user. User information can be acquired from a user DB 61 described later.

  When registering an image in the registration server 50, the registration terminal 20 can perform a similar image check that checks whether an image similar to the image to be registered is already registered in the registration server 50. The registration terminal 20 can perform a similar image check using the weighting information.

  The registration server 50 transmits to the image search server 10 the image, the linkage information, and the weighting information that are set to be public among the images (feature information), the linkage information, and the weighting information registered in the image search DB 11 ′. , It is registered in the image search DB 11. For example, the registration server 50 communicates with the image search server 10 at a predetermined timing such as at night, and is set to be openable among images registered in the image search DB 11 ′ of the registration server 50, and An image not registered in the image search DB 11 of the image search server 10 is searched. Then, the registration server 50 transmits the searched image and the weighting information and linkage information associated with the image to the image search server 10. The image search server 10 registers the image, linkage information, and weighting information transmitted from the registration server 50 in the image search DB 11.

  The user management server 60 includes a user DB 61 and manages information on users who use the information processing system 1. For example, the user DB 61 includes information on customers 62a, 62b, 62c,..., Information on users who can use the functions of the registration terminal 20 (registration tool 21), and information on the administrator who manages the image search server 10 and the registration server 50. Information is registered.

  The user management server 60 manages user information in four layers, for example, a general administrator, a domain administrator, a client administrator, and a client. The general administrator has the authority to access domain administrator, client administrator, and client information. The domain administrator has the authority to access the client administrator and client information, and the client administrator has the authority to access the client information. Clients are allowed to access only their own information. Each information registered in the user DB 61 can be browsed and edited according to the authority given to each level using the registration tool 21 from the registration terminal 20.

  FIG. 3 shows a configuration example of hardware of the registration terminal 20 applicable to each embodiment. In FIG. 3, the registration terminal 20 has a CPU 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a display control unit 2004, a storage 2006, and a data interface (I / F) with respect to a bus 2000. Interface) 2007, input device 2008, and communication I / F 2009 are connected. The registration terminal 20 can be configured using a general computer device.

  The storage 2006 is a storage medium capable of storing data in a nonvolatile manner, and for example, a hard disk drive is used. The storage 2006 is not limited to this, and a non-volatile semiconductor memory such as a flash memory may be used. The storage 2006 stores programs to be executed by the CPU 2001 and various data.

  The CPU 2001 controls the entire registration terminal 20 using the RAM 2003 as a work memory in accordance with programs stored in the ROM 2002 and the storage 2006.

  The display control unit 2004 converts the display control signal generated by the CPU 2001 into a signal that can be displayed by the display device 2005 and outputs the converted signal. The display control unit 2004 drives the display device 2005 based on the display control signal generated by the CPU 2001. The display device 2005 is driven by the display control unit 2004 and performs display according to the display control signal.

  The data I / F 2007 performs data input / output with an external device. As the data I / F 2007, for example, an interface such as USB (Universal Serial Bus) or Bluetooth (registered trademark) can be applied. A communication I / F 2009 performs communication via a network by wireless communication under the control of the CPU 2001.

  The input device 2008 includes a pointing device such as a mouse and a keyboard, for example, and accepts user input. The user can issue an instruction to the registration terminal 20 by operating the input device 2008 according to the display on the display device 2005, for example.

  FIG. 4 shows a configuration example of hardware applicable to the image search server 10 according to each embodiment. In FIG. 4, the image search server 10 is connected to a bus 1000 with a CPU 1001, a ROM 1002, a RAM 1003, a storage 1004, and a communication unit 1005. The storage 1004 is a storage medium capable of storing data in a nonvolatile manner, and is a hard disk drive, for example. The storage 1004 is not limited to this, and a nonvolatile semiconductor memory such as a flash memory may be used.

  The CPU 1001 controls the entire image search server 10 using the RAM 1003 as a work memory in accordance with programs stored in the ROM 1002 and the storage 1004.

  The storage 1004 stores programs executed by the CPU 1001 and various data. The above-described image search DB 11 is configured using a storage area on the storage 1004. In FIG. 4, the storage 1004 is shown to be configured by one piece of hardware, but this is not limited to this example. For example, a plurality of storage apparatuses are integratedly managed as one storage 1004. It may be. Further, the image search DB 11 is preferably stored in a storage 1004 as hardware different from the storage 1004 in which programs executed by the CPU 1001 and various data are stored.

  The communication unit 1005 includes a communication I / F that performs communication via a network by wireless communication under the control of the CPU 1001.

  In FIG. 4, the image search server 10 is shown to be configured by one piece of hardware, but this is not limited to this example, and the image search server 10 is a plurality of server devices having an equivalent configuration. You may comprise by controlling these collectively. The registration server 50 can be realized with the same configuration as that of the image search server 10, and thus the description thereof is omitted here.

  FIG. 5 shows a configuration example of hardware of the search terminal 30 applicable to each embodiment. In FIG. 5, the search terminal 30 has a CPU 3001, a ROM 3002, a RAM 3003, a display control unit 3004, a storage 3006, a data I / F 3007, an input device 3008, a communication I / F 3009 with respect to a bus 3000. An imaging unit 3010 is connected. In the search terminal 30, a GPS unit 3011 may be further connected to the bus 3000.

  The storage 3006 is a storage medium capable of storing data in a nonvolatile manner, and is a nonvolatile semiconductor memory such as a flash memory, for example. Not limited to this, a hard disk drive may be used as the storage 3006. The storage 3006 stores programs executed by the CPU 3001 and various data. Note that the storage 3006 and the ROM 3002 may share one rewritable nonvolatile semiconductor memory, for example.

  The CPU 3001 controls the entire search terminal 30 using the RAM 3003 as a work memory in accordance with programs stored in the ROM 3002 and the storage 3006. The display control unit 3004 converts the display control signal generated by the CPU 3001 into a signal that can be displayed by the display device 3005 and outputs the signal.

  The data I / F 3007 performs data input / output with an external device. As the data I / F 3007, for example, an interface such as USB or Bluetooth (registered trademark) can be applied.

  The display control unit 3004 drives the display device 3005 based on the display control signal generated by the CPU 3001. The display device 3005 includes an LCD (Liquid Crystal Display), for example, and is driven by the display control unit 3004 to perform display according to the display control signal.

  The input device 3008 accepts user input. The user can issue an instruction to the search terminal 30 by operating the input device 3008 according to the display on the display device 3005, for example. Note that it is preferable that the input device 3008 and the display device 3005 are integrally configured to output a control signal corresponding to the pressed position and to be configured as a touch panel that transmits the image of the display device 3005.

  A communication I / F 3009 performs communication via a network by wireless communication under the control of the CPU 3001.

  The imaging unit 3010 includes an optical system, an imaging device, and a control drive circuit for the optical system and the imaging device. The imaging unit 3010 performs a predetermined process on the imaging signal output from the imaging device and outputs the captured image as a digital signal. The imaging unit 3010 executes functions such as imaging and zooming according to an instruction by a user operation on the input device 3008. The captured image output from the imaging unit 3010 is sent to the CPU 3001 via the bus 3000 and subjected to predetermined image processing according to a program. A captured image output from the imaging unit 3010 and subjected to image processing can be stored in the storage 3006, for example. Further, the CPU 3001 can read a captured image from the storage 3006 and transmit it to the network via the communication I / F 3009.

  The GPS unit 3011 receives a GPS (Global Positioning System) signal, and calculates a current position composed of latitude and longitude based on the received signal.

  FIG. 6 is a functional block diagram illustrating an example of functions of the registration terminal 20 according to each embodiment. In FIG. 6, the registration terminal 20 includes a registration tool 21, which includes an acquisition unit 201, an extraction unit 202, a determination unit 203, a notification unit 204, a communication unit 205, a display unit 206, An input unit 207, a storage unit 208, an area setting unit 210, and a weight setting unit 211 are included.

  These acquisition unit 201, extraction unit 202, determination unit 203, notification unit 204, communication unit 205, display unit 206, input unit 207, storage unit 208, region setting unit 210 and weight setting unit 211 are programs that run on the CPU 2001. It is realized by. However, the present invention is not limited to this, and some or all of these units may be configured by hardware circuits that operate in cooperation with each other.

  The communication unit 205 functions as a transmission unit that controls communication via the network and transmits data on the network and a reception unit that receives data from the network.

  The display unit 206 generates display information to be displayed on the display device 2005. That is, in the registration terminal 20, the CPU 2001 generates a display control signal based on the display information generated by the display unit 206, and passes the generated display control signal to the display control unit 2004. The display control unit 2004 performs display according to the display information on the display device 2005 according to the display control signal. Hereinafter, a series of operations for displaying on the display device 2005 in accordance with the display information generated by the display unit 206 will be described such as the display unit 206 performing display.

  The input unit 207 receives information input by an input operation on the input device 2008.

  The acquisition unit 201 acquires an image from outside the registration terminal 20. For example, the acquisition unit 201 acquires an image from the customer 62a. The acquisition unit 201 may acquire an image transmitted from another device of the customer 62a via a network, or may receive an image via a recording medium such as a CD (Compact Disk) or a DVD (Digital Versatile Disk). May be. Furthermore, the acquisition unit 201 may acquire an image from a print medium such as paper. Moreover, the acquisition part 201 acquires the information for the cooperation information linked | related with an image from the customer 62a.

  The extraction unit 202 extracts an image in a predetermined area in the image acquired by the acquisition unit 201. The extraction unit 202 cuts out and extracts, for example, an image in an area set by the area setting unit 210 described later from the image acquired by the acquisition unit 201. The determination unit 203 performs determination processing based on, for example, similarity. The notification unit 204 notifies information indicating the determination result by the determination unit 203.

  The storage unit 208 controls writing and reading of information with respect to the storage 2006 and the RAM 2003.

  The area setting unit 210 sets an area for the image acquired by the acquisition unit 201 based on the information received by the input unit 207. For example, the area setting unit 210 sets an area corresponding to the range on the image in response to a range specifying operation for specifying the range on the image performed on the input device 2008.

  The weight setting unit 211 performs weighting by setting a weight value for the region set by the region setting unit 210. The weight setting unit 211 may weight the area based on information received by the input unit 207 or may use a predetermined value.

  The acquisition unit 201, extraction unit 202, determination unit 203, notification unit 204, communication unit 205, display unit 206, input unit 207, storage unit 208, area setting unit 210, and weight setting unit 211 included in the registration tool 21 described above are This is realized by an information processing program operating on the CPU 2001. This information processing program is provided by the registration server 50 when the registration terminal 20 connects to the registration server 50, for example.

  For example, the registration terminal 20 is equipped with a browser application (hereinafter referred to as a browser) equivalent to that installed in a general computer. The browser operates on the CPU 2001, reads a file described in accordance with, for example, HTML (HyperText Markup Language), and performs a predetermined operation. When the registration terminal 20 accesses the registration server 50 using a browser, the information processing program on the registration server 50 is read into the browser.

  The information processing program is, for example, each of the above-described units constituting the registration tool 21 (acquisition unit 201, extraction unit 202, determination unit 203, notification unit 204, communication unit 205, display unit 206, input unit 207, storage unit 208, area The module configuration includes a setting unit 210 and a weight setting unit 211). As actual hardware, a browser operating on the CPU 2001 executes an information processing program read from the registration server 50, whereby the above-described units are expanded on a main storage device (for example, the RAM 2003), and the respective units are stored on the main storage device. To be generated.

  For example, the information processing program may be stored in advance in the storage 2006 and provided. The information processing program can be stored on a computer connected to a network and provided by being downloaded via the network by the communication I / F 2009. The information processing program may be configured to be provided or distributed via a network.

  However, the present invention is not limited to this, and the information processing program may be provided by being recorded in a computer-readable recording medium such as a CD or a DVD in an installable or executable format file. In this case, the information processing program recorded on the recording medium is supplied to the registration terminal 20 via an external drive device connected to the data I / F 2007, for example. Not limited to this, an information processing program recorded in a recording medium is temporarily stored on another information processing apparatus such as a personal computer, and communication via the communication I / F 2009 or data I / F 2007 is performed from the other information processing apparatus. May be supplied to the registration terminal 20.

  FIG. 7 is a functional block diagram illustrating an example of functions of the search terminal 30 applicable to each embodiment. 7, the search terminal 30 includes an imaging processing unit 301, a search request unit 302, a selection unit 303, a reception unit 304, a transmission unit 305, an input unit 306, and a display unit 307.

  These imaging processing unit 301, search request unit 302, selection unit 303, reception unit 304, transmission unit 305, input unit 306, and display unit 307 are realized by a program that operates on the CPU 3001. Not only this but the imaging processing part 301, the search request | requirement part 302, the selection part 303, the receiving part 304, the transmission part 305, the input part 306, and the hardware which operate | moves a part or all in cooperation with each other. It may be realized by a wear circuit.

  The imaging processing unit 301 activates a camera function included in the search terminal 30 to control the imaging unit 3010 and outputs a captured image. The reception unit 304 performs reception processing for data transmitted via the network. The transmission unit 305 performs a data transmission process via the network. The input unit 306 receives an input operation on the input device 3008 included in the search terminal 30 and acquires data corresponding to the operation. The display unit 307 controls display of information on the display device 3005 included in the search terminal 30. The input unit 306 and the display unit 307 constitute a UI (User Interface) of the search terminal 30.

  The search request unit 302 generates a search request for requesting a search based on the captured image acquired by the imaging processing unit 301. The search request unit 302 transmits the captured image and the generated search request from the transmission unit 305 to the image search server 10. The selection unit 303 generates a selection screen for selecting desired information from the cooperation information returned from the image search server 10 in response to the search request, and passes the selection screen to the display unit 307. In addition, the selection unit 303 performs processing corresponding to the selected cooperation information according to the input operation received by the input unit 306 according to the selection screen.

(Image registration processing according to each embodiment)
Next, an image registration process according to each embodiment will be described. FIG. 8 is a flowchart illustrating an example of an image registration process executed by the registration tool 21 in the registration terminal 20 according to the embodiment. Prior to the processing of the flowchart of FIG. 8, information (user ID and password) of a user who uses the registration tool 21 is registered in the user DB 61 by the user management server 60. In the following, it is assumed that images are registered in units of campaigns.

  When the registration tool 21 is activated in the registration terminal 20, the registration tool 21 performs login processing for the user in step S100. For example, the registration tool 21 displays the login screen 400 illustrated in FIG. In the example of FIG. 9, the login screen 400 includes an input area 401 for inputting user information (user ID and password), a button 402 for instructing login, and a button 403 for requesting reissue of a password. And are arranged.

  For example, when the user ID and password are input in the input area 401, the registration tool 21 transmits the input user ID and password to the user management server 60 and requests authentication of the user ID. When the registration tool 21 receives information from the user management server 60 indicating that the user ID has been successfully authenticated, the registration tool 21 shifts the processing to step S101.

  In step S101, the registration tool 21 causes the display unit 206 to display the menu screen 410 illustrated in FIG. In the example of FIG. 10, the menu screen 410 includes tabs 411a, 411b, and 411c, a button group 412, a search input area 413, a list display area 414, and an end button 415.

  The tabs 411a, 411b, and 411c are tabs for selecting the menu item and determining the operation mode of the registration tool 21. The tabs 411a and 411b are tabs for shifting to the user management mode, and an administrator list and a client list are displayed by operating the tabs 411a and 411b, respectively. The tab 411c is a tab for shifting to the image registration mode, and a campaign list is displayed by operating the tab 411c.

  The list display area 414 displays a list corresponding to the menu item selected by the tabs 411a, 411b and 411c. In the example of FIG. 10, the tab 411b is operated and the client list is selected. In the following, each row of the list is called a record, and each column is called an item.

  The button group 412 selects a process for the list displayed in the list display area 414. In the example of FIG. 10, the button group 412 includes four buttons, and each button is assigned functions “new registration”, “edit”, “delete”, and “CSV output” from the left. The “new registration” button adds a new record to the list displayed in the list display area 414. The “edit” button edits the information of the record selected in the list displayed in the list display area 414. The “delete” button deletes the information of the record selected in the list displayed in the list display area 414. “CSV output” outputs the information of the list displayed in the list display area 414 as a text file in CSV (Comma-Separated Values) format.

  In the search input area 413, a record is selected from a list displayed in the list display area 414 based on a selected or input search key. The end button 415 ends the process by the registration tool 21.

  Returning to FIG. 8, when the menu is displayed in step S101, the process proceeds to the next step S102, and the registration tool 21 operates any of the tabs 411a, 411b and 411c, and the end button 415. , Wait for the process to be selected. If the end button 415 is operated, the registration tool 21 shifts the process to step S103, performs a logout process for the user, and ends the operation of the registration tool 21.

  In step S102, when the tab 411a or the tab 411b is operated and the user management mode is selected, the registration tool 21 shifts the processing to step S104. In step S104, the registration tool 21 displays one of the administrator list screen and the client list screen depending on which of the tabs 411a and 411b is operated in step S102. When an operation to the effect that the process on the administrator list screen or the client list screen has been completed is performed, the registration tool 21 returns the process to step S101 to display a menu.

  Furthermore, when the tab 411c is operated and the image registration mode is selected in step S102, the registration tool 21 shifts the processing to step S105. In step S105, the registration tool 21 displays the registration information list screen 420 illustrated in FIG. 11 on the display unit 206, and waits for a user operation. In FIG. 11, a registration information list screen 420 includes buttons 421a to 421d, a search input area 422, a list display area 423, and an end button 424.

  The list display area 423 displays a list of campaigns registered in the registration server 50. At this time, the registration tool 21 can select a campaign that can be accessed by the user logged in to the registration tool 21 from the campaigns registered in the registration server 50 and display the campaign in the list display area 423. For example, the registration tool 21 acquires information on the logged-in user from the user management server 60, and acquires a list of campaigns accessible to the user from the metadata DB of the registration server 50 based on this information.

  The button 421 a is a “new registration” button for adding a new record to the list displayed in the list display area 423. The button 421 b is an “edit” button for editing information of a record selected in the list displayed in the list display area 423. The button 421c is a “delete” button for deleting information of a record selected in the list displayed in the list display area 423. The button 421d is a button for setting whether the campaign registered in the record selected in the list displayed in the list display area 423 is open or closed.

  In the example of FIG. 11, in the list displayed in the list display area 423, each record includes “campaign name”, “client name”, “user ID”, “person in charge”, “number of public pages”, “use” Each item includes “number of pages”, “last updated date”, “status”, “administrator in charge”, “affiliated domain”, “remarks”, and “CSV output”. The record information includes the metadata of the campaign indicated in the record and related campaign information, and is stored in, for example, the metadata DB included in the image search DB 11 ′ in the registration server 50.

  The record is created for each campaign and is associated with one or more images included in one campaign. For example, the record is generated based on metadata registered in the metadata DB. The registration tool 21 can acquire information on records displayed in the list display area 423 from the metadata DB included in the image search DB 11 ′ of the registration server 50.

  In each record of the list displayed in the list display area 423, the item “campaign name” indicates a name given to the campaign registered in the record. The item “client name” indicates the owner of the campaign of the record. The item “user ID” indicates the ID of the owner of the campaign of the record. The item “person in charge” indicates the name of the user who has input the campaign information of the record.

  The item “number of public pages” indicates the number of public pages set to be public among pages included in the campaign of the record. For example, a page in which a target area described later is designated as a public page. The item “number of used pages” indicates the total number of pages included in the campaign of the record. That is, the number of pages indicated in the item “number of used pages” is displayed even if the campaign of the record is in a non-public state. It should be noted that a record in which the items “number of public pages” and “number of used pages” are both “0” indicates that only the campaign information is registered and no image is registered yet.

  The item “last update date” indicates the date when the campaign information of the record was last edited. The item “state” indicates whether the campaign of the record is in a public state or a private state. The item “administrator in charge” indicates an upper manager of the user who is in charge of the campaign of the record. The item “affiliation domain” indicates information on the domain to which the item “administrator in charge” belongs. In the item “Remarks”, arbitrary information can be input. The item “CSV output” outputs the record information as a CSV text file.

  When the registration information list screen 420 is displayed in step S105, the registration tool 21 shifts the processing to the next step S110, and waits for operations on the buttons 421a to 421d and the end button 424. Hereinafter, the processing according to the operation on each button will be described according to the following items (1) to (5).

(1) In step S110, when the button 424 for instructing the end of the process is operated, the registration tool 21 returns the process to step S101, for example, and displays a menu.

(2) In step S110, the registration tool 21 moves the process to step S120 when the button 421d for setting the publication and non-publication of the campaign is operated in a state where the record is selected in the list display area 423. Let In step S120, the registration tool 21 causes the display unit 206 to display a disclosure setting screen for setting the disclosure and non-disclosure of the campaign indicated in the selected record.

  FIG. 12 shows an example of a public setting screen. In FIG. 12, the public setting screen 430 includes a public setting unit 431, an OK button 432a, and a cancel button 432b. In the example of FIG. 12, the disclosure setting unit 431 can exclusively set disclosure and non-disclosure by radio buttons, and the value of the item “publication status” of the selected record is set as an initial value. When the OK button 432a is operated, the registration tool 21 sets the public state set to public or non-public in the public setting unit 431 for the record. When the disclosure state is set, the registration tool 21 updates the display of the list display area 423 and returns the process to step S105. In addition, when the cancel button 432b is operated, the registration tool 21 sets the publishing state of the record as the state before displaying the publishing setting screen 430, and returns the process to step S105.

(3) In step S110, if the record 421 is selected in the list display area 423 and the button 421c for designating image deletion is operated, the registration tool 21 shifts the processing to step S121. In step S121, the registration tool 21 deletes the information of the selected record. At this time, it is preferable that the registration tool 21 displays a confirmation screen such as whether or not to really delete the record and whether or not to delete the record including information on the image associated with the record. When the record is deleted, the registration tool 21 updates the display of the list display area 423 and returns the process to step S105.

(4) In step S110, when the button 421b for instructing image editing is operated, the registration tool 21 shifts the processing to step S122. In step S <b> 122, the registration tool 21 acquires an image included in the campaign indicated by the record selected in the list displayed in the list display area 423 of the registration information list screen 420 from the registration server 50. When the cooperation information is associated with the acquired image, the registration tool 21 also acquires this cooperation information. Then, the registration tool 21 shifts the process to step S124.

(5) In step S110, when the button 421a for instructing new registration is operated in step S110, the registration tool 21 shifts the processing to step S123 and executes new campaign registration processing. In step S123, the registration tool 21 first causes the display unit 206 to display a campaign registration screen 440 as illustrated in FIG. 13 for newly registering a campaign.

  In FIG. 13, the campaign registration screen 440 includes input areas 441 and 442 for inputting a campaign name and a user ID, and an OK button 443a and a cancel button 443b. The character string input in the input area 441 becomes the value of the item “campaign name” in the record. Similarly, the character string input in the input area 442 becomes the value of the item “user ID” in the record.

  When the OK button 443a is operated, the registration tool 21 sets each character string input in the input areas 441 and 442 on the campaign registration screen 440 as the items “campaign name” and “user ID” of the record, To step S124.

  In step S124, the registration tool 21 causes the display unit 206 to display an editing screen for editing campaign information. FIG. 14 shows an example of an edit screen according to each embodiment. 14, the edit screen 480 includes a header area 481, a thumbnail display area 483, a page edit area 484, a preview area 485, a target area list area 486, a link icon list area 487, and a tool bar 488. .

  In the header area 481, the name of the campaign to be edited on the editing screen 480 is displayed on the left end, and on the right end is a user menu display section that displays a menu that can be used with the authority of the user logged in to the registration tool 21 482 is arranged.

The thumbnail display area 483 is an area in which thumbnail images obtained by reducing each image included in the campaign are displayed. For example, when the process proceeds from step S122 instructing editing of the campaign information described above to step S124, the registration tool 21 acquires the record selected from the list display area 423 acquired from the registration server 50 in step S122. Each image included in the campaign shown in FIG. 5 is acquired from the image DB 14 of the registration server 50. The registration tool 21 reduces each acquired image to generate a thumbnail image, and displays the thumbnail image in the thumbnail display area 483. In the example of FIG. 14, thumbnail images 4830 ₁ , 4830 ₂ , 4830 ₃ ,... Are displayed in the thumbnail display area 483.

  When the process proceeds from step S123 for instructing new registration of the above-described campaign information to step S124, the thumbnail display area 483 is blank. Further, by operating a scroll bar provided at the right end of the thumbnail display area 483, other thumbnail images virtually displayed outside the thumbnail display area 483 can be sequentially displayed in the thumbnail display area 483.

The page editing area 484 displays an image 4840 to be edited among the images included in the campaign. For example, the registration tool 21 displays the original image of the thumbnail image selected from the thumbnail images 4830 ₁ , 4830 ₂ , 4830 ₃ ,... Displayed in the thumbnail display area 483 as the image 4840 in the page editing area 484. In the example of FIG. 14, a thumbnail image 4830 ₂ is selected in the thumbnail display area 483, and the original image of the selected thumbnail image 4830 ₂ is displayed as an image 4840 in the page editing area 484. Also, in the thumbnail display area 483, the thumbnail image 4830 ₂ that is currently displayed is selected in the page editing area 484 is shown highlighted by the surrounding frame 4832.

  When a range is specified for the image 4840 displayed in the page editing area 484, the registration tool 21 can set the specified range as a target area. For example, the range is specified when the input unit 207 accepts an operation of the input device 2008 by the user. For example, by designating two arbitrary points in the image 4840 displayed in the page editing area 484, a rectangular range having the designated two points as diagonal vertices is designated. In the example of FIG. 14, target areas 4841a and 4841b are set for the image 4840 in the page editing area 484 by a rectangular range.

  The registration tool 21 can explicitly indicate the target areas 4841a and 4841b by displaying the set target areas 4841a and 4841b, for example, with the colors in the areas different from those outside the areas. In addition, the shape of the target region is not limited to a rectangle, and any shape can be set as long as it is a closed shape.

  The registration tool 21 generates identification information (such as a serial number) for identifying the set target areas 4841a and 4841b, and associates them with the target areas 4841a and 4841b. The identification information is generated so that the target area can be identified through each image included in the campaign to be edited.

  Although details will be described later, the registration tool 21 can associate one or more pieces of linkage information with the target areas 4841a and 4841b set in the page editing area 484.

Further, the registration tool 21 performs some editing, for example, designation of the target area in each of the thumbnail images 4830 ₁ , 4830 ₂ , 4830 ₃ ,... A marker image 4831 is added to the thumbnail image corresponding to the image.

  The preview area 485 displays a preview screen 4850 that emulates a screen in which the image being edited in the page editing area 484 is displayed on the display device 3005 of the search terminal 30 by the application program corresponding to the information processing system 1. In the preview area 485, the registration tool 21 explicitly displays the target areas 4841a and 4841b set in the page editing area 484 by highlighting or the like. In addition, the registration tool 21 displays information indicating the cooperation information set in each of the target areas 4841a and 4841b in the preview area 485 so as to correspond to each of the target areas 4841a and 4841b.

  In the target area list area 486, a list of cooperation information added to each target area set for each image included in the campaign to be edited is displayed as icon images 600, 600,. In the example of FIG. 14, the registration tool 21 displays the cooperation information together in the target area list area 486 for each piece of identification information (serial number) added to the corresponding target area. Further, in the target area list area 486, by operating the scroll bar provided at the right end, the link information of other target areas virtually displayed outside the target area list area 486 is displayed in the target area list area 486. It can be displayed sequentially.

  The link icon list area 487 displays the link information that can be set for the target area as a list using the icon images 600, 600,... Corresponding to each link information. The registration tool 21 moves the icon image 600 displayed in the link icon list area 487 by a so-called drag-and-drop operation, and superimposes the icon image 600 on the target area 4841a set in the page editing area 484, for example. 600,... Can be associated with the target area 4841a. In addition, a plurality of pieces of cooperation information can be associated with one target area.

  In the link icon list area 487, by operating a scroll bar provided at the right end, other icon images 600 virtually arranged outside the link icon list area 487 appear in the link icon list area 487 sequentially. Can be made.

  When cooperation information is associated with a target area, an icon image 600 indicating the associated cooperation information can be displayed at a predetermined position in the target area. This is preferable because it is possible to easily know which linkage information is associated with the target area 4841a. In addition, the association of the cooperation information with the target area is reflected in the display of the preview area 485.

  In the tool bar 488, a page specifying unit 4880 for specifying a page (image) displayed in the page editing area 484 and buttons or button groups 4881 to 4890 for executing various functions provided in the registration tool 21 are arranged. The A button 4881 is a button for returning from the editing screen 480 to the registration information list screen 420.

  A button 4882 is a button for uploading an image from the registration terminal 20 to the registration server 50. When the button 4882 is operated, the registration tool 21 causes the display unit 206 to display a file selection screen 460 as exemplified in FIG. In FIG. 15, a file selection screen 460 is a standard file selection screen provided by, for example, an OS (Operating System) operating on the CPU 2001 of the registration terminal 20, and includes a folder designation unit 461 and a file list display area 462. A file name display section 463, a file format designation section 464, and buttons 465 and 466.

  The file list display area 462 displays a list of files stored in the folder designated by the folder designation unit 461. The file format designation unit 464 designates the format of a file to be displayed in the file list display area 462. The file name display unit 463 displays the file name of the file selected in the file list display area 462. By selecting one or more files from the files displayed in the file list display area 462 and operating the button 465, for example, information indicating the selected files is passed from the OS to the registration tool 21. The registration tool 21 transmits the file indicated by the file information passed from the OS from the registration terminal 20 to the registration server 50, and executes uploading of the image to the registration server 50. A button 466 is used when returning to the editing screen 480 without uploading a file.

  Returning to the description of FIG. 14, the button group 4883 includes two buttons for adjusting the display size of the image 4840 displayed in the page editing area 484.

  A button 4884 switches an area setting mode for setting an area in the page editing area 484 to an object area setting mode for setting an object area. When the button 4884 is operated to switch the edit mode to the target area setting mode and the area is specified on the image 4840 displayed in the page edit area 484, the registration tool 21 selects the area as the target area. Set to.

  A button 4885 switches an area setting mode for designating an area in the page editing area 484 to a mask setting mode for designating a mask area that is not subject to extraction of feature information by the analysis unit 12. When the button 4885 is operated to switch the edit mode to the mask setting mode and the area is designated on the image 4840 displayed in the page edit area 484, the registration tool 21 sets the area as a mask area. Set.

  A button 4886 is a button for displaying a campaign setting screen for setting the details of the campaign. FIG. 16 shows an example of a campaign setting screen. In FIG. 16, the campaign setting screen 450 includes display areas 451a and 451b, input areas 451c to 451e, 453 and 454, a disclosure method setting unit 452, and buttons 455a and 455b. In the display areas 451a and 451b, the values of the items “user name” and “client name” of the record in the registration information list screen 420 shown in FIG. 11 are displayed in an unchangeable manner.

  In the input area 451c, a campaign name is input. In the input area 451c, the value of the record item “campaign name” in the registration information list screen 420 is input by default. The input area 451d is used to input a display name used when the search terminal 30 displays the campaign. In the input area 451e, for example, the name of the person in charge who registers the campaign is input.

  The publication method setting unit 452 sets a campaign publication method. Here, “public” means an operation in the search terminal 30 when a predetermined operation is performed on the search terminal 30. In the example of FIG. 16, the disclosure method setting unit 452 includes “collection”, a plurality of SNS (Social Networking Service) such as “SNS # A” and “SNS # B”, and “external application” as setting items. It has been. The disclosure method setting unit 452 can designate a plurality of disclosure methods at the same time.

  Among the disclosure methods that can be set by the disclosure method setting unit 452, the setting item “collection” stores campaign information in the search terminal 30. The setting items “SNS # A” and “SNS # B” respectively post campaign information to SNS # A and SNS # B. In the setting items “SNS # A” and “SNS # B”, by checking “Do you want to include a URL in the posting?”, It is possible to include a URL in a linkable manner with respect to the posted content. The setting item “external application” causes the search terminal 30 to display campaign information using an external application program for an application program corresponding to the information processing system 1.

  The input area 453 is a post message when posting the information of the campaign to SNS # A and SNS # B when the disclosure method setting unit 452 sets the disclosure by the setting items “SNS # A” and “SNS # B”. Enter. In the input area 454, for example, remarks regarding the campaign are input.

  The button 455a reflects the information input and set in the input areas 451c to 451e, 453, and 454 and the disclosure method setting unit 452 as campaign setting information, and returns to the editing screen 480. This campaign setting information is included in the metadata of the campaign. When the button 455a is operated, the registration tool 21 performs, for example, a campaign based on information input in the input areas 451c to 451e and 453 and 454 and setting values set by the disclosure method setting unit 452. Update the corresponding record or metadata. The button 455b returns to the editing screen 480 without updating the record and metadata corresponding to the campaign.

  The description returns to FIG. 14, and the button group 4887 includes three buttons that respectively execute copy, paste, and deletion of information.

  A button 4888 checks whether or not an image including a part similar to a part or all of the image (page) 4840 currently displayed in the page editing area 484 of the editing screen 480 is already registered in the image search server 10. Perform similar image check. This is because, for example, the registration server 50 is targeted instead of the image search server 10, and among the images registered in the registration server 50, an image for which “public” is set by the campaign setting information is targeted for checking. Do. However, the present invention is not limited to this, and an image registered as “non-public” registered in the registration server 50 may be further checked, or the image search server 10 may be directly checked. The registration tool 21 displays the search result transmitted from the registration server 50 on the display unit 206. The user can re-edit the image based on the search result.

  At this time, for example, the registration tool 21 can send only the image in the target area set according to the operation of the button 4884 to the registration server 50 as a check target, and can request a search for similar images. In addition, the registration tool 21 can exclude an image in the mask area set according to the operation of the button 4885 from the check target. Further, the registration tool 21 divides the image 4840 that is currently displayed in the page editing area 484, sends each of the divided images to the registration server 50, requests a similar image search, and resembles each image. An image check can also be performed.

  A button 4889 switches an area setting mode for setting an area in the page editing area 484 to an attention area setting mode for setting an attention area. When the button 4889 is operated to switch the editing mode to the attention area setting mode and the area is specified on the image 4840 displayed in the page editing area 484, the registration tool 21 pays attention to the area. Set to area. The registration tool 21 further sets a weight value for the set attention area and performs weighting. The registration tool 21 may weight the attention area based on a predetermined weight value or based on a weight value received by the input unit 207 in accordance with a user operation. The details of setting the attention area and weighting the attention area will be described later.

  A button 4890 instructs to save the campaign being edited on the edit screen 480.

  Here, returning to FIG. 8, when the button 4890 is operated, the registration tool 21 shifts the processing to step S <b> 125 and starts the campaign saving processing. When the saving process is started, the registration tool 21 shifts the process to step S126, and determines whether the campaign to be saved is a newly registered campaign. When the process is shifted from step S122 described above to step S126, the registration tool 21 determines that the campaign is not a newly registered campaign, and shifts the process to step S127.

  On the other hand, the registration tool 21 determines that the campaign is a new registration when the process proceeds from step S123 described above to step S126, and the process proceeds to step S130.

  In step S <b> 130, the registration tool 21 stores, in the registration server 50, an image including a part similar to a part or all of the above-described image for the image in which the target area is set among the images included in the campaign to be saved. A similar image check is performed to determine whether the image has already been registered. For example, the registration tool 21 transmits all images included in the campaign to be saved to the registration server 50 and requests a similar image search. If the registration tool 21 determines in step S131 that no similar image is registered in the registration server 50 based on the search result, the registration tool 21 shifts the processing to step S127.

  On the other hand, when the registration tool 21 determines in step S131 that a similar image is registered in the registration server 50 based on the search result, the registration tool 21 shifts the processing to step S132. In step S132, the registration tool 21 displays a warning screen illustrated in FIG. 17 on the display unit 206, for example. In FIG. 17, a warning message 471 indicating that a similar image has already been registered in the registration server 50 is displayed on the warning screen 470, and a “Yes” button 472 and a “No” button 473 are arranged.

  When the “No” button 473 is operated on the warning screen 470, the registration tool 21 can execute avoidance processing for registration of similar images. As an example, the registration tool 21 causes the display unit 206 to display a message that prompts execution of avoidance processing for similar image registration, and returns the processing to step S124. In this case, as the avoidance process, for example, change of an image included in a campaign to be saved, setting of a mask area for the image, change of a target area, and the like can be considered. When the avoidance process is executed, the process proceeds to step S127.

  On the other hand, when the “Yes” button 472 is operated on the warning screen 470, the registration tool 21 can shift the process to step S127 without executing the avoidance process described above, and execute image registration.

  In step S127, the registration tool 21 transmits campaign information to the registration server 50, and requests the registration server 50 to update the registered information. For example, the registration tool 21 transmits each image included in the campaign, each link information associated with each image, and metadata of the campaign to the registration server 50.

  Also, the registration tool 21 generates information indicating the target area based on each information set on the editing screen 480 and transmits the information to the registration server 50. In each embodiment, the information indicating the target area is described using, for example, (Extensible Markup Language), which is a kind of markup language described by defining the meaning and structure of data using tags.

  The registration server 50 updates the image DB 14, the linkage information DB 15, and the metadata DB based on each information transmitted from the registration terminal 20 in response to a request from the registration tool 21. At this time, the registration server 50 associates the file name of the file in which the XML code describing the information indicating the target area is stored with the image identification information, and stores them in the cooperation information DB 15.

  When the registration tool 21 transmits an information update request to the registration server 50 in step S127, the process returns to step S105.

  Here, an XML code describing information indicating a target area according to each embodiment will be schematically described. FIG. 18 shows an example of an XML code in which information indicating a target area is described according to each embodiment. In FIG. 18 and the following similar code examples, the numbers at the beginning of each line indicate a line number for explanation unless otherwise specified, and are distinguished from the code indicating the data body by the following colon (:). Has been. Moreover, the code illustrated in FIG. 18 is a pseudo code imitating a C language notation generally used as a programming language.

  In FIG. 18, the XML code 140 defines a page by a portion (1st to 16th lines) sandwiched between outermost tags “<page>” and “</ page>”. That is, one image is associated with the portion sandwiched between the tags “<page>” and “</ page>”. Between each of the 2nd to 12th lines and the 13th to 15th lines (the 14th line is shown by omitting a plurality of lines), it is sandwiched between the tags "<hotspot>" and "</ hotspot>" Each target area is defined by each part. Thus, a plurality of target areas can be defined for one page (one image).

  In the third line, the range in the page of the target area shown in the second to twelfth lines is, for example, the coordinates (upper left corner) by the portion sandwiched between the tags “<area>” and “</ area>”. x1, y1) and the coordinates (x2, y2) of the lower right corner.

  Tags "<link>" and "</ link>" in the ranges of the 4th to 8th lines and the 9th to 11th lines (the 10th line is shown by omitting a plurality of lines), respectively. One piece of linkage information is defined by each portion between the two. In this way, a plurality of pieces of cooperation information can be defined for one target area. For example, as illustrated in the fourth to eighth lines in FIG. 18, the title of the cooperation information is defined by the portion sandwiched between the tags “<title>” and “</ title>”. The type of the linkage information is defined by the portion sandwiched between the tags “<type>” and “</ type>”.

  In this example, as link information, address information on a network such as the Internet 41 is defined using a URI (Uniform Resource Identifier) by a portion sandwiched between tags “<URI>” and “</ URI>”. ing. In this case, by adding an argument to the URI, for example, the behavior of the browser application that has read the URI can be specified. Further, the cooperation information is not limited to the URI. By defining tags in advance, various types of linkage information can be described. For example, information for starting a specific application program can be described as the linkage information.

  In this example, in the linkage information defined in the part sandwiched between the tags "<link>" and "</ link>", the tags "<title>" and "</ title>" and the tag "<type" > ”And“ </ type> ”and tags“ <URI> ”and“ </ URI> ”define three items.

  This is not limited to this example, and more items can be defined in the portion between the tags “<link>” and “</ link>”. Image data may be defined as an item in a portion between tags “<link>” and “</ link>”, and the image data itself of the target area may be described.

  As described above, since the XML code 140 defines the target area in the page and the cooperation information in the target area, the function based on the cooperation information associated with the object included in the target area in the page can be executed. it can.

(First embodiment)
Next, a first embodiment will be described. The image search system according to the first embodiment sets a region of interest in an image (referred to as a search target image) registered in the image search DB 11 and sets a weight value w to the set region of interest to perform weighting. . Then, in the image search with respect to the image search DB 11 based on the image transmitted from the search terminal 30 (referred to as a search target image), the image search system sets the similarity s between the search target image and the search target image to the attention area. The image search is performed by changing based on the set weight value.

  In the first embodiment, in this way, the attention area weighted to the image to be searched is set, and the similarity s with the search target image is changed based on the weight value w of the attention area. In particular, it is possible to set a part to be searched or a part to be searched.

  Hereinafter, the weighting process and the image search for the image to be searched according to the first embodiment will be described in more detail. First, the setting of the attention area for the searched image and the weighting process for the attention area will be described.

  FIG. 19 shows an example of a searched image. In FIG. 19, the searched image 500 includes objects 510 a to 510 c, objects 511 a to 511 c, and an object 512. For example, it is assumed that the searched image 500 is a product catalog whose name is indicated by an object 512, and the objects 510a to 510c and the objects 511a to 511c indicate the product and the seller of the product, respectively. Further, it is assumed that the searched image 500 is displayed on the editing screen 480.

  FIG. 20 is a flowchart illustrating an example of attention area setting and weighting processing according to the first embodiment. For example, the registration tool 21 starts the processing according to the flowchart of FIG. 20 when a button 4889 arranged on the tool bar 488 is operated on the above-described editing screen 480 (see FIG. 14).

  In step S200, the registration tool 21 designates a range on the image to be set as the attention area according to the user operation. For example, the registration tool 21 uses the input device 2008 as a pointing device to set the start point and the end point when the drag operation is performed between two arbitrary points of the search target image 500 displayed on the editing screen 480 as the diagonal vertex. Can be designated as the range of the region of interest.

  In the next step S201, the registration tool 21 sets a weight value w, which is a value indicating the weight, for the attention area whose range is specified in step S200. An example of a weight value input screen for setting the weight value w will be described later.

  FIG. 21 shows an example in which a region of interest and a weight value w are set for the searched image 500. In the example of FIG. 21, the attention area 520b is set including the object 510b included in the searched image 500, and the weight value w = 1.0 is set for the attention area 520b. The similarity s is a value in the range of 0 ≦ s ≦ 1, and the weight value w can be set in the range of −1.0 ≦ w ≦ 1.0.

  In the next step S <b> 202, the registration tool 21 determines whether or not all the setting of the attention area for the search target image 500 has been completed, for example, in response to a user operation accepted by the input unit 207. If the registration tool 21 determines that the setting of the attention area has not been completed yet, the registration tool 21 returns the processing to step S200 and accepts the setting of the next attention area. That is, the registration tool 21 can set a plurality of attention areas and weight values w for one searched image 500.

  FIG. 22 shows an example in which a plurality of attention areas and weight values w are set for the searched image 500. In the example of FIG. 22, in addition to the attention area 520b shown in FIG. 21 described above, an attention area 520a including the object 512 and an attention area 520c including the object 511c are set. A weight value w = −0.7 is set for the attention area 520a, and a weight value w = −0.4 is set for the attention area 520c.

  If the registration tool 21 determines in step S202 that all the attention areas have been set for the searched image 500, the registration tool 21 shifts the processing to step S203. In step S203, the registration tool 21 generates an XML code in which information indicating the attention area and the weight value w is described. When the XML code is generated, a series of processes for setting the attention area and weighting is completed.

  For example, as illustrated in FIG. 23A, the registration tool 21 generates an XML code 141 in which coordinate information indicating a range on the searched image 500 and a weight value w are associated with each region of interest. be able to. In the example of FIG. 23A, the XML code 141 is a portion (one line) sandwiched between the outermost tags “<page>” and “</ page>”, similar to the XML code 140 shown in FIG. A page (searched image 500) is defined by the (first to 14th lines). And in each range from the 2nd line to the 5th line, the 6th line to the 9th line, the 10th line to the 13th line, by the part sandwiched between the tags “<weightarea>” and “</ weightarea>”, Each region of interest is defined.

  In the range from the 2nd line to the 5th line, the 3rd line is the page of the attention area shown in the 2nd line to the 5th line by the part sandwiched between the tags "<area>" and "</ area>" Is defined by the upper left corner coordinates (x3, y3) and the lower right corner coordinates (x4, y4), for example. Further, the fourth line defines the weight value w set for the attention area shown in the second to fifth lines by the portion sandwiched between the tags “<weight>” and “</ weight>”. Is done. The same applies to each range of the 6th to 9th lines and the 10th to 13th lines.

  The XML code 141 in FIG. 23A is transmitted to the registration server 50 in association with the image identification information of the corresponding image, for example, similarly to the XML code 140 described above.

  In the example of FIG. 23A, the XML code 141 in which the information indicating the attention area and the weight value w is described is generated independently of the XML code 140 indicating the target area, but this is limited to this example. Not. For example, as illustrated in FIG. 23B, the registration tool 21 can also generate an XML code 141 ′ in which information indicating the attention area and the weight value w is embedded in the XML code indicating the target area.

  In the example of FIG. 23B, the second to fifteenth lines correspond to the portions sandwiched between the tags “<page>” and “</ page>” of the XML code 140 in FIG. doing. The 16th to 27th lines correspond to the 2nd to 13th lines in FIG. 23A, and define the attention area and the weight value w. The XML code 140 ′ combines the second to fifteenth lines and the sixteenth to twenty-seventh lines with tags “<page>” and “</ page>”, so that A region of interest and a weight value w are defined.

  FIG. 24 shows an example of a weight value input screen relating to the processing of step S201 and step S202 in the flowchart of FIG. 20 described above. In FIG. 24, an input area 491 and buttons 492 to 495 are arranged on the weight input screen 490. The input area 491 is an area for inputting a weight value w set for the range specified in step S200. In this example, the input area 491 can input a value in the range of −1.0 ≦ w ≦ 1.0 as the weight value w.

  The button 492 is a button for redoing the range specification. When the button 492 is operated, for example, the range designation in the immediately preceding step S200 is canceled, and the range designation operation in step S200 is again awaited.

  The button 493 determines the weight value w input in the input area 491 immediately before (step S201 in FIG. 20), returns the process to step S200, and shifts the process to the setting of the next attention area for the searched image 500. (“No” in step S202 of FIG. 20). For example, when the button 493 is operated, the registration tool 21 causes the storage unit 208 to temporarily associate the weight value w input in the input area 491 immediately before with the information indicating the range of the attention area in the RAM 2003 or the like. Remember me.

  The button 494 confirms the weight value w input in the input area 491 immediately before (step S201 in FIG. 20), and ends the setting of the attention area for the searched image 500 (“Yes” in step S202 in FIG. 20). ) Button. The button 495 is a button for canceling the attention area setting process for the image to be searched 500. When the button 495 is operated, the registration tool 21 discards, for example, the information indicating the range of the attention area set by the series of processes according to the flowchart of FIG. 20 and the weight value w, and the attention area and the weight value w. End the setting process.

  Next, an image search process according to the first embodiment will be described. FIG. 25 is a flowchart illustrating an example of image search processing in the image search server 10 according to the first embodiment. The search terminal 30 transmits a captured image obtained by capturing the subject 40 and an image search request based on the captured image to the image search server 10.

  The image search server 10 receives the captured image and the image search request transmitted from the search terminal 30 through the communication unit 16. The communication unit 16 passes the received captured image and the image search request to the search unit 13. The search unit 13 receives a captured image and an image search request from the communication unit 16, and receives an image search request from the search terminal 30 (step S300).

  In the next step S301, in accordance with the received image search request, the search unit 13 searches the image search DB 11 using the captured image received from the communication unit 16 in step S300 as a search target image, and is registered in the image search DB 11. Of the images to be searched, images to be searched whose similarity s with the image to be searched is a first threshold value or more are extracted as search results. The similarity s takes a value of 0 ≦ s ≦ 1, and for example, s = 0.5 is set as a threshold value. The image search server 10 can extract a plurality of searched images as the search result.

  In the next step S302, the search unit 13 determines whether or not a search target image in which a weighted attention area is set exists in the search target image extracted in step S301. For example, when the XML code 141 describing the information on the attention area is associated with the image identification information of the image to be searched extracted in step S301, the search unit 13 determines that the attention area weighted to the image to be searched is Judge that it is set.

  When the search unit 13 determines that there is no searched image in which the weighted attention area is set in the searched image extracted in step S301, the process proceeds to step S310.

  On the other hand, if the search unit 13 determines that there is a search target image in which a weighted attention area is set in the search target image extracted in step S301, the process proceeds to step S303.

  Henceforth, the process of step S303-step S309 is performed for every to-be-searched image extracted by step S301. Here, it is assumed that m searched images are extracted in step S301, and the processing in steps S303 to S309 is performed for the jth (1 ≦ j ≦ m) searched images among the m searched images. Shall be executed. Further, the processing in steps S303 to S307 is executed for each region of interest included in one search target image. Here, the number of attention areas included in one image to be searched is n, and the processes in steps S303 to S307 are executed for the i-th (1 ≦ i ≦ n) attention area among the n attention areas. Shall be.

In step S303, the search unit 13 executes an image search based on the search target image with respect to the i th attention area T _ij included in the j th searched image S _j extracted in step S301. For example, the search unit 13 cuts out an image in the range of the attention area T _ij from the searched image S _j and obtains the similarity s _ij between the cut-out image and the search target image.

  With reference to FIG. 26, the process of cutting out an image of the region of interest from the image to be searched according to the first embodiment will be described more specifically. In FIG. 26, the same reference numerals are given to the portions common to FIG. 22 described above, and detailed description thereof is omitted.

As an example, a search target image 500 in FIG. 26 is set as a search target image S _j currently being processed. The search unit 13 cuts out images in the range of each region of interest 520a, 520b, and 520c set in the search target image 500, and sets the cut out images as images 520a ′, 520b ′, and 520c ′, respectively. Store in the RAM 1003. These images 520a ′, 520b ′ and 520c ′ are actually image feature information.

In step S303, for example, when i = 1 and the attention area 520a is the first attention area _T1j , the search unit 13 stores the image 520a ′ corresponding to the attention area 520a stored in the RAM 1003 and the search target image. The similarity s _1j is obtained.

  In the above description, the search unit 13 cuts out an image in the range of each region of interest from the search target image in response to the image search request transmitted from the search terminal 30, but this is not limited to this example. For example, it is conceivable to prepare in advance an image in the range of each region of interest set in the search target image.

  More specifically, as illustrated in FIG. 27, the image search server 10 receives the images 501, 502, and 503 in the range of the attention areas 520 a, 520 b, and 520 c set in the search target image 500. In association with the search image 500, for example, it is stored in advance in the image DB. In step S303, an image search is performed on each of the images 501, 502, and 503 stored in the image DB.

  In this method of preparing the image of each region of interest in advance, it is not necessary to cut out the image of the region of interest for each image search request from the search terminal 30, and therefore, for each image search request from the search terminal 30. Compared with the above-described method of cutting out an image, a faster search process can be expected. On the other hand, since the image of each region of interest set for each image to be searched is stored in the image DB 14 in advance, the image DB 14 has a larger capacity than the above-described method of cutting out an image for each image search request from the search terminal 30. Is required.

Returning to the description of FIG. 25, in the next step S304, the search unit 13 determines whether or not the similarity s _ij obtained in step S303 is equal to or greater than a second threshold value. When the search unit 13 determines that the similarity s _ij is less than the second threshold, the search unit 13 shifts the process to step S305, and the search unit 13 processes the attention area T _ij corresponding to the similarity s _ij. Exclude from Then, the process proceeds to step S307.

On the other hand, if the search unit 13 determines in step S304 that the similarity s _ij is greater than or equal to the second threshold value, the search unit 13 passes the similarity s _ij and information on the attention area T _ij to the changing unit 17, The process proceeds to step S306. In step S306, the changing unit 17 weights the similarity s _ij of the attention area T _ij with the weight value w _ij set in the attention area T _ij based on the information passed from the search section. For example, the changing unit 17, a value obtained by adding the weight value w _ij to similarity s _ij, and modified similarity s _ij of the attention area T _ij '. Then, the process proceeds to step S307.

In step S307, the search unit 13 determines whether or not the processing has been completed for all the attention areas T _ij set in the search target image S _j that is currently processed. If it is determined that the search has not ended, the search unit 13 returns the process to step S303, and executes the process for the next attention area T _{(i + 1) j} set in the searched image S _j . In the example of FIG. 26, for example, when the attention area 520b is the second attention area T _2j , the similarity s _2j between the image 520b ′ obtained by cutting out the image in the range of the attention area 520b and the search target image is obtained. It is done.

On the other hand, if the search unit 13 determines in step S307 that the process has been completed for all the attention areas T _ij set in the search target image S _j , the process proceeds to step S308.

In step S308, the changing unit 17 sets the weight value w _ij of each attention area T _ij that is not excluded in step S305 among the attention areas T _ij set in the search target image S _j that is currently processed. There each similarity s _ij 'of the addition, adding the similarity s _j to the search target image of the search target image S _j. Then, the changing unit 17 sets the similarity s _j ′ obtained by the addition as the changed similarity of the searched image S _j .

The process of step S308 will be described more specifically with reference to FIG. In the example of FIG. 28, it is assumed that two attention areas T ₁₀ and T _{20 in} which weight values w ₁₀ and w ₂₀ are set are set for the image to be searched S ₀ , respectively. Further, the similarity of the image to be searched S ₀ to the search target image is the similarity s _0, and the similarities to the search target images of the attention areas T ₁₀ and T ₂₀ are the similarity s ₁₀ and s ₂₀ , respectively. Suppose that

In the first embodiment, the similarity s ₀ ′ of the image to be searched S ₀ changed based on the weight values w ₁₀ and w ₂₀ set in the attention areas T ₁₀ and T ₂₀ is used as a predetermined function f. The following formula (1) is used. In the first embodiment, the function f uses an operation of adding elements in parentheses. That is, the function f (w ₁₀ , s ₁₀ ) = w ₁₀ + s ₁₀ .
s ₀ ′ = s ₀ + f (w ₁₀ , s ₁₀ ) + f (w ₂₀ , s ₂₀ ) (1)

For this way, the changing unit 17 according to the first embodiment, based on the set weighting value w _ij to each region of interest T _ij, the search target image of the search image S _j that contains the region of interest T _ij The similarity value s _j is changed, and the changed similarity s _j ′ is obtained.

  In Expression (1), the function f is not limited to the operation of adding elements in parentheses. For example, the function f may be a multiplication with respect to elements in parentheses, and other operations can be applied. When the function f is multiplication with respect to elements in parentheses, the weight value w may be set to 0 ≦ w ≦ 2, for example.

In the next step S309, the search unit 13 determines whether or not the processing has been completed for all the searched images S _j extracted in step S301. If it is determined that the search has not ended, the search unit 13 returns the process to step S303, and executes the process for the next search target image S _{j + 1} extracted in step S301.

That is, for the next image to be searched S _{j + 1} , the search unit 13 sets each of the images in the range of each region of interest T _{i (j + 1)} set in the image to be searched S _{j + 1} and the search target image. The similarity s _{i (j + 1)} is obtained. The changing unit 17 adds each weight value w _{i (} set to each attention area T _{i (j + 1)} to each similarity s _{i (j + 1)} obtained for the searched image S _{j + 1. j + 1)} determine the respective similarities s _i which is obtained by adding each _{(j + 1)} '. Then, the changing unit 17 adds the obtained s _{i (j + 1)} ′ to the similarity s _{j + 1} between the image to be searched S _{j + 1} and the image to be searched for, and the image to be searched S _{j + 1.} For the modified similarity s _{j + 1} ′.

On the other hand, if the search unit 13 determines in step S309 that the process has been completed for all the search target images _Sj extracted in step S301, the process proceeds to step S310. In step S310, the search unit 13 compares the changed similarities s _j ′ obtained for the search target images S _{j in} the processes of steps S303 to S309 described above.

In the next step S311, as a result of the comparison in step S310, the search unit 13 outputs the search target image S _j having the changed similarity s _j ′ having the largest value as a search result.

The processing of step S310 and step S311 will be described more specifically with reference to FIG. In the example of FIG. 29, four searched images S ₁ , S ₂ , S ₃ and S ₄ are extracted in step S301. For each searched image S ₁ , S ₂ , S ₃ and S ₄ , one attention area T ₁₁ , T ₁₂ , T ₁₃ and T ₁₄ is set, and each attention area T ₁₁ , T ₁₂ , T ₁₃ and T ₁₄ is set. the _14, it is assumed that the weight value w ₁₁ respectively, w _12, w ₁₃ and w ₁₄ are set. Further, in the example of FIG. 29, the similarities of the search target images S ₁ , S ₂ , S ₃ and S ₄ with respect to the search target image are the similarities s ₁ , s ₂ , s ₃ and s ₄ , respectively. Assume that the similarities of the attention regions T ₁₁ , T ₁₂ , T _13, and T ₁₄ with respect to the search target images are similar to s ₁₁ , s ₁₂ , s _13, and s ₁₄ , respectively.

In the case of this example, the modified similarities s ₁ ′, s ₂ ′, s ₃ ′ and s ₄ ′ of the searched images S ₁ , S ₂ , S ₃ and S ₄ obtained in step S308 are as follows: (2) to (5). The meaning of the function f is the same as that in the above formula (1).
s ₁ ′ = s ₁ + f (w ₁₁ , s ₁₁ ) (2)
s ₂ '= s ₂ + f (w ₁₂ , s ₁₂ ) (3)
s ₃ '= s ₃ + f (w ₁₃ , s ₃₁ ) (4)
s ₄ '= s ₄ + f (w ₁₄ , s ₁₄ ) (5)

The search unit 13 compares the similarities s ₁ ′, s ₂ ′, s ₃ ′, and s ₄ ′ determined by the equations (2) to (5) (step S310), and the similarities s ₁ ′, The similarity s ′ having the largest value is selected from s ₂ ′, s ₃ ′ and s ₄ ′. Then, the search unit 13 outputs the search target image S corresponding to the selected similarity s ′ as a search result.

As an example, the following values are considered for each of the searched images S _{1 to} S ₄ .
S ₁ : s ₁ = 0.6, w ₁₁ = 0.5, s ₁₁ = 0.7
S ₂ : s ₂ = 0.7, w ₁₂ = −0.3, s ₁₂ = 0.7
S ₃ : s ₃ = 0.8, w ₁₃ = 0.3, s ₁₃ = 0.8
S ₄ : s ₄ = 0.6, w ₁₄ = −0.7, s ₁₄ = 0.6

In this case, the modified similarities s ₁ ′, s ₂ ′, s ₃ ′, and s ₄ ′ of the searched images S _{1 to} S ₄ are the values described above in the above formulas (2) to (5). Is substituted and is obtained as in the following formulas (6) to (9).
s ₁ '= s ₁ + w ₁₁ + s ₁₁ = 0.6 + 0.5 + 0.7 = 1.8 (6)
s ₂ ′ = s ₂ + w ₁₂ + s ₁₂ = 0.7−0.3 + 0.7 = 1.1 (7)
s ₃ '= s ₃ + w ₁₃ + s ₁₃ = 0.8 + 0.3 + 0.8 = 1.9 (8)
s ₄ '= s ₄ + w ₁₄ + s ₁₄ = 0.6−0.7 + 0.6 = 0.5 (9)

From equations (6) to (9), it can be seen that among the similarities s ₁ ′ to s ₄ ′, the similarity s ₃ ′ = 1.9 in equation (8) is the largest value. Therefore, the detection unit 13 outputs the search target image S ₃ corresponding to the similarity s ₃ ′ as a search result for the image search request by the search terminal 30.

  As described above, according to the first embodiment, the attention area set in the search target image is weighted, and the similarity of the search target image with respect to the search target image is changed according to this weighting. Therefore, a target image that includes a portion similar to the image in the important region by weighting the attention region set for the important region (the portion to be searched) in the searched image with a larger weight value. Thus, it is possible to search the image to be searched including the important region with higher accuracy. As a result, an image search that more accurately reflects the intention of the user (provider of the image to be searched) can be performed. In addition, since the image search is performed using information on portions other than the region of interest, it is possible to avoid a decrease in search accuracy caused by excluding portions other than the region of interest.

  As described above, a negative weight value can be set for the region of interest. Thereby, it is possible to set a portion (referred to as a search suppression region) that is not desired to be searched in the search target image. In this case, it is possible to prevent the search target image from being searched by the image search based on the search target image including an image similar to the image in the search suppression region.

  The search suppression area can also be set as a mask area. FIG. 30 shows an example of masking the search suppression area according to the first embodiment. In FIG. 30, the same reference numerals are given to the portions common to FIG. 22 described above, and detailed description thereof will be omitted.

  In the example of FIG. 30, the mask area 521 is set with the object 510b as a search suppression area. For example, the registration tool 21 forcibly sets the similarity s in the area designated as the mask area 521 to 0. As an example of such a setting, when the function f of the above equation (1) is an addition operation, the weight value w for the mask area 521 is a sign of positive or negative with the similarity s obtained in step S303 and the absolute value being equal. A setting that adaptively generates a value obtained by inverting the above is conceivable. Further, for example, if the calculation of the function f in Expression (1) is a multiplication calculation, the weight value w may be set to 0.

  However, the present invention is not limited to this, and the feature amount in the mask area 521 may be ignored when searching for an image.

(Application example of the first embodiment)
Next, an application example of the first embodiment will be described. Here, an example in which a business card image is searched for by the image search system according to the first embodiment will be described. FIG. 31 shows an example of a business card image. FIG. 31A shows an example of a business card image 550a of a business card with a face photo. In the example of FIG. 31A, the business card image 550a includes objects 551 to 555. For example, the object 551 is a company name logo image, the object 552 is a company name mark image, the object 553 is a character string image indicating a name, the object 554 is a face photograph image, and the object 555 is another character string image. .

  FIG. 31B shows an example of a business card image 550b without a face photo. In FIG. 31 (b), the same reference numerals are given to portions common to the above-described FIG. 31 (a), and detailed description thereof is omitted. In the example of FIG. 31 (b), the business card image 550 b includes objects 551 and 553 and an object 556. The object 556 is assumed to be a character string image indicating attributes for a company name and name such as a department name and address.

  It is assumed that a plurality of business card images are registered in the image search DB 11 of the image search server 10, and further, information of a person with a name corresponding to the business card image is registered as linkage information. As an example, the user obtains a business card as an entity corresponding to the business card image 550a or the business card image 550b, and uses the image search system according to the first embodiment to search for information on a person with a name on the business card. Suppose you want to get In this case, the user captures a business card with the search terminal 30, and acquires a business card image 550a and a business card image 550b as captured images. The search terminal 30 transmits the acquired captured image to the image search server 10 together with an image search request for requesting an image search based on the captured image.

  As described above, the image search server 10 uses the captured image (that is, the business card image 550a and the business card image 550b) transmitted from the search terminal 30 as a search target image, and extracts a feature amount from the search target image. Then, the image search server 10 obtains the similarity between the feature amount extracted from the search target image and the feature amount of each image registered in the image search DB 11, and image identification information corresponding to the feature amount having a high similarity degree. The link information associated with is transmitted to the search terminal 30.

  Here, in the business card image 550a, for example, the object 552 (company name mark image) is an important search element. Similarly, the object 554 that is a face photograph is also an important search element. At this time, the face photo includes more feature points than a character string image, a mark image, or the like. Therefore, in the image search based on the business card image 550a with a business card with a face photo, another business card including a similar face image is included. The possibility that the image is retrieved is considered high.

  In addition, in the business card image 550b that does not include a face photo, if a large number of business card images including employees 551 of the same company name logo image 551 are registered in the image search DB 11, the object 551 is displayed. It is thought that it is necessary to mask.

FIG. 32 shows an example of weighting for enabling a search by name based on the business card images 550a and 550b to be executed with higher accuracy. In FIG. 32, the weight value w is shown as the importance level Im. As shown in Expression (10), the importance Im is a value obtained by adding 1 to the weight value w. When the weight value w is in a range of −1 ≦ w ≦ 1, the importance Im is 0 ≦ Im ≦ 2. When the specified range is important for the search, a value greater than 1 is set as the importance Im, and when it is desired to suppress the search by the specified range, a value less than 1 is set as the importance Im. It is considered that the weight value w can be set more intuitively by using this importance degree Im.
Im = w + 1 (10)

  FIG. 32A shows an example of weighting for a business card image 550a with a face photo. In this example, importance Im = 2.0 is set in the area 560c including the object 553 based on the character string image of the name, which is more important. Also, the importance Im = 0.3 is set in the parts 560a and 560b including the objects 551 and 552 of the company name logo image and the company name mark image, respectively, to suppress the search, and the object 554 by the face photograph image is set. Importance Im = 0.6 is set in the area 560d to be included.

  By setting the importance Im of each of the areas 560a to 560d in this way, the influence on the search by the face photo image, the company logo image, and the company name mark image is weakened, and the character string image of the name is further emphasized. Searches using character string images of names can be executed with higher accuracy. As a result, an image search that more accurately reflects the user's intention can be performed.

  FIG. 32B shows an example of weighting for the business card image 550b without a face photo. Also in this example, the importance Im = 2.0 is set in the region 560f including the object 553 by the name character string image, which is more important, as in the example of FIG. In addition, the importance Im = 0.1 is set in the area 560e including the object 551 of the company name logo image, which is the part for which the search is to be suppressed. In the case of this example as well, the influence on the search by the company name logo image is weakened, the name string image is more emphasized, and the search by the name string image can be executed with higher accuracy. Thus, an image search that more accurately reflects the user's intention can be performed.

(Second Embodiment)
Next, a second embodiment will be described. In the second embodiment, the weighting according to the first embodiment described above is applied to the similarity check (step S124, step S130) described with reference to FIG.

  Here, the similarity check will be described. If a plurality of images having similar feature information are registered in the image DB 14 of the image search server 10, the image search server 10 may be erroneously recognized during the image search. Therefore, when registering an image from the registration terminal 20 to the registration server 50 using the registration tool 21, it is preferable to check whether an image similar to the image has already been registered in the registration server 50. Then, the registration terminal 20 performs processing for avoiding erroneous recognition at the time of image search according to the check result.

  Here, when the similarity is determined using the image feature information, for example, an image partially similar to the image to be checked can be searched as a similar image. As an example, consider a case where an image 570 including a plurality of objects 571a, 571b, and 571c is registered in the registration server 50 using the registration tool 21, as illustrated in FIG. At this time, it is assumed that an image 572 including only the object 573 having high similarity with the object 571a, which is exemplified in FIG. 33B, has already been registered in the registration server 50.

  In this case, the registration tool 21 transmits the image 570 from the registration terminal 20 to the registration server 50 and requests the registration server 50 to check a similar image for the image 570. The registration server 50 returns information indicating the image 572 including the object 573 having a high similarity to the object 571b included in a partial area in the image 570 transmitted by the registration tool 21 to the registration terminal 20 as a search result. Therefore, the registration tool 21 determines that an image similar to the image 570 is already registered in the registration server 50.

  Here, the user operating the registration tool 21 has the object 571a included in the image 570 that is determined to have a high similarity to the image determined to be already registered in the registration server 50. , 571b and 571c, it is difficult to know. That is, the image is registered as feature information in the image DB 14 of the registration server 50, and it is considered difficult for the user to know the original image from the feature information. Therefore, there is a possibility that it is extremely difficult for the user to determine what editing is to be performed on the image 570 in order to avoid the determination that the similar image is registered in the registration server 50.

  Therefore, in the second embodiment, a check area is set for an image to be checked, and a similar image check is executed for images in the set check area. Then, according to the check result, weighting is performed to suppress the search for the check area.

  More specifically, the registration tool 21 cuts out an image in the set area from the check target image, and transmits the cut image to the registration server 50 together with an image search request for requesting an image search for the image. In response to this image search request, the registration server 50 searches the image search DB 11 ′ for an image having a similarity equal to or higher than a threshold with respect to the image transmitted from the registration terminal 20, and stores the search result as the registration terminal 20. Send to.

  In the registration terminal 20, when the search result transmitted from the registration server 50 indicates that an image having a similarity equal to or higher than a threshold is registered, the registration tool 21 is an image similar to an image in the set area. Is already registered in the registration server 50, a negative weight value is set for the area to suppress the search for the area.

  FIG. 34 is a flowchart illustrating an example of weighting processing in the similar image check according to the second embodiment. For example, when it is determined that the registration tool 21 is a new registration in step S126 in the flowchart of FIG. 8, or the button 4888 arranged on the editing screen 480 illustrated in FIG. 14 is operated and a similar image check is instructed. In this case, the processing according to the flowchart of FIG. 34 is started.

  Prior to the start of the processing according to the flowchart of FIG. 34, it is assumed that an image to be subjected to a similar image check is already displayed in the page editing area 484 of the editing screen 480. Hereinafter, it is assumed that the image 570 described with reference to FIG. 33A is used as an image to be subjected to the similar image check, and the image 570 is appropriately referred to as a check target image 570.

  In step S <b> 400, the registration tool 21 designates an area (referred to as a check area) on an image that is a target for determining the presence or absence of a similar image for the check target image 570 in accordance with a user operation. At this time, it is assumed that the registration tool 21 can specify a plurality of check areas for the check target image 570. FIG. 35 shows an example in which a plurality of check areas 574a, 574b, and 574c including objects 571a, 571b, and 571c are designated in the check target image 570, respectively.

  Henceforth, the process of step S401-step S405 becomes a process for every check area | region. In step S401, the registration tool 21 requests the registration server 50 to search for an image in the check area for one of the check areas specified in step S400. For example, the registration tool 21 uses the extraction unit 202 to extract an image in the check area 574a designated in step S400, and transmits the extracted image to the registration server 50 together with an image search request for the image.

  The registration server 50 receives the image and the image search request transmitted from the registration terminal 20, and obtains the similarity s between the image and each image registered in the image search DB 11 ′ in response to the image search request. . The registration server 50 transmits the highest similarity s among the obtained similarities s to the registration terminal 20 as a search result. The similarity s as a search result transmitted from the registration server 50 is received by the registration terminal 20 and acquired by the registration tool 21 (step S402).

  In the next step S403, the registration tool 21 uses the determination unit 203 to compare the similarity s acquired in step S402 with the third threshold value and perform threshold determination. If the registration tool 21 determines that the similarity s is less than the third threshold value, the registration tool 21 returns the process to step S401, and executes the process for the next check area (for example, the check area 574b).

  On the other hand, if the registration tool 21 determines in step S403 that the similarity s is greater than or equal to the third threshold, the registration tool 21 shifts the processing to step S404. In step S404, the registration tool 21 sets a negative weight value w that suppresses the search, for the check area (for example, the check area 574a) currently being processed. For example, the registration tool 21 sets a weight value w = −1 for the check area.

  In the next step S405, the registration tool 21 determines whether or not the processing for all the check areas designated in step S400 has been completed. If the registration tool 21 determines that the processing for all check areas has not been completed, it returns the process to step S401 and executes the process for the next check area (for example, the check area 574b).

  If the registration tool 21 determines in step S405 that the processing for all check areas has been completed, the registration tool 21 shifts the processing to step S406. In step S406, the registration tool 21 generates an XML code indicating each check area and the weight value w. The code generated in step S406 can be the same as the code 141 described with reference to FIG. 23A, for example, and detailed description thereof is omitted here. When the XML code is generated, a series of similar image check processes is terminated.

  FIG. 36 shows an example of a result of the similar image check for the check target image 570 according to the second embodiment. In the example of FIG. 36, in the threshold determination in step S403, it is determined that the acquired similarity s is not less than the third threshold only in the check area 574a, and acquired in the other check areas 574b and 574c, respectively. In this example, the similarity s is determined to be less than the third threshold. In this case, a negative weight value w is set for the check area 574a, and no weight value w is set for the other check areas 574b and 574c.

  As described above, in the second embodiment, an image search is performed for each check region image specified for the check target image, and a similar image check is performed based on the similarity s obtained as a result of the image search. Is going. Then, as a result of the similar image check, a check area having a high similarity s is regarded as an image similar to the area already registered, and a weight value for suppressing the search for the check area. Is set.

  Therefore, even when a new image including an image similar to an already registered image is newly registered, it is possible to suppress a newly registered image from being searched for an already registered image. In this case, it is not necessary to edit the newly registered image.

  In the above description, when the registration tool 21 determines in step S403 that the similarity s is greater than or equal to the third threshold, the registration tool 21 sets a negative weight value w for the check area. This is not limited to this example. For example, as described above, the setting may be made such that the similarity s in the check area is forcibly set to 0, or the feature amount in the check area may be ignored.

(First Modification of Second Embodiment)
Next, a first modification of the second embodiment will be described. In the above-described second embodiment, the weight value w for suppressing the search is set for the check area having the similarity s greater than or equal to the threshold. On the other hand, in the first modification of the second embodiment, another check area different from the check area where the similarity s is greater than or equal to the threshold, that is, the check area where the similarity s is less than the threshold is set as the important area. For example, a positive weight value w greater than 0 is set. By doing in this way, it is possible to suppress a search for a region that is considered that a similar image has already been registered.

  FIG. 37 is a flowchart of an example illustrating weighting processing in the similar image check according to the first modification of the second embodiment. For example, when it is determined that the registration tool 21 is a new registration in step S126 in the flowchart of FIG. 8, or the button 4888 arranged on the editing screen 480 illustrated in FIG. 14 is operated and a similar image check is instructed. In this case, the processing according to the flowchart of FIG. 37 is started. Also, it is assumed that an image (for example, a check target image 570) that is a target of a similar image check has already been displayed in the page edit area 484 of the edit screen 480 prior to the start of the processing according to the flowchart of FIG.

  In step S410, the registration tool 21 specifies one or more check areas for the check target image 570 in accordance with a user operation.

  Henceforth, the process of step S411-step S413 becomes a process for every check area | region. In step S411, the registration tool 21 requests the registration server 50 to search for an image in the check area for one check area in the check area specified in step S410, as in step S401 described above. In response to the image transmitted from the registration terminal 20 and the image search request, the registration server 50 obtains the similarity s between the image and each image registered in the image search DB 11 ′, and obtains the similarity s of the obtained similarity s. The highest similarity s is transmitted to the registration terminal 20 as a search result. The similarity s is received by the registration terminal 20 and acquired by the registration tool 21 (step S412).

  In the next step S413, the registration tool 21 determines whether or not the processing for all the check areas designated in step S410 has been completed. If the registration tool 21 determines that the processing for all check areas has not been completed, the registration tool 21 returns the process to step S411 and executes the process for the next check area.

  If the registration tool 21 determines in step S413 that the processing for all the check areas has been completed, the registration tool 21 shifts the processing to step S414. In step S414, the registration tool 21 performs threshold determination based on each similarity s acquired in step S412 for each check area specified in step S410 by the determination unit 203, and the similarity s is less than the fourth threshold. Extract check areas. Then, the registration tool 21 sets a positive weight value w greater than 0 for the check area where the extracted similarity s is less than the fourth threshold. In the next step S415, the registration tool 21 generates an XML code indicating each check region and the weight value w.

  Here, the weight value w is collectively set for a plurality of check areas, but this is not limited to this example. Also in the first modification of the second embodiment, the method of sequentially setting the weight value w for each check area described in the flowchart of FIG. 34 in the second embodiment described above can be applied. is there. Similarly, the method of collectively setting the weight value w for a plurality of check areas according to the first modification of the second embodiment can be applied to the second embodiment described above. .

(Second modification of the second embodiment)
Next, a second modification of the second embodiment will be described with reference to FIGS. 38 and 39. FIG. In FIG. 38 and FIG. 39, the same reference numerals are given to the portions common to FIG. 35 described above, and detailed description thereof is omitted.

In the second modified example of the second embodiment, the check target image is divided and each divided area is set as a check area, and a similar image check is performed on each check area. FIG. 38A is an example in which the check target image 570 is divided into three in the vertical direction, and the check areas 575 _T , 575 _C, and 575 _B are formed from the top. Further, FIG. 38B divides the check target image 570 into four in a grid pattern, and the upper left check area 575 _LT , the upper right check area 575 _RT , the lower left check area 575 _LB, and the lower right check area. This is an example of forming 575 _RB .

  The division position and the number of divisions for dividing the check target image 570 are not particularly limited. The registration tool 21 may set the position designated by the user operation as the division position, or may set the division position according to the content of the check target image 570. Of course, the division position may be fixedly set as in the example of FIGS. 38 (a) and 38 (b). Hereinafter, as illustrated in FIG. 38B, the check target image 570 is described as being divided into four in a lattice shape.

For example, when the button 4888 is operated on the editing screen 480 and the similar image check is instructed, or in the similar image check in step S130 of FIG. 8, the registration tool 21 checks from the image 570 to the check areas 575 _LT 575 _RT , The images of 575 _LB and 575 _RB are cut out, and each cut out image is transmitted to the registration server 50.

Registration server 50, the image each of these check area 575 _LT, 575 _RT, 575 _LB and 575 _RB, executes the search processing of the similar image, and the check area 575 _LT, 575 _RT, 575 _LB and 575 _RB, image retrieval The similarity s with each image registered in the DB 11 ′ is obtained. Then, the maximum similarity s is obtained for each of the check areas 575 _LT , 575 _RT , 575 _LB, and 575 _RB . The registration server 50 transmits the maximum similarity s of the check areas 575 _LT , 575 _RT , 575 _LB, and 575 _{RB to} the registration terminal 20 as a search result. The registration terminal 20 passes each received search result to the registration tool 21.

  The registration tool 21 determines each similarity s shown in each search result as a threshold value, and an image similar to an image in the check region corresponding to the similarity s whose value is equal to or greater than the fourth threshold value is already stored in the image search DB 11. It is determined that it is registered (NG). Further, as a result of the threshold determination, the registration tool 21 does not register a similar image in the image search DB 11 ′ for the image in the check area corresponding to the similarity s whose value is less than the fourth threshold (OK). Is determined.

  The registration tool 21 sets, for example, a negative weight value w that suppresses the search for a check area in which it is determined that a similar image has already been registered in the image search DB 11 ′, and performs other checks. No weight value w is set for the area. Alternatively, the registration tool 21 uses, for example, a positive value greater than 0, for example, for a check area in which it is determined that a similar image is not already registered in the image search DB 11 ′. A weight value w is set.

FIG. 39 shows an example of a result of the similar image check for the check target image 570 according to the second modification of the second embodiment. In the example of FIG. 39, it is assumed that the similarity s acquired in the check region 575 _LT is equal to or greater than the fourth threshold value, and in the other check regions 575 _RT , 575 _LB and 575 _RB , the acquired similarity s is It is an example determined to be less than the fourth threshold. In this case, a negative weight value w is set for the check area 575 _LT , and no weight value w is set for the other check areas 575 _RT , 575 _LB, and 575 _RB .

Of course, also in the second modification example of the second embodiment, as in the first modification example of the second embodiment described above, the check areas 575 _RT and 575 where the similarity s is less than the fourth threshold value. A weight value w having a positive value greater than 0 may be set in _LB and 575 _RB , and the weight value w may not be set in the check region 575 _LT in which the similarity s is determined to be greater than or equal to the fourth threshold value. .

  In the second modification of the second embodiment, in addition to the effects of the second embodiment described above, the division position for the check target image 570 is determined in advance, so that the user operating the registration terminal 20 The burden on the similar image check can be reduced.

  Each of the above-described embodiments is a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the gist of the present invention.

1 Information processing system 10 Image search server 11, 11 ′ Image search DB
12 Analysis unit 13 Search unit 14 Image DB
15 Cooperation information DB
20 registration terminal 21 registration tool 30 search terminal 50 registration server 60 user management server 61 user DB
201 Acquisition unit 202 Extraction unit 203 Determination unit 204 Notification unit 205 Communication unit 206 Display unit 210 Area setting unit 211 Weight setting unit 400 Login screen 410 Menu screen 420 Registration information list screen 480 Editing screen 484 Page editing area 485 Preview area 486 Target area list region 487 link icon list area 488 toolbar _{500, S 0, S 1,} S 2, S 3, S 4 the searched image _{520a, 520b, 520c, T 10} , T 11, T 12, T 13, T 14 region of interest _{574a, 574b, 574c, 575 T} , 575 C, 575 B, 575 LT, 575 RT, 575 LB, 575 RB check area

JP, 2014-010723, A

Claims (10)

A search unit that searches the search target image based on a similarity to the search target image from a registration unit that is registered in association with the terminal device, the search target image, and the link information, and the searched search target image An information processing system comprising: an image search device including a transmission unit that transmits associated cooperation information to the terminal device; and an information processing device that registers the search target image in the registration unit,
An area setting unit for setting an attention area for the search image;
A weight setting unit for setting a weight for the region of interest;
An information processing system comprising: a changing unit that changes the similarity based on the weight. The weight setting unit includes:
The information processing system according to claim 1, wherein the weight of a specified value is set. The changing unit is
A first similarity between the image of the region of interest and the search target image is weighted based on the weight, and the weighted first similarity is used as a second similarity between the search target image and the search target image. The information processing system according to claim 1, wherein the search unit obtains a third similarity for searching the image to be searched. The changing unit is
The information processing system according to claim 3, wherein the third similarity is obtained when the second similarity is equal to or greater than a threshold value. An extraction unit that extracts a predetermined region from a registration target image for registration in the registration unit;
A determination unit that transmits an image of the predetermined region to the search unit, and determines whether a similar image including an image similar to the predetermined region is registered in the registration unit based on a search result of the search unit; Further comprising
The region setting unit
When it is determined that the similar image is registered in the registration unit, an area different from the predetermined area of the registration target image is set as the attention area,
The weight setting unit includes:
The information processing system according to any one of claims 1 to 4, wherein a weight for increasing the similarity is set for the region of interest. An extraction unit that extracts a predetermined region from a registration target image for registration in the registration unit;
A determination unit that transmits an image of the predetermined region to the search unit, and determines whether a similar image including an image similar to the predetermined region is registered in the registration unit based on a search result of the search unit; Further comprising
The region setting unit
When it is determined that the similar image is registered in the registration unit, the predetermined area of the registration target image is set as the attention area,
The weight setting unit includes:
The information processing system according to any one of claims 1 to 4, wherein a weight for decreasing the similarity is set for the region of interest. A search unit that searches for a search target image based on a similarity to a search target image from a registration unit that is registered in association with the search target image and link information, and link information that is associated with the searched search target image An information processing apparatus for registering the search target image with respect to the registration unit of an image search apparatus comprising: a transmission unit that transmits to the terminal device;
An area setting unit for setting an attention area for the search image;
An information processing apparatus comprising: a weight setting unit configured to set a weight for changing the degree of similarity with respect to the attention area. A registration unit in which a search target image, linkage information, and weight information indicating a weight set in a region of interest provided in the search target image are associated and registered;
A search unit that searches for the image to be searched based on the similarity to the search target image transmitted from the terminal device;
A transmission unit that transmits cooperation information associated with the searched image to the terminal device;
An information processing apparatus comprising: a changing unit that changes the similarity based on the weight indicated by the weight information. A search unit that searches for a search target image based on a similarity to a search target image from a registration unit that is registered in association with the search target image and link information, and link information that is associated with the searched search target image An information processing method for registering the image to be searched with respect to the registration unit of an image search device comprising: a transmission unit that transmits to the terminal device,
A region setting step for setting a region of interest for the image to be searched;
A weight setting step for setting a weight for the region of interest;
An information processing method comprising: a changing step of changing the similarity based on the weight. A search unit that searches for a search target image based on a similarity to a search target image from a registration unit that is registered in association with the search target image and link information, and link information that is associated with the searched search target image An information processing program for causing a computer to execute an information processing method for registering the image to be searched with respect to the registration unit of an image search device including a transmission unit that transmits a message to the terminal device,
A region setting step for setting a region of interest for the image to be searched;
An information processing program comprising: a weight setting step for setting a weight for changing the similarity to the attention area.

Images