WO2012124149A1 - 画像処理装置、画像処理方法および制御プログラム - Google Patents
画像処理装置、画像処理方法および制御プログラム Download PDFInfo
- Publication number
- WO2012124149A1 WO2012124149A1 PCT/JP2011/056827 JP2011056827W WO2012124149A1 WO 2012124149 A1 WO2012124149 A1 WO 2012124149A1 JP 2011056827 W JP2011056827 W JP 2011056827W WO 2012124149 A1 WO2012124149 A1 WO 2012124149A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- query image
- unit
- query
- user
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/532—Query formulation, e.g. graphical querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/14—Digital output to display device ; Cooperation and interconnection of the display device with other functional units
- G06F3/1415—Digital output to display device ; Cooperation and interconnection of the display device with other functional units with means for detecting differences between the image stored in the host and the images displayed on the displays
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
- G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
Definitions
- the present invention relates to an image processing apparatus, an image processing method, and a control program for searching for an image.
- Patent Document 1 describes a technique for performing edge detection on a query image and searching for an image similar to the query image based on a schematic image including only line segment information. More specifically, Patent Literature 1 discloses a global threshold value based on the edge strength of the entire query image, and a local threshold value based on the edge strength of the target pixel and its neighboring pixels included in the query image. Schematic image creation that determines whether the pixel of interest is an edge based on the larger of the global threshold value and the local threshold value, and creates a schematic image from the query image An apparatus is described.
- Patent Document 2 describes an image processing apparatus that performs a search by combining a plurality of query images, and can specify a logical expression that defines a combination of query images by a sensuous operation. Yes.
- Patent No. 2806666 Publication Date: September 30, 1998)
- Japanese Patent Publication Japanese Patent Laid-Open No. 2010-250426 (Released on November 4, 2010)”
- the user when the user tries to search for an image showing the building A, the user uses the image of the building A as the query image. Further, it is assumed that the image of the building A, which is a query image, includes other objects and backgrounds in addition to the building A. In this case, in the technique described in Patent Document 1, since the entire region (all pixels) of the image of the building A, which is the query image, is used as the schematic image, the created schematic image includes objects other than the building A, Information such as background is included.
- the image search apparatus using the technique described in Patent Document 1 searches for an image in which an object similar to the building A is captured when searching with the created schematic image, Search for an image that includes the background. That is, the image search device also searches for an image other than an image similar to the building A desired by the user. Therefore, when the technique described in Patent Document 1 is used, there is a problem that the image search apparatus performs extra search processing.
- the image search apparatus using the technique described in Patent Document 1 does not show the building A other than the image showing the object similar to the building A, and other than the building A.
- An image showing an object or background is also displayed as a search result. Therefore, it takes time and effort for the user to search for an image showing the building A from the search results.
- the image processing apparatus described in Patent Literature 2 when the user tries to search for an image in which the flower B and the flower C are captured, the user uses the image of the flower B and the image of the flower C as the query image. And In this case, in the technique described in Patent Document 2, an image in which an object similar to the flower B is captured and an image in which an object similar to the flower C is captured are once searched, and the logical product of each search result is calculated. Present to the user. That is, the image processing apparatus described in Patent Literature 2 also searches for an image other than an image in which an object similar to the flower B desired by the user and an object similar to the flower C are shown. Therefore, there is a problem that the image processing apparatus described in Patent Document 2 performs an extra search process.
- the present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus, an image processing method, and a control program for efficiently searching for an image that a human thinks is intuitively similar to a query image. It is to be realized.
- an image processing apparatus includes a query image specifying unit that specifies a query image based on designation from a user, an object detection unit that detects an object from the query image, and the object detection By combining a plurality of the objects detected by the means, a combined query image generating means for generating a combined query image including a plurality of the objects, and searching for an image similar to the combined query image based on the combined query image And an image search means.
- an image processing method includes a query image specifying step that specifies a query image based on designation from a user, an object detection step that detects an object from the query image, and the object detection
- a composite query image generating step for generating a composite query image including a plurality of the objects by combining a plurality of the objects detected in the step, and an image similar to the composite query image based on the composite query image
- an image search step for searching.
- a user searches for an image similar to one or a plurality of query images including a plurality of objects
- the user generally searches for an image similar to an image including a plurality of objects included in these query images. It is expected that
- the object detecting means detects an object from the query image specified by the query image specifying means based on a designation from the user, and the synthesized query image generating means synthesizes a plurality of the objects detected by the object detecting means.
- a synthesized query image including a plurality of the objects is generated, and the image search means searches for an image similar to the synthesized query image based on the synthesized query image.
- the object can be automatically detected from the query image specified by the user. Therefore, by searching using a synthetic query image including an object detected from the query image, it is possible to efficiently search for an image that the user is looking for. That is, it is possible to reduce unnecessary search processing in the image processing apparatus.
- the search is performed in units of a combined query image including a plurality of objects instead of searching in units of objects, thereby reducing the processing load of the image processing apparatus. be able to.
- the image processing apparatus includes a query image specifying unit that specifies a query image based on designation from a user, an object detection unit that detects an object from the query image, and a detection by the object detection unit.
- a composite query image generating means for generating a composite query image including a plurality of the objects by combining a plurality of the objects, and an image search for searching for an image similar to the composite query image based on the composite query image Means.
- the image processing method includes a query image specifying step for specifying a query image based on a designation from a user, an object detection step for detecting an object from the query image, and the above-described detection detected in the object detection step.
- a combined query image generating step for generating a combined query image including a plurality of the objects by combining a plurality of objects, and an image searching step for searching for an image similar to the combined query image based on the combined query image; including.
- FIG. 1 is a block diagram illustrating an example of a main configuration of an image processing apparatus.
- the image processing apparatus generates search image information (synthetic query image) for searching for an image similar to a query image, based on one or more query images specified by a user.
- the image processing apparatus when searching for an image in which the person E is captured in front of the building D, such as the image SP1, an image QP1 in which the building D is captured and an image in which the person E is captured. Assume that the user selects QP2 as a query image. At this time, the image processing apparatus according to the present invention extracts information indicating the building D and information indicating the person E from the images QP1 and QP2. Next, the image processing apparatus according to the present invention combines the extracted information into one image, and creates a combined query image CQP1 that is a search image for searching the image SP1.
- the image processing apparatus searches for an image similar to the created composite query image CQP1, and displays the search result.
- the user selects a desired image from the displayed search results.
- the image processing apparatus creates the synthesized query image CQP1 similar to the image SP1 desired by the user from the query images (P1 and P2) designated by the user.
- FIG. 1 is a block diagram illustrating an example of a main configuration of the image processing apparatus 1.
- the image processing apparatus 1 includes a control unit 11, a storage unit 12, an image input unit 13, an operation unit (input unit) 14, a communication unit 15, and a display unit 16.
- the image processing apparatus 1 may include members such as a sound input unit and a sound output unit, but these members are not shown because they are not related to the feature points of the invention.
- the image input unit 13 acquires an image from the image providing device 3.
- the image providing device 3 may be anything as long as it is a device that provides a held image or an acquired image to another device.
- the image providing device 3 is a storage device such as a digital camera, a digital television, a PC, a mobile phone, a PDA (Personal Digital Assistant), a game machine, or a USB (Universal Serial Bus) memory.
- the operation unit 14 is for a user to input an instruction signal to the image processing apparatus 1 and operate the image processing apparatus 1.
- the operation unit 14 may be configured with input devices such as a keyboard, a mouse, a keypad, and operation buttons. Further, a touch panel in which the operation unit 14 and the display unit 16 are integrated may be used.
- the operation unit 14 may be a remote control device such as a remote controller separate from the image processing device 1.
- the communication unit 15 communicates with other devices such as the image database 2 by wireless communication means or wired communication means, and exchanges data according to instructions from the control unit 11. For example, the communication unit 15 acquires an image from the image database 2 in accordance with an instruction from the control unit 11.
- the image database 2 is a storage device or a Web server that holds images.
- the display unit 16 displays an image according to an instruction from the control unit 11.
- the display unit 16 only needs to display an image in accordance with an instruction from the control unit 11.
- an LCD liquid crystal display
- an organic EL display organic EL display
- a plasma display or the like can be applied.
- the control unit 11 performs various operations by executing a program read from the storage unit 12 to a temporary storage unit (not shown), and comprehensively controls each unit included in the image processing apparatus 1. is there.
- the control unit 11 includes, as functional blocks, a query image specifying unit (query image specifying unit) 21, an object extracting unit (object detecting unit) 22, a synthetic query image creating unit (combined query image generating unit) 25, This is a configuration provided with a synthetic query image adjusting unit (synthetic query image adjusting unit) 26 and a searching unit (image searching unit) 27.
- Each of the functional blocks (21, 22, 25 to 27) of the control unit 11 stores a program stored in a storage device in which a CPU (central processing unit) is realized by a ROM (read only memory) or the like (RAM ( It can be realized by reading and executing it in the temporary storage part realized by random (access memory).
- the query image specifying unit 21 specifies an image designated by the user as a query image. Specifically, when the user designates an image displayed on the display unit 16 using the operation unit 14, the query image specifying unit 21 acquires information indicating the image designated by the user from the operation unit 14. The image indicated by the acquired information is specified as the query image.
- the image specified by the user as the query image may be one or plural. That is, the image (query image) specified by the query image specifying unit 21 may be one or plural.
- the “image displayed on the display unit 16” means, for example, that the control unit 11 acquires an image from the image providing device 3 via the image input unit 13 and displays the acquired image on the display unit 16. It may be an image. In addition, the control unit 11 may acquire an image from the image database 2 via the communication unit 15 and display the acquired image on the display unit 16 such as when the user is browsing the Web. . Alternatively, the control unit 11 may read an image from the image storage unit 31 and display the read image on the display unit 16.
- the user is not limited to selecting a query image from the images displayed on the display unit 16.
- the user designates the directory address of the image data in the image database 2, the image providing device 3, or the image storage unit 31, and the images held by the image database 2, the image providing device 3, or the image storage unit 31 are stored. It may be specified as a query image.
- the query image specifying unit 21 acquires the specified image (query image) from the image input unit 13, from the image database 2 via the communication unit 15, or from the image storage unit 31.
- the query image specifying unit 21 outputs the acquired query image to the object extracting unit 22.
- the object extracting unit 22 acquires a query image from the query image specifying unit 21, detects an object from the acquired query image, and extracts the detected object.
- the object extraction unit 22 generates object information indicating the extracted object, and outputs the generated object information to the synthesized query image creation unit 25.
- the object extraction unit 22 may add position information and size information indicating the position and size of each object in the query image as additional information of the object to the object information.
- the object includes an object such as a building or a person, a background such as the sea or sunset sky, or an abstract pattern such as “ ⁇ ” or “ ⁇ ”.
- the object information may be information indicating the pixel value of the pixel group of the object region in the image, or information indicating the feature amount of the object such as edge information indicating the edge (contour) of the object. It may be. Further, the additional information of the object does not need to include both the position information and the size information, and may include at least one.
- the object extraction unit 22 includes an object detection unit 23 and a region extraction unit 24. More specifically, the object detection unit 23 and the region extraction unit 24 generate object information.
- the object detection unit 23 reads out an image template that is a standard image of the object from the storage unit 12, matches the query image with the image template, and the query image includes the same object as the matched image template. It is determined whether or not. If the object detection unit 23 determines that the same object as the matched image template is included, the object detection unit 23 extracts the object from the query image, and generates object information indicating the extracted object.
- the object detection unit 23 reads a feature amount template indicating a feature amount of a standard image of the object from the storage unit 12, calculates a feature amount of the query image, and calculates the feature amount of the query image and the feature amount template. Perform matching. Then, it is determined whether or not the query image includes the same object as the object having the feature amount indicated by the matched feature amount template. If the object detection unit 23 determines that the same object as the object having the feature amount indicated by the matched feature amount template is included, the object detection unit 23 extracts the object from the query image, and generates object information indicating the extracted object.
- the object detection unit 23 may detect a face, a person (body), a pet, a car, or the like, or may further detect an individual (such as an individual). Further, the object detection unit 23 may detect backgrounds (scenes) such as the sea, mountains, and sunset sky.
- the object detection unit 23 adds the object name information indicating the name of the object to the object information indicating the extracted object. It may be added as information.
- the region extraction unit 24 uses a region such as Saliency Map (see Non-Patent Document 1) or region segmentation processing (Segmentation: see Non-Patent Document 2) and other characteristic regions (pixel groups) from the query image. Is extracted, the extracted area is specified as the object area, and object information is generated.
- a region such as Saliency Map (see Non-Patent Document 1) or region segmentation processing (Segmentation: see Non-Patent Document 2) and other characteristic regions (pixel groups) from the query image. Is extracted, the extracted area is specified as the object area, and object information is generated.
- the region extraction unit 24 when using the Saliency Map, the region extraction unit 24 generates a feature map indicating the contrast of feature quantities such as color, brightness, and edge from the query image, and adds and averages each pixel of each feature map.
- a map is generated, and a region having a high contrast in SM (for example, a pixel group having a pixel value equal to or larger than a predetermined value) is extracted.
- Saliency Map is a model of human visual processing, and by extracting a region using Saliency Map, it is possible to automatically specify a region that is likely to be noticed by humans.
- an area dividing process by integrating adjacent pixels, an area dividing process by classifying pixel features, or an area dividing process by a technique called snakes using edges, etc. May be applied.
- the synthetic query image creation unit 25 selects one or a plurality of objects from the objects indicated by the object information generated by the object extraction unit 22, and creates a synthetic query image by pasting the selected objects on a blank image. Is.
- the composite query image creation unit 25 may select an object according to an instruction from the user input to the operation unit 14, or may select an object according to a predetermined selection rule.
- the selection rule may be implemented by all combinations of objects, or may be all combinations of two or more objects, all combinations of three or more objects, or the like.
- the object extraction unit 22 generates object information indicating three objects “L”, “M”, and “N”.
- the synthetic query image creation unit 25 selects “L”, “M”, “N”, “L and M”, “M and N”, “N and L”, and “L and M and N”. You may select by the combination of streets and create seven synthetic query images.
- rules for excluding combinations of objects may be determined in advance. For example, when it is determined that the combination of “M and N” is excluded as a rule for excluding the combination of each object, “L”, “M”, and “N” are performed based on the above selection rule. , “L and M”, “N and L”, and “L and M and N” are selected in combination (in this case, “L and M and N” are also combinations of “M and N”) May be excluded as being included). In this case, it is assumed that the object detection unit 23 specifies at least the names of the objects “M” and “N”.
- the combined query image creating unit 25 determines the position of the object in the blank image and the position and size indicated by the size information and the position information indicated by the additional information. Determine the size.
- the composite query image creation unit 25 determines the position and size of the object in the blank image according to a predetermined arrangement rule.
- the arrangement rules include, for example, dividing a blank image by the number of selected objects and arranging the objects so that the objects fit in the divided areas.
- the synthesized query image creating unit 25 outputs synthesized query image information indicating the created synthesized query image to the synthesized query image adjusting unit 26.
- the synthetic query image adjustment unit 26 adjusts the position and size of the object included in the synthetic query image created by the synthetic query image creation unit 25 in the synthetic query image.
- the combined query image adjustment unit 26 outputs combined query image information indicating the combined query image whose position and size have been adjusted to the search unit 27.
- the synthesized query image adjustment unit 26 may adjust the position and size of the object in the synthesized query image according to an instruction from the user input to the operation unit 14, or the synthesized query image according to a predetermined adjustment rule set in advance. You may adjust the position and size of the object.
- the adjustment rule is, for example, that a person object is placed in front of a building object when the object detection unit 23 specifies the name of the object included in the composite query image.
- the synthetic query image adjustment unit 26 detects the position and / or size of the object in the synthetic query image from the synthetic query image CQP1 shown in FIG. 2 to the synthetic query image shown in (a) or (b) of FIG. Adjust.
- the search unit 27 acquires the combined query image information from the combined query image adjustment unit 26, and selects the combined query image indicated by the acquired combined query image information from the images held in the image database 2 and / or the image storage unit 31. Search for similar images.
- the search unit 27 displays the search result on the display unit 16, and displays an image more similar to the synthetic query image at the top of the search result. If the index indicating the degree of similarity with the synthesized query image is the similarity, the search unit 27 displays an image with a higher similarity at a higher rank in the search result.
- the search unit 27 may determine the similarity of each search target image with the synthesized query image based on the position and / or size of the object in the synthesized query image. Specifically, when there are a plurality of search target images having the same object as the object included in the composite query image, the closer the position or size of the object in the search target image is to the position or size of the object in the composite query image The similarity may be increased.
- the search unit 27 may perform a search using only the synthetic query image selected (designated) by the user via the operation unit 14.
- the storage unit 12 stores programs and data referred to by the control unit 11, and includes an image storage unit 31, a template storage unit 32, an algorithm storage unit 33, a selection rule storage unit 34, an arrangement rule storage unit 35, and an adjustment.
- a rule storage unit 36 is provided.
- the image storage unit 31 stores an image designated by the user as a query image, a search target image to be searched by the search unit 27 based on the synthesized query image, and the like.
- the template storage unit 32 stores an image template and a feature amount template used when the object detection unit 23 detects an object.
- the algorithm storage unit 33 stores the above-described algorithm used when the region extraction unit 24 extracts a predetermined region.
- the selection rule storage unit 34 stores a selection rule used when the synthetic query image creation unit 25 automatically selects an object.
- the placement rule storage unit 35 stores a placement rule used when the synthetic query image creation unit 25 automatically determines the placement of an object.
- the adjustment rule storage unit 36 stores an adjustment rule used when the synthetic query image adjustment unit 26 automatically adjusts the position and size of an object.
- FIG. 4 is a flowchart illustrating an example of image search processing in the image processing apparatus 1.
- the processing example illustrated in FIG. 4 illustrates an example in which the synthetic query image creation unit 25 and the synthetic query image adjustment unit 26 create and adjust a synthetic query image based on a user instruction.
- the query image specifying unit 21 waits for an instruction to specify an image as a query image via the operation unit 14 (S1).
- the query image specifying unit 21 specifies an image specified by the user as a query image in accordance with the input instruction. (S2). Then, the query image specifying unit 21 outputs the specified query image to the object extracting unit 22.
- the object extracting unit 22 acquires a query image from the query image specifying unit 21, and extracts an object from the acquired query image (S3). Then, the object extraction unit 22 displays the extracted object on the display unit 16 (S4). Further, the object extraction unit 22 generates object information indicating the extracted object, and outputs the generated object information to the synthetic query image creation unit 25.
- the synthetic query image creation unit 25 waits for an instruction from the user to select an object to be incorporated into the synthetic query image from the objects displayed on the display unit 16 (S5).
- the process returns to S1 to return to the query image. Wait for it to be specified.
- the composite query image creation unit 25 creates a composite query image by pasting an object specified by the user on a blank image according to the input instruction. (S6). Then, the synthesized query image creation unit 25 displays the created synthesized query image on the display unit 16 (S7). Further, the synthesized query image creation unit 25 outputs synthesized query image information indicating the created synthesized query image to the synthesized query image adjustment unit 26.
- the synthetic query image adjustment unit 26 waits for an instruction to adjust the position or size of the object of the synthetic query image to be input (S8).
- the composite query image adjustment unit 26 combines the composite query image.
- the query image information is output to the search unit 27.
- the composite query image adjustment unit 26 determines the position or size of the object specified by the user in the composite query image according to the input instruction. Adjust (S9). Then, the synthesized query image adjustment unit 26 displays the adjusted synthesized query image on the display unit 16 (S10). Further, the synthesized query image adjustment unit 26 outputs synthesized query image information indicating the adjusted synthesized query image to the search unit 27.
- the search unit 27 obtains the composite query image information from the composite query image adjustment unit 26
- the search unit 27 waits for an input of a composite end instruction from the user via the operation unit 14 (S11).
- an instruction to add a query image is input instead of an instruction to end synthesis (NO in S11)
- the process returns to S1 and waits for a query image to be specified.
- the search unit 27 displays the composite query image indicated by the acquired composite query image information from the images held in the image database 2 and / or the image storage unit 31. An image similar to is searched (S12). Then, the search unit 27 displays the search result on the display unit 16 (S13).
- the synthetic query image creation unit 25 and the synthetic query image adjustment unit 26 perform automatically instead of the user's instruction, the determination process of S5 and S8 is not necessary. In this case, in S4, S7, and S10, the object or the synthesized query image may not be displayed on the display unit 16.
- FIGS. 5 and 6 are diagrams showing examples of display screens displayed on the display unit 16 in the image search process.
- the user selects an image acquired by the image input unit 13 from the image providing device 3 as a query image.
- the object extraction unit 22 extracts one of the most characteristic objects in each query image, and the synthetic query image creation unit 25 selects an object to be pasted based on a user instruction.
- the control unit 11 displays the input image P ⁇ b> 1 acquired by the image input unit 13 from the image providing device 3 on the display unit 16.
- the query image specifying unit 21 specifies the input image P1 as the query image QP1.
- the object extraction unit 22 extracts the object OBJ1 from the query image QP1, and displays the extracted object OBJ1 on the display unit 16, as shown in FIG.
- the synthesized query image creation unit 25 pastes the object OBJ1 on the blank image and composes it.
- a query image CQP1a is created.
- the composite query image creation unit 25 displays the created composite query image CQP1a on the display unit 16, as shown in FIG.
- the object OBJ1 included in the synthesized query image CQP1a is displayed as a line drawing, but the present invention is not limited to this.
- an image extracted as it is from the query image QP1 may be displayed.
- the control unit 11 displays information on the query image used to create the synthesized query image. Also good. In the example shown in FIG. 5D, the control unit 11 displays the thumbnail image of the query image QP1, the presence / absence of an object extracted from the query image QP1, and the presence / absence of face detection on the display unit 16.
- the image input unit 13 further acquires the input image P2 from the image providing device 3.
- the control unit 11 displays the input image P2 acquired by the image input unit 13 on the display unit 16 as illustrated in FIG.
- the query image specifying unit 21 specifies the input image P2 as the query image QP2.
- the object extraction unit 22 extracts the object OBJ2 from the query image QP2, and displays the extracted object OBJ2 on the display unit 16, as shown in FIG.
- the synthetic query image creation unit 25 adds the object OBJ2 to the synthetic query image CQP1a created earlier.
- the composite query image CQP1 is created by pasting.
- the composite query image creation unit 25 displays the created composite query image CQP1 on the display unit 16 as shown in FIG. At this time, when the user performs a screen display switching operation, as shown in FIG. 6D, the control unit 11 displays information related to the query image used to create the synthesized query image. Also good. In the example shown in (d) of FIG. 6, information regarding the query images QP1 and QP2 is displayed.
- FIG. 7 is a diagram illustrating another display screen example displayed on the display unit 16 in the image search process.
- the user selects images P1 and P2 acquired by the image input unit 13 from the image providing device 3 as query images.
- the object extracting unit 22 extracts a plurality of objects from each query image, and the synthesized query image creating unit 25 automatically selects an object and creates a synthesized query image.
- the object extraction unit 22 extracts objects from the query images QP1 and QP2, respectively, and extracts the objects OBJ1, OBJ3, and OBJ4 extracted from the query image QP1 and the query image QP2 as shown in FIG.
- the displayed objects OBJ2 and OBJ5 are displayed on the display unit 16.
- the composite query image creation unit 25 automatically selects an object from the objects OBJ1 to OBJ5, and combines the query image CQP1 including the objects OBJ1 and OBJ2, and the composite query image CQP2 including the objects OBJ3 and OBJ4. Create The synthesized query image creation unit 25 displays the created synthesized query images CQP1 and CQP2 on the display unit 16 as shown in FIG.
- an image processing apparatus includes a query image specifying unit that specifies a query image based on designation from a user, an object detection unit that detects an object from the query image, and the object detection By combining a plurality of the objects detected by the means, a combined query image generating means for generating a combined query image including a plurality of the objects, and searching for an image similar to the combined query image based on the combined query image And an image search means.
- an image processing method includes a query image specifying step that specifies a query image based on designation from a user, an object detection step that detects an object from the query image, and the object detection
- a composite query image generating step for generating a composite query image including a plurality of the objects by combining a plurality of the objects detected in the step, and an image similar to the composite query image based on the composite query image
- an image search step for searching.
- a user searches for an image similar to one or a plurality of query images including a plurality of objects
- the user generally searches for an image similar to an image including a plurality of objects included in these query images. It is expected that
- the object detecting means detects an object from the query image specified by the query image specifying means based on a designation from the user, and the synthesized query image generating means synthesizes a plurality of the objects detected by the object detecting means.
- a synthesized query image including a plurality of the objects is generated, and the image search means searches for an image similar to the synthesized query image based on the synthesized query image.
- the object can be automatically detected from the query image specified by the user. Therefore, by searching using a synthetic query image including an object detected from the query image, it is possible to efficiently search for an image that the user is looking for. That is, it is possible to reduce unnecessary search processing in the image processing apparatus.
- the search is performed in units of a combined query image including a plurality of objects instead of searching in units of objects, thereby reducing the processing load of the image processing apparatus. be able to.
- the image processing apparatus further includes a synthetic query image adjustment unit that adjusts at least one of the position and size of the object on the synthetic query image, and the image search unit includes the synthetic query image adjustment unit. It is preferable to search for an image similar to the adjusted composite query image.
- the user determines the position and / or size of an object included in each image by comparing each other.
- the synthesized query image adjusting unit adjusts at least one of the position and size of the object included in the synthesized query image generated by the synthesized query image generating unit on the synthesized query image.
- the image search means searches for an image similar to the adjusted composite query image by the composite query image adjustment means.
- the image processing apparatus further includes an input unit that receives an input from a user, and the synthetic query image generation unit detects the object detected by the object detection unit based on an instruction input to the input unit. It is preferable to select a plurality of objects and combine the selected objects.
- the synthesized query image generating means selects a plurality of objects detected by the object detecting means based on a user instruction input to the input means, and synthesizes the selected objects. . Therefore, the synthetic query image generation means can select an object similar to the object included in the image that the user is looking for. Therefore, it is possible to generate a composite query image including an object similar to an object included in the image that the user is looking for. Therefore, an image similar to the image that the user is looking for can be searched more effectively and efficiently.
- the image processing apparatus further includes an input unit that receives an input from a user, and the synthetic query image adjustment unit is configured to display the object on the synthetic query image based on an instruction input to the input unit. It is preferable to adjust at least one of position and size.
- the synthetic query image adjustment unit is configured to determine the position and size of the object on the synthetic query image generated by the synthetic query image generation unit based on a user instruction input to the input unit. Adjust at least one of them. Therefore, it is possible to generate a composite query image including an object included in the image that the user is looking for and an object that is similar in position and / or size on the image. Therefore, an image similar to the image that the user is looking for can be searched more effectively and efficiently.
- the image processing apparatus may be realized by a computer.
- a control program for causing the image processing apparatus to be realized by a computer by causing the computer to operate as each unit of the image processing apparatus, and A computer-readable recording medium on which it is recorded also falls within the scope of the present invention.
- control unit 11 may be configured by hardware logic, or may be realized by software using a CPU as follows.
- the image processing apparatus 1 includes a storage device (such as a CPU that executes instructions of a control program that realizes each function, a ROM that stores the program, a RAM that expands the program, a memory that stores the program and various data, and the like. Recording medium).
- An object of the present invention is a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the image processing apparatus 1 which is software that realizes the above-described functions is recorded so as to be readable by a computer. This can also be achieved by supplying the image processing apparatus 1 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).
- Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R.
- Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.
- the image processing apparatus 1 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network.
- the communication network is not particularly limited.
- the Internet intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available.
- the transmission medium constituting the communication network is not particularly limited.
- wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc.
- infrared rays such as IrDA and remote control, Bluetooth (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used.
- the present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.
- the present invention can be used for an image processing apparatus that searches for an image similar to a query image.
- Image Processing Device Operation Unit (Input Unit) 21 Query image specifying unit (query image specifying means) 22 Object extraction unit (object detection means) 25 synthetic query image creation unit (synthetic query image generation means) 26 Synthetic Query Image Adjustment Unit (Synthetic Query Image Adjustment Means) 27 Retrieval part (image retrieval means)
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Library & Information Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Processing Or Creating Images (AREA)
- Image Processing (AREA)
Abstract
Description
本発明に係る画像処理装置は、ユーザの指定した1または複数のクエリ画像に基づいて、クエリ画像に類似する画像を検索するための検索用画像情報(合成クエリ画像)を生成するものである。
図1は、画像処理装置1の要部構成の一例を示すブロック図である。図1に示すように、画像処理装置1は、制御部11、記憶部12、画像入力部13、操作部(入力手段)14、通信部15および表示部16を備えている。なお、画像処理装置1は、音声入力部、音声出力部等の部材を備えていてもよいが、発明の特徴点とは関係がないため当該部材を図示していない。
次に、画像処理装置1における画像検索処理について図4に基づいて説明する。図4は、画像処理装置1における画像検索処理の一例を示すフローチャートである。図4に示す処理例では、合成クエリ画像作成部25および合成クエリ画像調整部26がユーザの指示に基づいて、合成クエリ画像を作成および調整を行う例を示す。
次に、上記画像検索処理において表示部16に表示される表示画面例を図5および図6に基づいて説明する。図5および図6は、画像検索処理において表示部16に表示される表示画面例を示す図である。図5および図6に示す例では、画像入力部13が画像提供装置3から取得した画像をユーザがクエリ画像として選択するものとする。また、オブジェクト抽出部22が各クエリ画像における最も特徴的なオブジェクトを1つ抽出するものとし、また、合成クエリ画像作成部25がユーザの指示に基づいて貼り付けるオブジェクトを選択するものとする。
次に、上記画像検索処理において表示部16に表示される他の表示画面例を図7に基づいて説明する。図7は、画像検索処理において表示部16に表示される他の表示画面例を示す図である。図7に示す例では、画像入力部13が画像提供装置3から取得した画像P1およびP2をユーザがクエリ画像として選択するものとする。また、オブジェクト抽出部22が各クエリ画像から複数のオブジェクトを抽出するものとし、また、合成クエリ画像作成部25が自動的にオブジェクトを選択して合成クエリ画像を作成するものとする。
本発明に係る画像処理装置は、上記課題を解決するために、ユーザからの指定に基づきクエリ画像を特定するクエリ画像特定手段と、上記クエリ画像からオブジェクトを検出するオブジェクト検出手段と、上記オブジェクト検出手段の検出した上記オブジェクトを複数合成することで、複数の上記オブジェクトを含む合成クエリ画像を生成する合成クエリ画像生成手段と、上記合成クエリ画像に基づいて、上記合成クエリ画像に類似する画像を検索する画像検索手段と、を備えることを特徴としている。
本発明は上述した実施形態に限定されるものではなく、請求項に示した範囲で種々の変更が可能である。すなわち、請求項に示した範囲で適宜変更した技術的手段を組み合わせて得られる実施形態についても本発明の技術的範囲に含まれる。
14 操作部(入力手段)
21 クエリ画像特定部(クエリ画像特定手段)
22 オブジェクト抽出部(オブジェクト検出手段)
25 合成クエリ画像作成部(合成クエリ画像生成手段)
26 合成クエリ画像調整部(合成クエリ画像調整手段)
27 検索部(画像検索手段)
Claims (6)
- ユーザからの指定に基づきクエリ画像を特定するクエリ画像特定手段と、
上記クエリ画像からオブジェクトを検出するオブジェクト検出手段と、
上記オブジェクト検出手段の検出した上記オブジェクトを複数合成することで、複数の上記オブジェクトを含む合成クエリ画像を生成する合成クエリ画像生成手段と、
上記合成クエリ画像に基づいて、上記合成クエリ画像に類似する画像を検索する画像検索手段と、を備えることを特徴とする画像処理装置。 - 上記合成クエリ画像上における上記オブジェクトの位置およびサイズの少なくとも何れかを調整する合成クエリ画像調整手段をさらに備え、
上記画像検索手段は、上記合成クエリ画像調整手段が調整した合成クエリ画像に類似する画像を検索することを特徴とする請求項1に記載の画像処理装置。 - ユーザからの入力を受け付ける入力手段をさらに備え、
上記合成クエリ画像生成手段は、上記入力手段に入力された指示に基づいて、上記オブジェクト検出手段の検出した上記オブジェクトを複数選択し、選択した複数のオブジェクトを合成することを特徴とする請求項1または2に記載の画像処理装置。 - ユーザからの入力を受け付ける入力手段をさらに備え、
上記合成クエリ画像調整手段は、上記入力手段に入力された指示に基づいて、上記合成クエリ画像上における上記オブジェクトの位置およびサイズの少なくとも何れかを調整することを特徴とする請求項2に記載の画像処理装置。 - ユーザからの指定に基づきクエリ画像を特定するクエリ画像特定ステップと、
上記クエリ画像からオブジェクトを検出するオブジェクト検出ステップと、
上記オブジェクト検出ステップにおいて検出された上記オブジェクトを複数合成することで、複数の上記オブジェクトを含む合成クエリ画像を生成する合成クエリ画像生成ステップと、
上記合成クエリ画像に基づいて、上記合成クエリ画像に類似する画像を検索する画像検索ステップとを含むことを特徴とする画像処理方法。 - ユーザからの指定に基づきクエリ画像を特定するクエリ画像特定ステップと、
上記クエリ画像からオブジェクトを検出するオブジェクト検出ステップと、
上記オブジェクト検出ステップにおいて検出された上記オブジェクトを複数合成することで、複数の上記オブジェクトを含む合成クエリ画像を生成する合成クエリ画像生成ステップと、
上記合成クエリ画像に基づいて、上記合成クエリ画像に類似する画像を検索する画像検索ステップとを含む画像処理方法をコンピュータに実行させるための制御プログラム。
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP11861215.9A EP2685420A4 (en) | 2011-03-11 | 2011-03-22 | IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND CONTROL PROGRAM |
CN201180068427XA CN103415868A (zh) | 2011-03-11 | 2011-03-22 | 图像处理装置、图像处理方法以及控制程序 |
US13/985,144 US20140112598A1 (en) | 2011-03-11 | 2011-03-22 | Image processing device, image processing method and control program |
KR1020137020491A KR20130107366A (ko) | 2011-03-11 | 2011-03-22 | 화상 처리 장치, 화상 처리 방법 및 제어 프로그램 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2011-054649 | 2011-03-11 | ||
JP2011054649A JP2012190349A (ja) | 2011-03-11 | 2011-03-11 | 画像処理装置、画像処理方法および制御プログラム |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2012124149A1 true WO2012124149A1 (ja) | 2012-09-20 |
Family
ID=46830258
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2011/056827 WO2012124149A1 (ja) | 2011-03-11 | 2011-03-22 | 画像処理装置、画像処理方法および制御プログラム |
Country Status (6)
Country | Link |
---|---|
US (1) | US20140112598A1 (ja) |
EP (1) | EP2685420A4 (ja) |
JP (1) | JP2012190349A (ja) |
KR (1) | KR20130107366A (ja) |
CN (1) | CN103415868A (ja) |
WO (1) | WO2012124149A1 (ja) |
Families Citing this family (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5788605B2 (ja) * | 2012-09-13 | 2015-10-07 | 株式会社Nttドコモ | ユーザインタフェース装置、検索方法及びプログラム |
JP6419560B2 (ja) * | 2014-12-05 | 2018-11-07 | 株式会社東芝 | 検索装置、方法及びプログラム |
CN105306678A (zh) * | 2015-09-14 | 2016-02-03 | 联想(北京)有限公司 | 一种信息处理方法及电子设备 |
KR101886182B1 (ko) * | 2016-09-28 | 2018-08-08 | 서강대학교산학협력단 | 내용 기반 이미지 검색을 위한 질의 이미지 생성 방법 및 시스템 |
KR101941693B1 (ko) * | 2017-05-31 | 2019-01-24 | 서강대학교산학협력단 | 내용 기반 이미지 검색방법 및 그에 따른 시스템 |
CN110147459B (zh) * | 2017-07-28 | 2021-08-20 | 杭州海康威视数字技术股份有限公司 | 一种图像检索方法、装置及电子设备 |
US11163819B2 (en) * | 2017-10-23 | 2021-11-02 | Adobe Inc. | Image search and retrieval using object attributes |
CN110929057A (zh) * | 2018-08-30 | 2020-03-27 | 深圳市蓝灯鱼智能科技有限公司 | 图像处理方法、装置和系统、存储介质及电子装置 |
KR102646344B1 (ko) * | 2019-09-06 | 2024-03-12 | 삼성전자주식회사 | 이미지를 합성하기 위한 전자 장치 및 그의 동작 방법 |
JP7302741B2 (ja) * | 2020-05-22 | 2023-07-04 | 日本電気株式会社 | 画像選択装置、画像選択方法、およびプログラム |
CN113468353A (zh) * | 2021-07-20 | 2021-10-01 | 柒久园艺科技(北京)有限公司 | 一种基于图形的游客互动方法、装置、电子设备及介质 |
US20230161809A1 (en) * | 2021-11-22 | 2023-05-25 | Comcast Cable Communications, Llc | Methods and systems for modifying content searches |
KR102539376B1 (ko) * | 2022-11-07 | 2023-06-02 | 주식회사 드랩 | 자연어 기반 상품 사진 생성 방법, 서버 및 컴퓨터 프로그램 |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2806666B2 (ja) | 1991-12-13 | 1998-09-30 | 工業技術院長 | 概略画像作成方法及び装置 |
JP2001043246A (ja) * | 1999-06-09 | 2001-02-16 | Internatl Business Mach Corp <Ibm> | マルチメディア・データに対するユーザの認識を理解するための対話型フレームワーク |
JP2010218336A (ja) * | 2009-03-18 | 2010-09-30 | Fuji Xerox Co Ltd | 文書処理システム、検索装置およびプログラム |
JP2010250426A (ja) | 2009-04-13 | 2010-11-04 | Seiko Epson Corp | 画像処理装置および印刷装置 |
JP2010256994A (ja) * | 2009-04-21 | 2010-11-11 | Pfu Ltd | 文字認識装置、文字認識方法、および、プログラム |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
TW501035B (en) * | 2001-03-20 | 2002-09-01 | Ulead Systems Inc | Interactive image searching method based on local object |
EP1302865A1 (en) * | 2001-10-10 | 2003-04-16 | Mitsubishi Electric Information Technology Centre Europe B.V. | Method and apparatus for searching for and retrieving colour images |
JP4066162B2 (ja) * | 2002-09-27 | 2008-03-26 | 富士フイルム株式会社 | 画像編集装置、画像編集プログラム並びに画像編集方法 |
JP5009577B2 (ja) * | 2005-09-30 | 2012-08-22 | 富士フイルム株式会社 | 画像検索装置および方法並びにプログラム |
JP2007122540A (ja) * | 2005-10-31 | 2007-05-17 | Canon Inc | 画像情報処理方法、および画像情報処理装置 |
JP5127067B2 (ja) * | 2009-03-06 | 2013-01-23 | パナソニック株式会社 | 画像検索装置及び画像検索方法 |
US9195898B2 (en) * | 2009-04-14 | 2015-11-24 | Qualcomm Incorporated | Systems and methods for image recognition using mobile devices |
US8392430B2 (en) * | 2009-09-23 | 2013-03-05 | Microsoft Corp. | Concept-structured image search |
-
2011
- 2011-03-11 JP JP2011054649A patent/JP2012190349A/ja active Pending
- 2011-03-22 WO PCT/JP2011/056827 patent/WO2012124149A1/ja active Application Filing
- 2011-03-22 CN CN201180068427XA patent/CN103415868A/zh active Pending
- 2011-03-22 EP EP11861215.9A patent/EP2685420A4/en not_active Withdrawn
- 2011-03-22 US US13/985,144 patent/US20140112598A1/en not_active Abandoned
- 2011-03-22 KR KR1020137020491A patent/KR20130107366A/ko not_active Application Discontinuation
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2806666B2 (ja) | 1991-12-13 | 1998-09-30 | 工業技術院長 | 概略画像作成方法及び装置 |
JP2001043246A (ja) * | 1999-06-09 | 2001-02-16 | Internatl Business Mach Corp <Ibm> | マルチメディア・データに対するユーザの認識を理解するための対話型フレームワーク |
JP2010218336A (ja) * | 2009-03-18 | 2010-09-30 | Fuji Xerox Co Ltd | 文書処理システム、検索装置およびプログラム |
JP2010250426A (ja) | 2009-04-13 | 2010-11-04 | Seiko Epson Corp | 画像処理装置および印刷装置 |
JP2010256994A (ja) * | 2009-04-21 | 2010-11-11 | Pfu Ltd | 文字認識装置、文字認識方法、および、プログラム |
Non-Patent Citations (3)
Title |
---|
"Digital image processing", 2 March 2009, COMPUTER GRAPHIC ARTS SOCIETY, pages: 196 - 199 |
LAURENT ITTI: "A Model of Saliency-Based Visual Attention for Rapid Scene Analysis", IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, vol. 20, no. 11, November 1998 (1998-11-01) |
See also references of EP2685420A4 |
Also Published As
Publication number | Publication date |
---|---|
US20140112598A1 (en) | 2014-04-24 |
EP2685420A4 (en) | 2014-09-17 |
JP2012190349A (ja) | 2012-10-04 |
KR20130107366A (ko) | 2013-10-01 |
CN103415868A (zh) | 2013-11-27 |
EP2685420A1 (en) | 2014-01-15 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2012124149A1 (ja) | 画像処理装置、画像処理方法および制御プログラム | |
CN108537859B (zh) | 使用深度学习的图像蒙板 | |
US10573041B2 (en) | Rear image candidate determination device, rear image candidate determination method, and rear image candidate determination program | |
KR102193567B1 (ko) | 복수의 이미지를 디스플레이하는 전자 장치 및 이의 이미지 처리 방법 | |
US9582610B2 (en) | Visual post builder | |
CN111541907B (zh) | 物品显示方法、装置、设备及存储介质 | |
KR102285699B1 (ko) | 이미지를 디스플레이하는 사용자 단말기 및 이의 이미지 디스플레이 방법 | |
CN105321171B (zh) | 针对实况相机馈送的图像分割 | |
US20120127066A1 (en) | Content output device, content output method, program, program recording medium, and content output integrated circuit | |
US20220174237A1 (en) | Video special effect generation method and terminal | |
JPWO2007004489A1 (ja) | 画像処理装置及び画像処理方法 | |
US10210598B2 (en) | Electronic device for displaying a plurality of images and method for processing an image | |
JP2008234124A (ja) | 代表色抽出方法、および代表色抽出装置 | |
US10115431B2 (en) | Image processing device and image processing method | |
CN112714257B (zh) | 显示控制方法、装置、电子设备及介质 | |
WO2023103949A1 (zh) | 视频处理方法、装置、电子设备及介质 | |
JP2016085534A (ja) | 画像処理装置、画像処理装置の制御方法およびプログラム | |
CN111526427A (zh) | 视频生成方法、装置及电子设备 | |
US20150371411A1 (en) | Computerized systems and methods for rendering a user interface element | |
US20070256012A1 (en) | User interface for browsing content, method of providing the user interface, and content browsing apparatus | |
CN107025295A (zh) | 一种照片电影制作方法及移动终端 | |
CN113242464A (zh) | 视频编辑方法、装置 | |
CN112822394A (zh) | 显示控制方法、装置、电子设备及可读存储介质 | |
US20230036950A1 (en) | Image processing | |
US20110304779A1 (en) | Electronic Apparatus and Image Processing Method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 11861215 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 20137020491 Country of ref document: KR Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2011861215 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13985144 Country of ref document: US |