JP2006313497A - Apparatus and method for retrieving image - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image retrieval apparatus enabling display of a multiplicity of reduced images in the form of being visually classified, and easy associative image retrieval based on an object. <P>SOLUTION: The image retrieval apparatus includes a tag generation section 12 for extracting the object from image data, and generating an object information tag including an object property and object area information, so as to relate to the image data; an image storage section 13 for storing image data; an original image set section 14 for setting original image to be represented with mosaics; an original image tile-shaped area division section 15 for dividing the original image into tile-shaped segment areas, and deciding an object segment of each tile-shaped segment area based on the object information tag; a piece image retrieval section 16 for retrieving piece images which are to be replacement candidates for the image in the tile-shaped segment area; a mosaic image generation section 17 for generating a mosaic image by replacing the image in each tile-shaped segment area with a piece image having a fit condition from among the candidate piece images; and a mosaic image display section 18 for displaying the mosaic image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image search apparatus for searching for an image from a large number of images.

  Conventionally, for example, photographic images are classified into folders and stored in a storage device for each event (shooting date) or the like, and thumbnail images of the images are created and associated. When searching for images, in general, a folder is selected based on the folder name, and if there are thumbnail images of images included in the folder, they are displayed side by side on the screen. If there is no thumbnail image, create a thumbnail image and display it side by side on the screen.

  In Patent Document 1, a user inputs a keyword, sets one or more keywords for each thumbnail image, and searches for images across multiple folders using the keyword selected by the user as a search key. A technique is disclosed.

Further, in Patent Document 2, when a reduced image such as a thumbnail image is displayed on the screen, as shown in FIG. 21, a plurality of images are stored with the time associated with each image added, and displayed on the screen. A method is disclosed in which each image is displayed in the order and interval corresponding to the time by setting the time axis and arranging each image in the vicinity of the position of the time axis corresponding to the time added for each image.
JP-A-11-96194 JP 2004-328265 A

  However, in the method of classifying photo images into folders and displaying thumbnail images of images included in the selected folder, the subject contents of the images included in the folder name cannot be stored with the folder name when the number of folders increases. For example, if there are several folders of similar events every year, such as sports competitions, you can't recall the people and scenes from that folder name. The thumbnail image is displayed to confirm the subject content. In addition, since the number of thumbnail images that can be displayed on the screen at that time is limited, it is necessary to switch thumbnail images to be displayed by scrolling or the like in order to confirm all the images when the number of images in the folder is large.

  On the other hand, in the method of searching for an image by keyword, the task of assigning a keyword to each image is very burdensome, and the criteria for selecting the keyword to be given depending on the interest and mood at that time can change. There is sex. For this reason, there is a problem that the keyword attached to the image to be searched is unclear and the search result is not necessarily expected for the search keyword selected at the time of the search.

  In addition, when displaying reduced images such as thumbnail images on the screen, the time axis is set and each image is displayed in the order and interval corresponding to the time. Although it is effective at times, there is a problem that it is not effective when the user forgets the shooting time or when searching for an image regardless of the shooting time.

  An object of the present invention is to provide an image search apparatus that can display many reduced images in a visually classified form on the screen and can easily perform an associative image search based on a subject.

  In order to solve the above-described problem, an image search apparatus according to a first aspect of the present invention is an image search apparatus that performs an image search using a mosaic image, the image input unit that performs input processing on a plurality of image data, A tag generation unit that extracts a subject from the image data input by an image input unit, generates a subject information tag including subject attributes and subject region information, and associates the tag with the image data; and an image that stores the plurality of image data A storage unit, an original image setting unit for setting an original image that is an image to be expressed by a mosaic, and dividing the original image into tiles, and each tile shape based on the subject information tag of the original image An original image tile-shaped region dividing unit that determines a subject segment of the divided region, and a piece image that is a replacement candidate for the image of the tile-shaped divided region of the original image is stored in the image A piece image search unit that searches from the image, a mosaic image generation unit that generates a mosaic image by replacing the image of each tiled divided area of the original image with a candidate image that satisfies the conditions, and a generation A mosaic image display unit configured to display the mosaic image, wherein the piece image search unit searches for a piece image including a subject of the same type as the original image based on the subject information tag, and the mosaic image generation unit Assigns a piece image including the same type of subject based on the subject classification of each tiled divided region of the original image.

  As a result, a subject information tag of the image is generated at the time of image input, and a mosaic image is generated by using this to place a piece image including the same type of subject for each subject region of the original image. Many reduced images can be displayed.

  An image search device according to a second aspect of the present invention is an image search device that performs an image search using a mosaic image, and an image input unit that inputs a plurality of image data, and the image that is input by the image input unit An image storage unit that stores data, an original image setting unit that sets an original image that is an image to be represented by a mosaic, a subject is extracted from the image data stored in the image storage unit, and subject attributes and subject areas A tag generation unit that generates a subject information tag including information and associates it with the image data, and divides the original image into regions in a tile shape, and passes the image data of the original image to the tag generation unit to generate the tag An original image tile that obtains the subject information tag of the original image from a unit and determines a subject classification of each tiled divided region based on the obtained subject information tag An area dividing unit, a piece image searching unit that searches the image storage unit for a piece image that is a replacement candidate for an image of the tiled divided region of the original image, and an image of each tiled divided region of the original image, A mosaic image generation unit that generates a mosaic image by replacing the candidate piece image with a suitable one, and a mosaic image display unit that displays the generated mosaic image, the piece image search unit includes: The subject information tag of the original image is acquired from the tag generation unit by passing the image data of the original image to the tag generation unit, and a subject of the same type as the original image is acquired based on the acquired subject information tag. The mosaic image generation unit searches for a piece image including the same, and the mosaic image generation unit includes a piece image including the same type of subject based on the subject classification of each tiled divided region of the original image. Assign.

  Thereby, a subject information tag of the image is generated at the time of dividing the original image and at the time of piece image search, and a mosaic image is generated by arranging piece images including the same kind of subject for each subject area of the original image. Therefore, many reduced images can be displayed in a visually classified form.

  In the image search device of the third aspect of the present invention, the original image setting unit is selected by a user from an image stored in the image storage unit or a piece image constituting a mosaic image displayed by the mosaic image display unit This is an image search device that sets a set image as an original image.

  As a result, the user selects the original image from the piece images constituting the completed mosaic image and recursively generates the mosaic image, thereby facilitating associative image retrieval.

  In the image search device of the fourth aspect of the invention, the original image setting unit is automatically selected from the images stored in the image storage unit or the piece images constituting the mosaic image displayed by the mosaic image display unit. This is an image search device that sets an obtained image as an original image.

  As a result, the original image is automatically selected from the piece images constituting the completed mosaic image, and the mosaic image is recursively generated, thereby enabling a search by screen saver-like automatic image browsing.

  In the image search device of the fifth aspect of the present invention, the mosaic image generating unit holds piece image use history information, and a piece image constituting the mosaic image displayed by the mosaic image display unit is used as a new original image. When regenerating a mosaic image, refer to the use history information, and an image that preferentially uses a piece image that has not been used when generating the previous mosaic image or a piece image that has been used less frequently until then. A search device.

  This increases the possibility that a different image will be used as a piece image every time. Therefore, even if the number of images increases, it is possible to search for images in a wide range by recursively generating mosaic images.

  An image search method according to a sixth aspect of the present invention is an image search method for performing an image search using a mosaic image, wherein the image input step inputs a plurality of image data, and the image input by the image input step A tag generation step of extracting a subject from the data, generating a subject information tag including subject attributes and subject region information and associating it with the image data; an image storage step of storing the plurality of image data in an image storage unit; and a mosaic An original image setting step for setting an original image that is an image to be expressed by the step of dividing the original image into regions in tiles, and subject classification of each tiled divided region based on the subject information tag of the original image An original image tiled area dividing step for determining the image, and a replacement candidate image for the tiled divided area image of the original image. A mosaic image is generated by replacing the image of each tile-shaped divided area of the original image with a piece of the candidate piece image that satisfies a condition, and a piece image search step of searching the image storage unit from the image storage unit. A mosaic image generation step; and a mosaic image display step for displaying the generated mosaic image. In the piece image search step, a piece image including a subject of the same type as the original image is searched based on the subject information tag. In the mosaic image generating step, piece images including the same type of subject are assigned based on the subject classification of each tiled divided region of the original image.

  In the image search method of the seventh aspect of the present invention, the original image setting step is selected by the user from the image stored in the image storage unit or the piece image constituting the mosaic image displayed in the mosaic image display step. This is an image search method for setting an acquired image as an original image.

  According to the present invention, a subject information tag of an image is generated, and a mosaic image is generated by using the image information to place a piece image including the same type of subject for each subject area of the original image. Many reduced images can be displayed. In addition, associative image search is facilitated by the user selecting the original image from the piece images constituting the mosaic image and recursively generating the mosaic image. Further, by automatically selecting an original image from piece images constituting a mosaic image and recursively generating a mosaic image, a search by screen saver-like automatic image browsing becomes possible. At this time, by preferentially arranging images different from those at the previous generation as piece images, there is a high possibility that a different image is used as a piece image each time, and even if the number of images increases, the mosaic image is recursively. By performing the generation, the image can be searched in a wide range.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Configuration of Embodiment 1)
FIG. 1 is a configuration diagram of an image search apparatus 1 according to Embodiment 1 of the present invention.

  The image search device 1 includes an image input unit 11, a tag generation unit 12, an image storage unit 13, an original image setting unit 14, an original image tile area division unit 15, a piece image search unit 16, a mosaic image generation unit 17, and The mosaic image display unit 18 is configured.

  The image input unit 11 processes input of image data. Here, the size of the input image data is limited to VGA (640 × 480 pixels).

  The tag generation unit 12 extracts a subject from the image data input by the image input unit 11, generates a subject information tag including subject attributes and subject region information, and associates the subject information tag with the image data.

The image storage unit 13 stores image data and a subject information tag associated therewith.
The original image setting unit 14 sets an image to be expressed by a mosaic as an original image.

  The original image tile area dividing unit 15 divides the original image into tile areas and determines subject segment values of the tile segment areas based on the subject information tag.

  The piece image search unit 16 searches for a piece image as a replacement candidate for the image of each tiled divided region. At this time, the piece image search unit 16 limits the piece image to be selected to an image including the same type of subject as the original image based on the subject information tag.

  The mosaic image generation unit 17 generates a mosaic image by finding and replacing piece images that satisfy the conditions from the candidate piece images for the image of each tiled divided region of the original image. At this time, the mosaic image generation unit 17 assigns piece images including the same type of subject based on the subject classification value of each tiled divided region of the original image.

  The mosaic image display unit 18 displays the generated mosaic image, and includes a selection pointer for the user to select a piece image, and a selection confirmation button for the user to confirm the selection.

  Further, the original image setting unit 14 sets an image selected by the user as an original image from the images stored in the image storage unit 13 or the piece images constituting the mosaic image displayed by the mosaic image display unit 18. .

(Operation of Embodiment 1)
In the image search apparatus 1 configured as described above, each of the components 11 to 18 operates as follows.

  Upon receiving an image input request from the user, the image input unit 11 reads designated image data and passes it to the tag generation unit 12. The tag generation unit 12 receives the image data from the image input unit 11, detects one or a plurality of subject areas included in the image, generates a subject information tag for each, and associates the subject information tag with the image data. Details of the subject information tag will be described later. The image input unit 11 acquires the subject information tag generated by the tag generation unit 12 and stores it in the image storage unit 13 together with the image data.

  The subject information tag generated by the tag generation unit 12 includes an image file name, a subject attribute name, and a subject area map. Subject attribute names are people, animals (dogs, cats, etc.), plants (flowers, grass, trees, etc.), vehicles (cars, trains, ships, etc.), buildings, seas (including rivers and lakes), and backgrounds ( Mountain, sky, etc.). The subject area map is an image that specifies an area of an image in which the subject indicated by the subject attribute name exists in a bitmap format. Specifically, 1 bit is assigned to each pixel point of the image, and “1” is set to the bit corresponding to the pixel point where the subject exists. “0” is set to the bit corresponding to the pixel point where the subject does not exist. Since the image data is VGA size, the subject area map has a size of 640 × 480/8 bytes. FIG. 2 shows an image 101 which is an example of an image based on the image data input to the image input unit 11. FIG. 3 shows subject attribute names in the subject information tag associated with the image 101. FIG. 4 shows a subject area map in the subject information tag.

  When the subject region is extracted from the image, the tag generation unit 12 prepares a template for each subject attribute name, and uses a generally known method of sequentially matching a partial region of the image for each.

A processing flow of the image input unit 11 and the tag generation unit 12 is shown in FIG.
The image input unit 11 acquires the designated image data (S11), the tag generation unit 12 generates a subject information tag (S12), and associates the image data with the subject information tag. The image input unit 11 stores the associated image data and the subject information tag in the image storage unit 13. The image input unit 11 determines whether or not the processing in the image input unit 11 and the tag generation unit 12 should be terminated (S14). If it is determined that the processing should not be terminated (N in S14), the image input unit 11 returns to the step of acquiring image data (S11). On the other hand, when the image input unit 11 determines that the process should be terminated (Y in S14), the processes of the image input unit 11 and the tag generation unit 12 are terminated.

  Next, details of the process (S12 in FIG. 5) in which the tag generation unit 12 generates the subject information tag will be described with reference to FIG.

  The tag generation unit 12 receives an image from the image input unit 11, sets an initial subject template (S21), and matches the received image using the template (S22). The tag generator 12 determines whether or not a subject has been extracted (S23), and if it is determined that a subject has been extracted (Y in S23), the tag generator 12 generates a subject information tag (S24).

  When the subject information tag is generated (S24), and when it is determined that the subject has not been extracted (N in S23), the tag generation unit 12 includes all the subject templates and the image received from the image input unit 11. It is determined whether matching has been performed (S25). When matching is performed on all subject templates and images from the image input unit 11 (Y in S25), the tag generation unit 12 regards an area that does not match any template as a background area. Then, a subject information tag is generated (S26).

  On the other hand, when matching between all the subject templates and the image from the image input unit 11 is not performed (N in S25), the tag generation unit 12 sets the next template (S27), and the template is set. Then, the process returns to the step of matching with the image received from the image input unit 11 (S22).

  After a plurality of image data and subject information tags are stored in the image storage unit 13 in this way, mosaic image generation processing is performed. The processing flow is as follows.

  At the first activation, the original image setting unit 14 issues an image selection message and prompts the user to select one of the images stored in the image storage unit 13 (S31 in FIG. 7). . The original image setting unit 14 waits for any image to be selected by the user (S32), and when any image is selected by the user, sets the selected image as the original image (S33). In the first embodiment, it is assumed that the image selected by the user is the image 101. The original image setting unit 14 passes the set original image to the original image tile-shaped region dividing unit 15.

  The original image tile area dividing unit 15 divides the original image into tile areas as shown in FIG. 8 (S41 in FIG. 9). Here, the size of each divided area is 20 × 15 pixels. As a result, the original image is divided into 32 × 32 regions. Then, the original image tile area dividing unit 15 designates the first divided area (S42), and determines the subject segment value of the area based on the subject area map in each subject information tag of the original image (S43). ). Next, the original image tile area dividing unit 15 determines whether or not the subject segment values of all the divided areas have been determined (S44). If it is determined that the subject segment values of all the divided regions have been determined (Y in S44), the processing for determining the subject segment values of the original image tile-shaped region segmenting unit 15 ends. On the other hand, if it is determined that the subject segment values of all the segment areas have not been determined (N in S44), the original image tile area segmentation unit 15 designates an area where the subject segment values are to be determined next (S45). Then, the process returns to the step of determining the subject segment value of the area (S43).

  In this way, the original image tile area dividing unit 15 determines subject segment values of all the divided areas as shown in FIG. At this time, the original image tile-shaped region dividing unit 15 verifies the bit range corresponding to the divided region for each subject region map, and uses the subject attribute name having the highest occupation ratio in each divided region as the subject segment value of the divided region. Set. In the example shown in FIG. 10, all the divided areas constituting the image 101 are divided into “person division block”, “car division block”, and “background division block”. The original image tiled area dividing unit 15 passes the tiled divided area information of the original image and the subject segment value of each divided area to the piece image searching unit 16.

  The piece image search unit 16 searches the image storage unit 13 for an image (piece image) that is a replacement candidate based on the subject segment value of each divided region in the original image.

  Specifically, first, the piece image search unit 16 refers to the subject segment values of each divided region and generates a subject segment value list in which all subject segment values are collected (S51 in FIG. 11). Next, the piece image search unit 16 first designates an image to be examined for subject segment values included in the subject segment value list from the images stored in the image storage unit 13 (S52). The piece image search unit 16 reads all subject information tags of the designated image from the image storage unit 13 (S53), and the designated image has the same subject attribute name as the subject category value included in the subject category value list. It is checked whether or not it is attached (S54). If the specified image has the same subject attribute name as the subject segment value included in the subject segment value list (Y in S54), the piece image search unit 16 adds the image to the piece image group for the corresponding subject segment value. (S55).

  Then, the piece image search unit 16 determines whether or not the subject segment values included in the subject segment value list have been examined for all the images (S56). If it is determined that the subject segment values included in the subject segment value list have been investigated for all images (Y in S56), the piece image search unit 16 ends the process of examining the subject segment value list. On the other hand, when it is determined that the subject segment values included in the subject segment value list have not been investigated for all the images (N in S56), the piece image search unit 16 determines from the images stored in the image storage unit 13 that Next, an image to be examined for subject category values included in the subject category value list is designated (S57), and the process returns to the step of reading all subject information tags of the designated image from the image storage unit 13 (S53). If the designated image does not have the same subject attribute name as the subject category value included in the subject category value list (N in S54), the piece image search unit 16 adds all the images to the subject category value list. It is determined whether or not the included subject classification value has been investigated (S56).

  In this way, the piece image search unit 16 generates and holds piece image group information shown in FIG. 12 for all subject segment values of the divided areas in the original image.

  Subsequently, the mosaic image generation unit 17 receives the tiled divided area information of the original image and the piece image group information for the subject segment value of the divided area from the piece image search unit 16 and arranges the piece images in the divided areas. .

  Specifically, the mosaic image generating unit 17 designates the first divided area of the original image, reads the divided area image of the original image corresponding to the designation (S61 in FIG. 13), and the piece image for the subject segment value The first image is designated from the group (S62). Next, the mosaic image generation unit 17 reads the designated image, calculates an average luminance value for each of the divided region image and the designated image, and executes matching between the two (S63).

  When the divided region image does not match the designated image (N in S64), the mosaic image generating unit 17 determines whether or not matching processing has been performed on the divided region image and all the images in the piece image group. (S65). When the matching process is not performed on the divided region image and all the images in the piece image group (N in S65), the mosaic image generating unit 17 designates the next image from the piece image group (S66). Then, the process returns to the step (S63) in which matching is performed between the divided region image and the designated image.

  On the other hand, when the divided region image matches the designated image (Y in S64), and when the matching processing is performed on the divided region image and all the images in the piece image group (Y in S65), the mosaic is used. The image generation unit 17 resizes (reduces) the designated image and places it in the divided area (S67).

  Then, the mosaic image generating unit 17 determines whether or not the above-described processing has been performed for all the divided regions of the original image (S68). If it is determined that the processing has been performed (Y in S68), the mosaic image generating unit 17 The above process ends. On the other hand, when it is determined that the above-described processing has not been performed for all the divided areas of the original image (N in S68), the mosaic image generating unit 17 designates the next divided area, and the divided area corresponding to the designation. The image is read (S69), and the process returns to the step of designating the first image from the piece image group for the subject segment value (S62).

  As described above, the mosaic image generating unit 17 performs the matching process and the reduced image arrangement process for all the divided regions. The mosaic image generation unit 17 performs image matching using a known method. The mosaic image generation unit 17 obtains an average luminance value for each of the divided area image and the piece image of the original image using, for example, a method disclosed in Japanese Patent Application Laid-Open No. 11-345329, and determines whether the matching is successful or not based on the difference. When there is no piece image that meets the conditions, the image for which pass / fail is finally determined is placed.

  The mosaic image display unit 18 displays the mosaic image shown in FIG. 14 generated as described above. As shown in FIG. 15, the piece image (image 102) at the position where the user placed the selection pointer among the piece images constituting the mosaic image on the display screen is enlarged and displayed. Thereby, the user browses the image. When the user clicks on the enlarged piece image or when the user clicks the selection confirmation button while the piece image is enlarged, the original image setting unit 14 is activated, and the original image setting unit 14 A piece image is set as a new original image.

  The original image newly set by the original image setting unit 14 is transferred to the original image tile area dividing unit 15, and a new mosaic image shown in FIG. 16 is generated as described above. At this time, the mosaic image generation unit 17 holds the use history information such as the number of times of use so far and the use / non-use of the previous mosaic image generation for each piece image, and the piece image is generated in the second and subsequent generation of the mosaic image. Matching may be executed by preferentially selecting a piece image that has not been used for the previous mosaic image generation with reference to the use history information or a piece image that has been used less frequently. The format of piece image use history information is shown in FIG.

  In the first embodiment, the image data is limited to the VGA size, but the size is not necessarily limited. When a partial image (such as a divided image) of a photographic image is used as image data and the piece image (image 102) at the position where the selection pointer is placed is enlarged and displayed, the entire image can be displayed in addition to the partial image. You may do it.

  The subject area map is an image that specifies an area of the image where the subject indicated by the subject attribute name exists in the bitmap format. One bit is assigned to each pixel point of the image, and the pixel point where the subject exists is assigned. “1” is set in the corresponding bit, and “0” is set in the bit corresponding to the pixel point where the subject does not exist. However, instead of assigning 1 bit to each pixel point, a plurality of pixels may be used as one unit, and 1 bit may be assigned to that unit. For example, by setting 20 × 15 pixels as one unit and making it coincide with the size of one tile-shaped divided area, it is possible to directly determine the subject segment for each divided area from the subject area map. In this case, the size of the subject area map can be suppressed to 32 × 32/8 bytes.

  In the first embodiment, the original image setting unit 14 issues an image selection message at the first activation and prompts the user to select an image from the images stored in the image storage unit 13. Set as an image. However, the original image setting unit 14 may set, as the original image, an image that is automatically selected based on a criterion such as a shooting date and time.

  Further, when the user selects one of the piece images constituting the mosaic image displayed by the mosaic image display unit 18, the original image setting unit 14 is activated, and the original image setting unit 14 converts the selected piece image to a new piece image. Suppose you set it as the original image. However, when the mosaic image is displayed by the mosaic image display unit 18, one of the piece images constituting the displayed mosaic image is automatically selected at random or based on a criterion such as the shooting date and time, and the original image setting unit 14. May set the selected piece image as a new original image.

(Effect of Embodiment 1)
The image search apparatus 1 according to the first embodiment generates a subject information tag of an image at the time of image input, selects a piece image that includes the same type of subject for each subject region of the original image, and uses that as a mosaic image. Since it is generated, many reduced images can be displayed in a visually classified form. Further, when the user selects an original image from piece images constituting a mosaic image, the image search device 1 recursively generates a mosaic image based on the selected original image. This facilitates associative image retrieval. At this time, by preferentially selecting an image different from the previous generation as a piece image, there is a high possibility that a different image is used as a piece image each time. Therefore, even if the number of images increases, it is possible to search for images in a wide range by recursively generating mosaic images.

  Further, by automatically selecting an original image from piece images constituting a mosaic image and recursively generating a mosaic image, a search by screen saver-like automatic browsing becomes possible.

(Configuration of Embodiment 2)
FIG. 18 is a configuration diagram of the image search device 2 according to Embodiment 2 of the present invention.

  The image search device 2 includes an image input unit 21, a tag generation unit 22, an image storage unit 23, an original image setting unit 24, an original image tile-shaped region division unit 25, a piece image search unit 26, a mosaic image generation unit 27, and The mosaic image display unit 28 is configured.

  The image input unit 21 processes input of image data. Here, the size of the input image data is limited to VGA (640 × 480 pixels).

The image storage unit 23 stores the image data input by the image input unit 21.
The original image setting unit 24 sets an image to be expressed by a mosaic as an original image.

  The tag generation unit 22 extracts a subject from the image data stored in the image storage unit 23, generates a subject information tag including subject attributes and subject area information, and associates the subject information tag with the image data.

  The original image tile area dividing unit 25 divides the original image into tile areas, and also acquires the subject information tag of the original image by passing the original image data to the tag generation unit 22, and based on the acquired subject information tag. The subject segment value of each tiled divided area is determined.

  The piece image search unit 26 searches the image storage unit 23 for piece images that are candidates for replacement with respect to the images in each tiled divided region. At this time, the piece image search unit 26 acquires the subject information tag of the image data by passing the image data to the tag generation unit 22 and searches for the piece image based on the acquired subject information tag. Further, the piece image search unit 26 limits the piece image to be selected to an image including the same kind of subject as the original image.

  The mosaic image generation unit 27 generates a mosaic image by finding and replacing the piece images that satisfy the conditions from the candidate piece images for the image of each tiled divided region. At this time, the mosaic image generating unit 27 assigns piece images including the same type of subject based on the subject segment value of each tiled divided region.

  The mosaic image display unit 28 displays the generated mosaic image, and includes a selection pointer for the user to select a piece image, and a selection confirmation button for the user to confirm the selection.

  Further, the original image setting unit 24 sets an image selected by the user as an original image from the images stored in the image storage unit 23 or the piece images constituting the mosaic image displayed by the mosaic image display unit 28. .

(Operation of Embodiment 2)
In the image search apparatus 2 configured as described above, the respective components 21 to 28 operate as follows.

  In response to an image input request from the user, the image input unit 21 reads designated image data and stores it in the image storage unit 23.

  After a plurality of pieces of image data are stored in the image storage unit 23, a mosaic image generation process is performed. The processing flow is as follows.

  In the first activation, the original image setting unit 24 issues an image selection message (S31 in FIG. 7) and prompts the user to select one of the images stored in the image storage unit 23. . When any image is selected by the user, the selected image is set as an original image (S33 in FIG. 7). The original image setting unit 24 passes the set original image to the original image tile area dividing unit 25.

  As shown in FIG. 8, the original image tile area dividing unit 25 divides the original image into tile areas (S71 in FIG. 19). Here, the size of each divided area is 20 × 15 pixels. As a result, the original image is divided into 32 × 32 regions. Next, the original image tile area dividing unit 25 determines whether or not there is a subject information tag associated with the original image data (S72). When the subject information tag associated with the original image data does not exist (when the first mosaic image is generated, etc .: Y in S72), the original image tile area dividing unit 25 passes the original image data to the tag generating unit 22.

  The tag generation unit 22 detects one or a plurality of subject areas included in the original image data, generates subject information tags for each of them (S73), and passes them to the original image tile area division unit 25.

  The subject information tag includes an image file name, a subject attribute name, and a subject area map. Subject attribute names are people, animals (dogs, cats, etc.), plants (flowers, grass, trees, etc.), vehicles (cars, trains, ships, etc.), buildings, seas (including rivers and lakes), and backgrounds ( Mountain, sky, etc.). The subject area map is an image that specifies an area of an image in which the subject indicated by the subject attribute name exists in a bitmap format. Specifically, 1 bit is assigned to each pixel point of the image, and “1” is set to the bit corresponding to the pixel point where the subject exists. “0” is set to the bit corresponding to the pixel point where the subject does not exist. Since the image data is VGA size, the subject area map has a size of 640 × 480/8 bytes. FIG. 2 shows an image 101 which is an example of the original image. 3 and 4 show subject information tags associated with the image 101. FIG.

  When the subject region is extracted from the image, the tag generation unit 22 prepares a template for each subject attribute name, and uses a generally known technique that sequentially matches a partial region of the image. The tag generation unit 22 holds the generated subject information tag in association with image data. Note that the process in which the tag generation unit 22 generates the subject information tag has been described with reference to FIG. 6 in the first embodiment, and thus the description thereof is omitted here.

  When there is a subject information tag associated with the original image data (N in S72 of FIG. 19), and when a subject information tag is generated by the tag generation unit 22 (S73), the original image tiled region dividing unit 25 Acquires the subject information tag of the original image from the tag generator 22, designates the first divided region (S74), refers to the subject region map in each subject information tag, and determines the subject segment value of that region. (S75). Next, the original image tile area dividing unit 25 determines whether or not the subject segment values of all the divided areas have been determined (S76). When it is determined that the subject segment values of all the divided areas have been determined (Y in S76), the process of determining the subject segment values of the original image tile area segmentation unit 25 ends. On the other hand, when it is determined that the subject segment values of all the segment areas have not been determined (N in S76), the original image tile area segmentation unit 25 designates an area where the subject segment values are to be determined next (S77). ), The process returns to the step of determining the subject segment value of the area (S75).

  In this way, the original image tile area dividing unit 25 determines subject segment values for all the divided areas as shown in FIG. At this time, the original image tile-shaped region dividing unit 25 verifies the bit range corresponding to the divided region for each subject region map, and uses the subject attribute name having the highest occupancy rate in each divided region as the subject division value of the divided region. Set. In the example shown in FIG. 10, all the divided areas constituting the image 101 are divided into “person division block”, “car division block”, and “background division block”. The original image tiled area dividing unit 25 passes the tiled divided area information of the original image and the subject segment value of each divided area to the piece image searching unit 26.

  The piece image search unit 26 searches the image storage unit 23 for an image (piece image) that is a replacement candidate based on the subject segment value of each divided region in the original image.

  Specifically, first, the piece image search unit 26 refers to the subject segment values of each divided region and generates a subject segment value list in which all subject segment values are collected (S81 in FIG. 20). Next, the piece image search unit 26 first designates an image to be examined for subject segment values included in the subject segment value list from the images stored in the image storage unit 23 (S82), and the subject of the image It is determined whether an information tag exists (S83). When there is no subject information tag (when the first mosaic image is generated or when there is an added image, etc .: Y in S83), the piece image search unit 26 passes the image data to the tag generation unit 22, and the tag generation unit 22 A subject information tag is generated (S84). The processing performed by the tag generation unit 22 is the same as the processing in step 73 in FIG. Further, since the processing has been described with reference to FIG. 6 in Embodiment 1, the description thereof is omitted here.

  When the subject information tag exists (N in S83) and when the subject information tag is generated (S84), the piece image search unit 26 acquires the subject information tag of the specified image from the image storage unit 23 ( In S83, it is checked whether or not the designated image has the same subject attribute name as the subject segment value included in the subject segment value list (S85). If the specified image has the same subject attribute name as the subject segment value included in the subject segment value list (Y in S85), the piece image search unit 26 adds the image to the piece image group for the corresponding subject segment value. (S86).

  Then, the piece image search unit 26 determines whether or not the subject category values included in the subject category value list have been examined for all images (S87). If it is determined that the subject category values included in the subject category value list have been investigated for all images (Y in S87), the piece image search unit 26 ends the process of examining the subject category value list. On the other hand, if it is determined that the subject segment values included in the subject segment value list have not been investigated for all images (N in S87), the piece image search unit 26 determines whether the images stored in the image storage unit 23 are Next, an image to be examined for subject category values included in the subject category value list is designated (S88), and the process returns to the step of determining whether or not the subject information tag of the designated image exists (S83). If the designated image does not have the same subject attribute name as the subject category value included in the subject category value list (N in S85), the piece image search unit 26 adds all the images to the subject category value list. It is determined whether or not the included subject classification value has been investigated (S87).

  In this way, the piece image search unit 26 generates and holds piece image group information shown in FIG. 12 for all subject segment values of the divided areas in the original image.

  Subsequently, the mosaic image generating unit 27 receives the tiled divided area information of the original image and the piece image group information for the subject segment value of the divided area from the piece image search unit 26, and arranges the piece images in the divided areas. .

  Specifically, the mosaic image generating unit 27 designates the first divided area of the original image, reads the divided area image of the original image corresponding to the designation (S61 in FIG. 13), and the piece image corresponding to the subject segment value The first image is designated from the group (S62). Next, the mosaic image generating unit 27 reads the designated image, calculates an average luminance value for each of the divided region image and the designated image, and executes matching between them (S63).

  When the divided area image does not match the designated image (N in S64), the mosaic image generating unit 27 determines whether or not the matching process has been performed on the divided area image and all the images in the piece image group. (S65). When the matching process is not performed on the divided region image and all the images in the piece image group (N in S65), the mosaic image generating unit 27 designates the next image from the piece image group (S66). Then, the process returns to the step (S63) in which matching is performed between the divided region image and the designated image.

  On the other hand, when the divided region image matches the designated image (Y in S64), and when the matching processing is performed on the divided region image and all the images in the piece image group (Y in S65), the mosaic is used. The image generation unit 27 resizes (reduces) the designated image and places it in the divided area (S67).

  Then, the mosaic image generating unit 27 determines whether or not the above-described processing has been performed for all the divided regions of the original image (S68). If it is determined that the processing has been performed (Y in S68), the mosaic image generating unit 27 The above process ends. On the other hand, when it is determined that the above-described processing has not been performed for all the divided areas of the original image (N in S68), the mosaic image generating unit 27 designates the next divided area, and the divided area corresponding to the designation. The image is read (S69), and the process returns to the step of designating the first image from the piece image group for the subject segment value (S62).

  As described above, the mosaic image generation unit 27 performs the matching process and the reduced image arrangement process for all the divided regions. The mosaic image generating unit 27 performs image matching using a known method. The mosaic image generating unit 27 obtains an average luminance value for each of the divided area image and the piece image of the original image, for example, using a method disclosed in Japanese Patent Application Laid-Open No. 11-345329, and determines whether the matching is successful or not based on the difference. When there is no piece image that meets the conditions, the image for which pass / fail is finally determined is placed.

  The mosaic image display unit 28 displays the mosaic image shown in FIG. 14 generated as described above. As shown in FIG. 15, the piece image (image 102) at the position where the user placed the selection pointer among the piece images constituting the mosaic image on the display screen is enlarged and displayed. Thereby, the user browses the image. When the user clicks on the enlarged piece image or when the user clicks the selection confirmation button while the piece image is enlarged, the original image setting unit 24 is activated, and the original image setting unit 24 A piece image is set as a new original image.

  The original image newly set by the original image setting unit 24 is transferred to the original image tile-shaped region dividing unit 25, and a new mosaic image shown in FIG. 16 is generated as described above. At this time, the mosaic image generation unit 27 holds the usage history information such as the number of times of use so far and the use / non-use of the previous mosaic image generation for each piece image, and the piece image is generated in the second and subsequent generation of the mosaic image. Matching may be executed by preferentially selecting a piece image that has not been used for the previous mosaic image generation with reference to the use history information or a piece image that has been used less frequently. The format of piece image use history information is shown in FIG.

  In the second embodiment, the image data is limited to the VGA size, but the size is not necessarily limited. When a partial image (such as a divided image) of a photographic image is used as image data and the piece image (image 102) at the position where the selection pointer is placed is enlarged and displayed, the entire image can be displayed in addition to the partial image. You may do it.

  The subject area map is an image that specifies the area of the image where the subject indicated by the subject attribute name exists in the bitmap format. One bit is assigned to each pixel point of the image, and the pixel point where the subject exists is “1” is set in the corresponding bit, and “0” is set in the bit corresponding to the pixel point where the subject does not exist. However, instead of assigning 1 bit to each pixel point, a plurality of pixels may be used as one unit, and 1 bit may be assigned to that unit. For example, by setting 20 × 15 pixels as one unit and making it coincide with the size of one tile-shaped divided area, it is possible to directly determine the subject segment for each divided area from the subject area map. In this case, the size of the subject area map can be suppressed to 32 × 32/8 bytes.

  In the second embodiment, the original image setting unit 24 issues an image selection message at the first activation, prompts the user to select an image from the images stored in the image storage unit 23, and selects the selected image as the original image. Set as an image. However, the original image setting unit 24 may set, as the original image, an image that is automatically selected based on a criterion such as a shooting date and time.

  Further, when the user selects one of the piece images constituting the mosaic image displayed by the mosaic image display unit 28, the original image setting unit 24 is activated, and the original image setting unit 24 adds the selected piece image to a new piece image. Suppose you set it as the original image. However, when a mosaic image is displayed by the mosaic image display unit 28, one of the piece images constituting the displayed mosaic image is automatically selected at random or based on a criterion such as the shooting date and time, and the original image setting unit 24 May set the selected piece image as a new original image.

(Effect of Embodiment 2)
According to the image search device 2 of the second embodiment, the same effects as those of the first embodiment can be obtained.

  That is, the image search device 2 generates a subject information tag of an image when dividing an original image and searching for a piece image, and uses this to select a piece image that includes the same type of subject for each subject area of the original image. Since a mosaic image is generated, many reduced images can be displayed in a visually classified form. Further, when the user selects an original image from piece images constituting a mosaic image, the image search device 2 recursively generates a mosaic image based on the selected original image. This facilitates associative image retrieval. At this time, by preferentially selecting an image different from the previous generation as a piece image, there is a high possibility that a different image is used as a piece image each time. Therefore, even if the number of images increases, it is possible to search for images in a wide range by recursively generating mosaic images.

  Further, by automatically selecting an original image from piece images constituting a mosaic image and recursively generating a mosaic image, a search by screen saver-like automatic browsing becomes possible.

  Since the image search apparatus according to the present invention can be realized only by two-dimensional image processing, it can be used not only in a PC but also in AV equipment (including mobile equipment) that handles photographic images.

Configuration diagram of image search apparatus according to Embodiment 1 The figure which shows an example of the image in a subject information tag The figure which shows an example of the subject attribute name in a subject information tag The figure which shows an example of the subject area map in a subject information tag Processing flow diagram for image input and tag generation Detailed tag generation process flow diagram Original image setting process flow chart Diagram showing tiled area division of original image Flow chart of subject classification decision processing Diagram showing subject classification of original image Piece image group information generation processing flow chart Diagram showing piece image group information Mosaic image generation process flow chart A diagram representing a mosaic image Diagram showing enlarged display of piece image A diagram representing a mosaic image A diagram showing piece image usage history information Configuration diagram of image search apparatus according to Embodiment 2 Flow chart of subject classification decision processing Piece image group information generation processing flow chart A figure showing a conventional reduced image screen display

Explanation of symbols

1, 2 Image search device 11, 21 Image input unit 12, 22 Tag generation unit 13, 23 Image storage unit 14, 24 Original image setting unit 15, 25 Original image tile area division unit 16, 26 Piece image search unit 17, 27 Mosaic image generation unit 18, 28 Mosaic image display unit 101, 102 Image data

Claims (7)

An image search device that performs an image search using a mosaic image,
An image input unit for processing a plurality of image data;
A tag generation unit that extracts a subject from the image data input by the image input unit, generates a subject information tag including subject attributes and subject region information, and associates the subject information tag with the image data;
An image storage unit for storing the plurality of image data;
An original image setting unit for setting an original image that is an image to be expressed by a mosaic;
An original image tiled region dividing unit that divides the original image into tiles and determines subject classification of each tiled divided region based on the subject information tag of the original image;
A piece image search unit that searches the image storage unit for a piece image that is a replacement candidate for the image of the tiled divided area of the original image;
A mosaic image generating unit that generates a mosaic image by replacing the image of each tiled divided region of the original image with a candidate image that satisfies the conditions;
A mosaic image display unit for displaying the generated mosaic image,
The piece image search unit searches for a piece image including a subject of the same type as the original image based on the subject information tag,
The mosaic image generating unit assigns piece images including the same type of subject based on the subject classification of each tiled divided region of the original image. An image search device that performs an image search using a mosaic image,
An image input unit for processing a plurality of image data;
An image storage unit for storing the image data input by the image input unit;
An original image setting unit for setting an original image that is an image to be expressed by a mosaic;
A tag generation unit that extracts a subject from the image data stored in the image storage unit, generates a subject information tag including subject attributes and subject region information, and associates the subject information tag with the image data;
The original image is divided into tiles, and the subject information tag of the original image is acquired from the tag generator by passing image data of the original image to the tag generator, and the acquired subject information Based on the tag, the original image tiled area dividing unit that determines the subject classification of each tiled divided area;
A piece image search unit that searches the image storage unit for a piece image that is a replacement candidate for the image of the tiled divided area of the original image;
A mosaic image generating unit that generates a mosaic image by replacing the image of each tiled divided region of the original image with a candidate image that satisfies the conditions;
A mosaic image display unit for displaying the generated mosaic image,
The piece image search unit acquires the subject information tag of the original image from the tag generation unit by passing image data of the original image to the tag generation unit, and based on the acquired subject information tag, Search for a piece image that contains the same type of subject as the original image,
The mosaic image generating unit assigns piece images including the same type of subject based on the subject classification of each tiled divided region of the original image. The original image setting unit sets, as an original image, an image selected by a user from an image stored in the image storage unit or a piece image constituting a mosaic image displayed by the mosaic image display unit. 3. The image search device according to 1 or 2. The original image setting unit sets an automatically selected image as an original image from the images stored in the image storage unit or the piece images constituting the mosaic image displayed by the mosaic image display unit. Or the image search apparatus of 2 description. The mosaic image generation unit
Holds piece image usage history information,
When re-generating a new mosaic image using a piece image constituting the mosaic image displayed by the mosaic image display unit as an original image, it is used when generating a previous mosaic image with reference to the use history information. The image search device according to claim 3 or 4, wherein a piece image that has not been used or a piece image that has been used less frequently is preferentially used. An image search method for performing an image search using a mosaic image,
An image input step for input processing of a plurality of image data;
A tag generation step of extracting a subject from the image data input by the image input step, generating a subject information tag including subject attributes and subject region information, and associating the subject information tag with the image data;
An image storage step of storing the plurality of image data in an image storage unit;
An original image setting step for setting an original image which is an image to be expressed by a mosaic;
An original image tiled region dividing step for dividing the original image into tiles and determining subject classification of each tiled divided region based on the subject information tag of the original image;
A piece image search step of searching the image storage unit for a piece image that is a candidate for replacement with respect to the image of the tiled divided area of the original image;
A mosaic image generating step of generating a mosaic image by replacing the image of each tiled divided region of the original image with a piece image that satisfies the conditions from the candidate piece images;
A mosaic image display step for displaying the generated mosaic image,
In the piece image search step, based on the subject information tag, a piece image including the same type of subject as the original image is searched,
In the mosaic image generating step, an image search method for allocating piece images including the same type of subject based on the subject classification of each tiled divided region of the original image. The original image setting step sets an image selected by the user as an original image from the images stored in the image storage unit or the piece images constituting the mosaic image displayed in the mosaic image display step. The image search method described.

Images