JP2013061974A - Image processing apparatus and method, information processing apparatus and method, program, and image processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily search a desired image in a device with relatively small processing capability.SOLUTION: An image analysis part analyzes an image by applying image processing to the image. A similar feature database stores data showing features of the image obtained as a result of the image processing of the image in the image analysis part. A transmission control part controls transmission of features to be recorded in a digital still camera for recording information related to the image as data of the same structure as structure of the similar feature database to the digital still camera. The present invention can be applied to a hard disk recorder or a game device with a CPU and a hard disk to which a personal computer and the digital still camera can be connected.

Description

  The present disclosure relates to an image processing device and method, an information processing device and method, a program, and an image processing system, and in particular, an image processing device and method, an information processing device and method, a program, and a program that can extract image features. The present invention relates to an image processing system.

  Patent Document 1 proposes various functions for detecting a face, extracting image features, and searching for an image in a small CE (consumer electronic) device such as a digital still camera.

JP 2004-62868 A

  However, in a small CE device, since the capability of the installed processor is limited, the image can actually be analyzed only in a limited range. Therefore, sufficient analysis cannot be performed, thereby limiting the use of the analysis result and its accuracy.

  In face detection, unless the resolution of an image that can be used for analysis is extremely low, analysis requires a very long time, and the processing time greatly exceeds the time that the user can wait for. If the resolution of the image is extremely low, it is impossible to detect a small face, particularly a face in a group photo, and it is not possible to meet the needs such as searching for a group photo.

  In addition, if such processing is performed on a digital still camera or the like, the processing is concentrated on the digital still camera, so the power consumed by the processor of the digital still camera is increased, which is the original purpose of the digital still camera. There are also disadvantages such as the time that can be taken is shortened and the number of images that can be taken is reduced.

  On the other hand, with the spread of digital still cameras and the mounting of still camera functions in mobile phones, opportunities for taking photographs (still images) are steadily increasing in daily life. However, the method of searching for an image when the photographed image is to be browsed on the digital still camera body is only to display and browse reduced images (so-called thumbnail images) in the order of shooting. The convenience of searching for a digital still camera is far inferior to the convenience of an image management program executed on a personal computer or the like.

  For this reason, a digital still camera having a large-capacity storage and a photo album function is required to have a function of easily searching for an image that the user wants to see.

  The present invention has been made in view of such a situation, and makes it possible to easily search for a desired image in a device having a relatively small processing capability.

  An image processing apparatus according to a first aspect of the present disclosure includes a feature extraction unit that extracts a feature of an image, an association unit that relates the feature extracted from the image by the feature extraction unit to the image, and a device. In order to transmit the feature of the image, and to record the feature of the image transmitted by the device by the device, search for an image related to the feature, and present the searched image. Transmission control means for controlling transmission of the image features to the device by the transmission means.

  The image processing apparatus may further include receiving means for receiving the image transmitted from the device.

  The feature is information about a face.

  The feature is information about color.

  In the image processing method according to the first aspect of the present disclosure, the image processing apparatus extracts image features, relates the features extracted from the images to the images, and transmits the image features to a device. The device records the characteristics of the transmitted image, retrieves an image associated with the feature, and controls transmission of the image feature to the device to present the retrieved image.

  An information processing apparatus according to a second aspect of the present disclosure includes: a receiving unit that receives a feature extracted from an image by a server and related to the image; and a recording unit that records the feature received by the receiving unit Search means for searching for an image related to the feature, and presentation means for presenting the image searched for by the search means.

  An image capturing unit that captures an image can be further provided.

  The feature is information about a face.

  The feature is information about color.

  In the information processing method according to the second aspect of the present disclosure, an information processing apparatus receives a feature extracted from an image by a server, records the received feature, and searches for an image associated with the feature. Then, the retrieved image is presented.

  A program according to a second aspect of the present disclosure includes: a receiving unit that receives a feature extracted from an image and related to the image by a server; and a recording unit that records the feature received by the receiving unit; The computer functions as search means for searching for an image related to the feature and presentation control means for controlling the presentation of the image searched by the search means.

  An image processing system according to a third aspect of the present disclosure includes a server and an image processing device, and the server includes feature extraction means for extracting image features, and association means for relating the features to the image. And transmitting means for transmitting the characteristics to the image processing apparatus, wherein the image processing apparatus receives the characteristics, recording means for recording the characteristics received by the receiving means, and the characteristics Search means for searching for an image related to, and presentation means for presenting the image searched by the search means.

  In a first aspect of the present disclosure, image features are extracted, features extracted from the image are related to the image, and the image features are transmitted to a device. In doing so, the device records the features of the transmitted image, retrieves the image associated with the feature, and presents the retrieved image feature to the device to present the retrieved image. Be controlled.

  In the second aspect of the present disclosure, a feature extracted from an image is received by a server, and the received feature is recorded. Then, an image related to the feature is searched, and the searched image is presented.

  In the third aspect of the present disclosure, the server extracts features of the image, the features extracted from the image are related to the image, and the features of the image are transmitted to a device. In doing so, the device records the features of the transmitted image, retrieves the image associated with the feature, and presents the retrieved image feature to the device to present the retrieved image. Be controlled. Further, the feature extracted by the server from the image is received by the image processing apparatus, and the received feature is recorded. Then, an image related to the feature is searched, and the searched image is presented.

  According to the present disclosure, it is possible to search for an image in a device. In particular, it is possible to easily search for a desired image in a device having a relatively small processing capability.

It is a figure which shows the structure of the image processing system of one embodiment of this invention. It is a block diagram which shows the example of a structure of a digital still camera. It is a block diagram which shows the example of a structure of a server. It is a figure which shows the structure of the function implement | achieved by MPU which performs a program. It is a figure which shows the structure of the function implement | achieved by CPU which performs a program. It is a block diagram which shows the example of a structure of an image analysis part. It is a flowchart explaining the process of imaging | photography. It is a figure which shows correlation with a main image and a reduced image. It is a flowchart explaining the process of backup. It is a flowchart explaining the process of image analysis. It is a figure explaining the production | generation of a color histogram. It is a figure explaining the production | generation of a vertical component histogram and a horizontal component histogram. It is a figure explaining the production | generation of a vertical component histogram and a horizontal component histogram. It is a figure explaining the backup of an image, and the writing-back of metadata. It is a figure which shows the specific example of metadata. It is a figure which shows the structure of the metadata stored in a content database or a content database. It is a figure which shows the structure of the metadata stored in the metadata stored in the content database, and the similar feature database. It is a figure which shows the structure of a similar feature item. It is a flowchart explaining the process of acquisition of an image. It is a figure explaining acquisition of an image and writing of metadata. It is a flowchart explaining the process of a search. It is a figure explaining the correlation with the metadata and image which are common in a digital still camera and a server. It is a flowchart explaining the process of a search. It is a figure which shows the example of a display of a reduction image. It is a figure which shows the example of a display of a reduction image. It is a flowchart explaining the search process of a similar image. It is a figure which shows the structure of metadata and distance. It is a figure which shows the correlation of each record of a content database, a similar result database, and a time group database. It is a figure which shows the example of the display of a similar order. It is a figure explaining the switch of the display of a similar order, and the display of a time series. It is a flowchart explaining the process of a search. It is a figure explaining the switch of the display of a similar order, and the display of a time series. It is a block diagram which shows the example of a structure of a color feature extraction part. It is a figure which shows the example of the correspondence information currently recorded on the association degree extraction part correspondence holding | maintenance part. It is a figure which shows the logical structure of the relevance degree recorded on an extraction feature holding part. It is a flowchart explaining the detail of the process of color feature extraction. It is a flowchart explaining the detail of the process of relevance extraction. It is a figure which shows the RGB color space. It is a figure which shows L * a * b * space. It is a figure which shows the example of a white subspace and a black subspace. It is a figure which shows the example of a saturation boundary and a brightness | luminance boundary. It is a figure which shows the example of each subspace of green, blue, red, and yellow. It is a flowchart explaining the other example of the detail of a relevance degree extraction process. It is a flowchart explaining the further another example of the detail of a relevance degree extraction process. It is a figure which shows the example of judgment data. It is a flowchart explaining the further another example of the detail of a relevance degree extraction process. It is a flowchart explaining the process of a search. It is a figure which shows the example of the image of GUI in a search. It is a figure which shows the example of the searched image.

  Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described.

  FIG. 1 is a diagram showing a configuration of an image processing system according to an embodiment of the present invention. A digital still camera 11 that is an example of a device captures an image and supplies the captured image to a server 13 that is an example of an image processing apparatus. The mobile phone 12, which is an example of a device, captures an image and supplies the captured image to the server 13. In this case, the digital still camera 11 and the mobile phone 12 generate a reduced image obtained by reducing the image from the captured image.

  The digital still camera 11, the mobile phone 12, or the server 13 is also an example of a display control device.

  The server 13 includes a personal computer, a stationary recorder, a game device, a dedicated device, or the like, and records an image supplied from the digital still camera 11 or the mobile phone 12. In addition, the server 13 performs image processing on an image supplied from the digital still camera 11 or the mobile phone 12 and extracts image features. The server 13 supplies the data obtained as a result to the digital still camera 11 or the mobile phone 12.

  Furthermore, the server 13 acquires an image from the Web server 15-1 or the Web server 15-2 via the network 14, and records the acquired image. In addition, the server 13 performs image processing on the image acquired from the Web server 15-1 or the Web server 15-2, and generates a reduced image obtained by reducing the image from the acquired image. The server 13 supplies the data obtained as a result of the image processing to the digital still camera 11 or the mobile phone 12 together with the reduced image.

  The digital still camera 11 or the mobile phone 12 searches for a desired image from the recorded images based on the data obtained as a result of the image processing supplied from the server 13. Further, the server 13 searches for a desired image from the recorded images based on the data obtained as a result of the image processing.

  Since the digital still camera 11, the mobile phone 12, and the server 13 search for an image based on the same data obtained as a result of image processing, a desired image can be similarly searched.

  FIG. 2 is a block diagram showing a configuration of the digital still camera 11. The digital still camera 11 includes a photographing lens 31, an aperture 32, an imaging device 33, an analog signal processing unit 34, an A / D (Analog to Digital) converter 35, a digital signal processing unit 36, an MPU (Micro Processing Unit) 37, and a memory 38. , D / A (Digital to Analog) converter 39, monitor 40, compression / decompression unit 41, card I / F (interface) 42, memory card 43, AF (auto focus) motor zoom motor 44, control circuit 45, EEPROM (Electrically (Erasable Programmable Read Only Memory) 46, communication unit 47, communication unit 48, and input unit 49.

  The photographing lens 31 forms an optical image of the subject on the light receiving surface of the imaging device 33 via the diaphragm 32. The photographing lens 31 is composed of one or a plurality of lenses. The taking lens 31 may be a single focus lens or a variable focal length such as a zoom lens.

  The diaphragm 32 adjusts the amount of optical image formed on the light receiving surface of the imaging device 33.

  The imaging device 33 includes a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor, and converts an optical image formed on the light receiving surface into an electrical signal. The imaging device 33 supplies the electrical signal obtained by the conversion to the analog signal processing unit 34.

  The analog signal processing unit 34 includes a sampling hold circuit, a color separation circuit, a gain adjustment circuit, and the like, applies a correlated double sampling (CDS) process to the electrical signal from the imaging device 33, and converts the electrical signal to R, G, Separated into B color signals, the signal level of each color signal is adjusted (pre-white balance processing). The analog signal processing unit 34 supplies the color signal to the A / D converter 35.

  The A / D converter 35 converts each color signal into a digital signal and supplies the digital signal to the digital signal processing unit 36.

  The digital signal processing unit 36 includes a luminance / color difference signal generation circuit, a sharpness correction circuit, a contrast correction circuit, a white balance correction circuit, and the like. Based on the control of the MPU 37, the digital signal is converted into a luminance signal (Y signal) and a color difference signal. (Cr, Cb signal). The digital signal processing unit 36 supplies a digital signal to which various processes are applied to the memory 38.

  The MPU 37 is an embedded processor, and executes a program to control the entire digital still camera 11.

  The memory 38 includes a DRAM (Dynamic Random Access Memory) or the like, and temporarily stores the digital signal supplied from the digital signal processing unit 36 based on the control of the MPU 37. The D / A converter 39 reads a digital signal from the memory 38, converts the read digital signal into an analog signal, and supplies the analog signal to the monitor 40. The monitor 40 includes an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, and displays an image based on an analog signal supplied from the D / A converter 39.

  The digital signal in the memory 38 is periodically rewritten by the electrical signal output from the imaging device 33, and the analog signal generated from the digital signal is supplied to the monitor 40, whereby the image formed on the imaging device 33. Is displayed on the monitor 40 in real time.

  When displaying a GUI (Graphical User Interface) image on the monitor 40, the MPU 37 writes image data for displaying the GUI image in the memory 38, and the D / A converter 39 converts the image data into an analog signal. The GUI image is displayed on the monitor 40 based on the analog signal.

  Under the control of the MPU 37, the compression / decompression unit 41 encodes the digital signal stored in the memory 38 using a scheme such as JPEG (Joint Photographic Experts Group) or JPEG2000. The compression / decompression unit 41 supplies the image data obtained by the encoding to the memory card 43 via the card I / F (interface) 42. The memory card 43 has a built-in semiconductor memory or HDD (Hard Disk Drive) and is detachably attached to the digital still camera 11. When the memory card 43 is attached to the digital still camera 11, the memory card 43 is electrically connected to the card I / F 42. Connecting. The memory card 43 records the image data supplied from the card I / F 42.

  The card I / F 42 controls recording of image data to the electrically connected memory card 43 and reading of image data from the memory card 43 in accordance with a command from the MPU 37.

  The image data recorded on the memory card 43 is read out via the card I / F 42 and decoded into a digital signal by the compression / decompression unit 41.

  The AF motor zoom motor 44 is driven by the control circuit 45 and moves the photographic lens 31 (a lens constituting the photographic lens 31) with respect to the imaging device 33 so as to change the focal point and focal length of the photographic lens 31. The control circuit 45 drives the AF motor zoom motor 44 and controls the aperture 32 and the imaging device 33 in accordance with a command from the MPU 37.

  The EEPROM 46 stores programs executed by the MPU 37 and various data.

  The communication unit 47 is configured to comply with a standard such as USB (Universal Serial Bus) or IEEE (Institute of Electrical and Electronic Engineers) 1394, and transmits / receives various data to / from the server 13 via a wired transmission medium. .

  The communication unit 48 is configured to comply with a standard such as IEEE802.11a, IEEE802.11b, IEEE802.11g, or Bluetooth, and transmits / receives various data to / from the server 13 via a wireless transmission medium.

  The input unit 49 includes a switch, a button, a touch panel, or the like, and supplies a signal corresponding to an operation applied by the user to the MPU 37.

  Although it has been described that the image data is recorded on the memory card 43, the medium on which the image data is recorded is not limited to a semiconductor memory or a magnetic disk, and may be an optical disk or a magneto-optical disk. Various media that can be read and written can be used in accordance with a method based on optical, optical, or quantum, or a combination thereof. These media may be built in the digital still camera 11.

  Hereinafter, the image data is also simply referred to as an image.

  FIG. 3 is a block diagram illustrating an example of the configuration of the server 13. A CPU (Central Processing Unit) 71 executes various processes according to a program stored in a ROM (Read Only Memory) 72 or a storage unit 78. A RAM (Random Access Memory) 73 appropriately stores programs executed by the CPU 71 and data. These CPU 71, ROM 72, and RAM 73 are connected to each other by a bus 74.

  An input / output interface 75 is also connected to the CPU 71 via a bus 74. The input / output interface 75 is connected to an input unit 76 such as a keyboard, mouse, and microphone, and an output unit 77 such as a display and a speaker. The CPU 71 executes various processes in response to commands input from the input unit 76. Then, the CPU 71 outputs the processing result to the output unit 77.

  The storage unit 78 connected to the input / output interface 75 includes, for example, a hard disk, and stores a program executed by the CPU 71 and various data. The communication unit 79 is configured to comply with a standard such as USB or IEEE1394, and transmits / receives various data to / from the digital still camera 11 or the mobile phone 12 via a wired transmission medium, or IEEE802.11a, It is configured to comply with standards such as IEEE802.11b, IEEE802.11g, or Bluetooth, and transmits / receives various data to / from the digital still camera 11 or the mobile phone 12 via a wireless transmission medium. The communication unit 80 communicates with the Web server 15-1 or the Web server 15-2 via the network 14 such as the Internet or a local area network.

  A program may be acquired via the communication unit 80 and stored in the storage unit 78.

  The drive 81 connected to the input / output interface 75 drives a removable medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 78 as necessary.

  FIG. 4 is a diagram illustrating a configuration of functions realized by the MPU 37 that executes a program. By executing the program, the MPU 37 performs shooting control unit 101, reduced image generation unit 102, metadata generation unit 103, entry generation unit 104, recording control unit 105, display control unit 106, search unit 107, and transmission control unit 108. A reception control unit 109, an image holding unit 110, a content database 111, a similar feature database 112, a similar result database 113, a time group database 114, and a search result holding unit 115.

  The photographing control unit 101 controls photographing in the digital still camera 11 by controlling the photographing lens 31 through the digital signal processing unit 36 and the memory 38 through the control circuit 45. The shooting control unit 101 records the shot image in the recording area of the memory card 43 serving as the image holding unit 110.

  The reduced image generation unit 102 reads out a digital signal of the captured image from the memory 38, reduces the captured image, and generates a reduced image. The generated reduced image is supplied to the memory card 43 via the card I / F 42 and recorded in a recording area of the memory card 43 as the image holding unit 110.

  For example, when a high-resolution image having 3 to 4 million pixels is captured based on the control of the imaging control unit 101, the reduced image generation unit 102 uses the digital still camera 11 from the captured image. A reduced image having the same size as or equivalent to a 640 pixel × 480 pixel VGA (Video Graphics Array) suitable for browsing with the above method is generated.

  Note that the reduced image generation unit 102 may read an image from the image holding unit 110, reduce the read image, and generate a reduced image.

  Hereinafter, in order to distinguish between a reduced image and a captured image, the captured image is referred to as a main image. Note that when there is no need to distinguish between the reduced image and the main image, they are simply referred to as images.

  Although details will be described later, the main image and the reduced image are linked by data recorded in the content database 111.

  The metadata generation unit 103 generates metadata about the main image. For example, the metadata generation unit 103 generates metadata stored in EXIF (Exchangeable Image File Format) data standardized by JEIDA (Japanese Electronic Industry Development Association).

  The entry generation unit 104 is configured as a database management system, and generates an entry for the main image and the reduced image when the main image is captured. The generated entry is stored in the content database 111.

  The recording control unit 105 controls recording of the main image and the reduced image in the image holding unit 110.

  The display control unit 106 controls the display of the reduced image and the GUI image on the monitor 40.

  Based on the data stored in the content database 111, the similar feature database 112, the similar result database 113, or the time group database 114, the search unit 107 uses the reduced image or the main image recorded in the image holding unit 110. A desired reduced image or main image is searched. The search unit 107 causes the search result holding unit 115 to store data corresponding to the search result.

  The search unit 107 includes a distance calculation unit 121. The distance calculation unit 121 calculates a distance indicating the degree of similarity between two images from data indicating image features stored in the similar feature database 112. The distance calculation unit 121 records the calculated distance in the similarity result database 113.

  The transmission control unit 108 controls the communication unit 47 to control transmission of the main image or the reduced image to the server 13 by the communication unit 47. The reception control unit 109 controls the communication unit 47 to control reception of image features obtained by applying various types of image processing to images transmitted from the server 13 by the communication unit 47.

  The image holding unit 110 is constructed in the recording space of the memory card 43 and records a main image or a reduced image.

  The content database 111, the similar feature database 112, the similar result database 113, and the time group database 114 are composed of a predetermined recording space of the memory card 43 and respective database management systems.

  The content database 111 stores data for specifying an image and various metadata of the image corresponding to the data. The similar feature database 112 stores data indicating image features obtained as a result of image processing of images in the server 13.

  The similarity result database 113 stores a distance indicating the degree of similarity between two images calculated by the distance calculation unit 121 of the search unit 107.

  The time group database 114 stores information for specifying images belonging to each group when the user classifies the images into groups.

  The search result holding unit 115 records data according to the search result. For example, the search result holding unit 115 is input based on the relevance indicating the degree to which the image is recalled by a predetermined color name extracted based on the color of the pixel of the image and the operation from the user. The search result of the image of the color corresponding to the weight retrieved from the color weight represented by the color name is recorded.

  Details of the relevance will be described later.

  FIG. 5 is a diagram illustrating a configuration of functions realized by the CPU 71 that executes a program. By executing the program, the CPU 71 executes an image analysis unit 131, a reduced image generation unit 132, a metadata generation unit 133, an entry generation unit 134, a recording control unit 135, a display control unit 136, a search unit 137, and a transmission control unit 138. -1 and transmission control unit 138-2, reception control unit 139-1 and reception control unit 139-2, image holding unit 140, content database 141, similar feature database 142, similar result database 143, time group database 144, relevance An extraction unit correspondence holding unit 145, an extraction feature holding unit 146, and a search result holding unit 147 are realized.

  The image analysis unit 131 extracts image features. That is, the image analysis unit 131 analyzes the image by applying image processing to the image. The image analysis unit 131 supplies the image features obtained as a result of the image processing to the similar feature database 142 or the transmission control unit 138-1.

  FIG. 6 is a block diagram illustrating an example of the configuration of the image analysis unit 131. The image analysis unit 131 includes a face image detection unit 161 and a similar feature amount extraction unit 162.

  The face image detection unit 161 extracts image features that are information related to the face image included in the image. For example, the face image detection unit 161 extracts image features such as the number of face images included in the image, the position of the face image in the image, the size of the face image, or the orientation of the face in the face image. To do.

  The similar feature amount extraction unit 162 extracts the feature amount of the image for obtaining the degree of similarity of the images. The similar feature amount extraction unit 162 includes a similar feature vector calculation unit 171 and a color feature extraction unit 172. The similar feature vector calculation unit 171 extracts a feature whose degree of similarity between the two images is calculated from the features of the two images. The color feature extraction unit 172 extracts, from the image, a degree of association indicating the degree to which the image is recalled by a predetermined color name based on the color of the pixel of the image. In other words, the color feature extraction unit 172 extracts a feature indicating the number of pixels that are classified into a color with a predetermined name among the pixels of the image.

  Returning to FIG. 5, the reduced image generation unit 132 reduces and reduces the main image acquired from the Web server 15-1 or the Web server 15-2 via the network 14 under the control of the reception control unit 139-2. Generate an image. The generated reduced image is recorded in the image holding unit 140.

  Note that the reduced image generation unit 132 may read an image from the image holding unit 140, reduce the read image, and generate a reduced image.

  The metadata generation unit 133 generates metadata about the main image. For example, the metadata generation unit 133 generates metadata stored in EXIF format data standardized by JEIDA.

  The entry generation unit 134 is configured as a database management system, and generates an entry of the main image acquired from the digital still camera 11 under the control of the reception control unit 139-1. Alternatively, the entry generation unit 134 acquires a main image from the Web server 15-1 or the Web server 15-2 via the network 14 under the control of the reception control unit 139-2, and generates a reduced image from the main image. If so, entries for the main image and the reduced image are generated. The generated entry is stored in the content database 141.

  The recording control unit 135 controls recording of the main image and the reduced image in the image holding unit 140.

  The display control unit 136 controls display of the main image and the GUI image on the output unit 77 which is a display.

  Based on the data stored in the content database 141, the similar feature database 142, or the time group database 144, the search unit 137 selects a desired main image or reduced image from the main image or the reduced image recorded in the image holding unit 140. Search for reduced images. Alternatively, the search unit 137 searches for a desired main image or reduced image from the main image or reduced image recorded in the image holding unit 140 based on the data stored in the extracted feature holding unit 146. The search unit 137 stores data corresponding to the search result in the search result holding unit 147.

  The search unit 137 includes a distance calculation unit 151. The distance calculation unit 151 calculates a distance indicating the degree of similarity between two images from data indicating image features stored in the similar feature database 142. The distance calculation unit 151 records the calculated distance in the similarity result database 143.

  The transmission control unit 138-1 controls the communication unit 79 to cause the communication unit 79 to transmit the image characteristics obtained as a result of image processing in the image analysis unit 131 to the digital still camera 11. The reception control unit 139-1 controls the communication unit 79 to cause the communication unit 79 to receive the main image and the reduced image transmitted from the digital still camera 11.

  The transmission control unit 138-2 controls the communication unit 80. The transmission control unit 138-2 causes the communication unit 80 to transmit an image request to the Web server 15-1 or the Web server 15-2 via the network 14. The reception control unit 139-2 controls the communication unit 80 to cause the communication unit 80 to receive the main image transmitted from the Web server 15-1 or the Web server 15-2.

  The image holding unit 140 is constructed in a recording space of the storage unit 78 composed of a hard disk or the like, and records a main image or a reduced image. The image holding unit 140 may be constructed in a recording space of a removable medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory that is mounted on the drive 81.

  The content database 141, the similar feature database 142, the similar result database 143, and the time group database 144 are configured by a predetermined recording space of the storage unit 78 and respective database management systems.

  The content database 141 stores data specifying an image and various metadata of the image corresponding to the data. The similar feature database 142 stores data indicating image features obtained as a result of image processing of the image by the image analysis unit 131.

  The similarity result database 113 stores a distance indicating the degree of similarity between two images calculated by the distance calculation unit 151 of the search unit 137.

  The time group database 144 stores information for specifying images belonging to each group when the user classifies the images into groups.

  The association degree extraction unit correspondence holding unit 145 indicates the correspondence between the color name in the color feature extraction unit 172 and the association degree extraction unit that extracts the association degree for each color (details will be described later with reference to FIG. 33). Record correspondence information.

  The extracted feature holding unit 146 holds a degree of association indicating the degree to which an image is recalled by a predetermined color name extracted by the color feature extracting unit 172.

  The search result holding unit 147 extracts the degree of relevance indicating the degree to which the image is recalled by a predetermined color name, extracted based on the color of the pixel of the image, and the search condition input according to the operation from the user. The search result of the image of the color corresponding to the search condition retrieved from is recorded. For example, the search result holding unit 147 records the search result of the color image corresponding to the weight searched from the relevance and the search condition that is the weight of the color represented by the color name.

  Next, processing for extracting features from an image and recording the extracted features in the server 13 and the digital still camera 11 will be described.

  First, with reference to the flowchart of FIG. 7, the photographing process of the digital still camera 11 will be described.

  In step S11, the photographing control unit 101 controls the photographing lens 31 through the digital signal processing unit 36, the memory 38, the AF motor zoom motor 44, and the control circuit 45 to photograph the subject. In step S12, the imaging control unit 101 causes the compression / decompression unit 41 to encode the digital signal stored in the memory 38 using a method such as JPEG or JPEG2000 to generate a main image that is image data. The imaging control unit 101 causes the image holding unit 110 to record the main image.

  In addition, the metadata generation unit 103 generates metadata about the main image. For example, the metadata generation unit 103 generates metadata such as the shooting time or shooting condition of the main image stored in EXIF format data standardized by JEIDA.

  In step S <b> 13, the reduced image generation unit 102 reads a digital signal of the captured image from the memory 38, reduces the captured image, and generates a reduced image. The reduced image generation unit 102 causes the image holding unit 110 to record the reduced image.

  In step S <b> 14, the entry generation unit 104 generates an entry for the main image and the reduced image. The entry generation unit 104 adds (stores) the generated entry to the content database 111 in association with the metadata generated by the metadata generation unit 103, and the process ends.

  Since metadata such as shooting time or shooting conditions is stored in the content database 111, the main image or the reduced image can be searched based on the shooting time or shooting conditions.

  Also in the mobile phone 12, the same processing as the photographing processing shown in the flowchart of FIG. 7 is executed.

  In this way, as shown in FIG. 8, when the digital still camera 11 or the mobile phone 12 captures an image, the metadata associated with the main image 201 is stored in the content database 111. At the same time, a reduced image 202 obtained by reducing the main image 201 is generated, and the metadata stored in the content database 111 and the reduced image 202 are related to each other.

  Next, a backup process of the server 13 when an image captured by the digital still camera 11 is backed up to the server 13 will be described with reference to a flowchart of FIG. The backup process of the server 13 is started by starting the program when a USB cable, one end of which is connected to the digital still camera 11, is connected to the server 13, for example.

  In step S31, the transmission control unit 138-1 and the reception control unit 138-1 of the server 13 cause the communication unit 79 to connect to the digital still camera 11.

  In step S <b> 32, the transmission control unit 138-1 and the reception control unit 138-1 of the server 13 cause the communication unit 79 to acquire the main image 201 and the reduced image 202 from the digital still camera 11. For example, in step S <b> 32, the transmission control unit 138-1 causes the communication unit 79 to transmit a transmission request for the main image 201 and the reduced image 202 to the digital still camera 11. Then, since the digital still camera 11 transmits the main image 201 and the reduced image 202, the reception control unit 138-1 sends the main image 201 and the reduced image 202 transmitted from the digital still camera 11 to the communication unit 79. To receive. The reception control unit 138-1 supplies the acquired (received) main image 201 and reduced image 202 to the image holding unit 140.

  In step S <b> 33, the image holding unit 140 records the main image 201 and the reduced image 202 acquired from the digital still camera 11.

  In step S <b> 34, the image analysis unit 131 analyzes the image recorded in the image holding unit 140.

  Note that the image analysis unit 131 may analyze the main image 201 or the reduced image 202.

  Details of the image analysis processing in step S34 will be described with reference to the flowchart of FIG.

  In step S41, the face image detection unit 161 of the image analysis unit 131 detects a face image from the image. That is, in step S41, the face image detection unit 161 extracts image features that are information related to the face image included in the image. For example, in step S41, the face image detection unit 161 detects the number of face images included in the image, the position of the face image in the image, the size of the face image, or the orientation of the face in the face image. Extract features.

  More specifically, for example, the face image detection unit 161 identifies a pixel having a pixel value indicating a color belonging to a predetermined color range corresponding to the color of human skin among the pixels of the image. Then, the face image detection unit 161 sets, as a face image, an area formed by pixels adjacent to each other by a predetermined number or more among the pixels specified by color.

  The face image detection unit 161 counts the number of detected face images. Further, the face image detection unit 161 assumes that the position of the face image in the image is relative to the entire image in the vertical direction of the face image when the overall height and width of the image are each 1. Detect position and lateral position.

  Further, the face image detection unit 161 sets the height of the face image relative to the entire image as the size of the face image in the image when the overall height and the overall width of the image are 1, respectively. And detect width.

  Then, the face image detection unit 161 determines whether or not the plurality of patterns for each assumed face direction that are defined in advance match the selected face image, and determines the face orientation as the face orientation. The orientation of the face is detected by setting the orientation corresponding to the pattern that matches the image. In this case, the face image detection unit 161 detects the roll angle, pitch angle, and yaw angle of the face as the face direction for the selected face image.

  In step S42, the similar feature vector calculation unit 171 of the similar feature amount extraction unit 162 of the image analysis unit 131 calculates a similar feature vector that is a feature amount for obtaining the degree of similarity of images. That is, in step S42, the similar feature vector calculation unit 171 extracts features for which the degree of similarity between the two images is calculated from the features of the two images.

  For example, the similar feature vector calculation unit 171 calculates a similar feature vector that is a color histogram.

  More specifically, for example, as illustrated in FIG. 11, the similar feature vector calculation unit 171 reduces the color of 1677772161 colors of the 24-bit RGB main image 201 to 32 colors and reduces the color to 32 colors. An image 221 is generated. That is, a 5-bit RGB reduced color image 221 is generated. For example, the similar feature vector calculation unit 171 generates a reduced color image 221 by extracting a predetermined upper bit from the pixel value of each pixel of the main image 201.

  Then, the similar feature vector calculation unit 171 converts the color of each pixel of the subtractive color image 221 represented by RGB so as to represent L * a * b *. That is, the similar feature vector calculation unit 171 specifies a position in the L * a * b * space indicating the color of each pixel of the color-reduced image 221. In other words, for each pixel of the reduced color image 221, any one of the 32 colors (positions in the L * a * b * space) indicated by each pixel of the reduced color image 221 is specified.

  Further, the similar feature vector calculation unit 171 determines the number of pixels for each of the 32 colors, that is, the frequency for each color, and generates a color histogram for the subtractive color image 221. The scale of the color histogram indicates a color, and the frequency of the color histogram indicates the number (frequency) of pixels of the color.

  For example, the similar feature vector calculation unit 171 calculates similar feature vectors that are a vertical component histogram and a horizontal component histogram.

  In this case, first, as shown in FIG. 12, the similar feature vector calculation unit 171 divides the main image 201 into blocks 241 each having 16 pixels × 16 pixels, and each block 241 has a vertical direction (vertical) and a vertical direction. DFT (Discrete Fourier Transform) processing is applied in the horizontal direction (horizontal).

  That is, the similar feature vector calculation unit 171 applies DFT processing to 16 pixels arranged in one vertical column of each block 241 and extracts frequency components of the image of 16 pixels in one vertical column. Since each block 241 includes 16 columns of 16 pixels, the similar feature vector calculation unit 171 performs the frequency of 16 images by performing vertical (vertical) DFT processing for each block 241. The component will be extracted.

  Then, the similar feature vector calculation unit 171 accumulates (adds) the frequency components of the image obtained as a result of applying the vertical (vertical) DFT processing to each block 241 for each frequency. The similar feature vector calculation unit 171 calculates the maximum value from eight lower frequency components excluding the DC component among the values obtained by integrating the results of applying the vertical (vertical) DFT processing to each block 241. Extract ingredients. In this case, when the maximum value is less than the predetermined threshold value, the processing result of the block 241 is discarded.

  The similar feature vector calculation unit 171 accumulates the maximum value for each block 241 in this way for each of the eight frequencies for the image, thereby obtaining the frequency of the maximum value for the eight frequencies as shown in FIG. A vertical component histogram is generated. The scale of the vertical component histogram indicates the frequency of the image, and the frequency of the vertical component histogram indicates the number (frequency) at which the frequency component is maximum.

  Similarly, the similar feature vector calculation unit 171 applies DFT processing to 16 pixels arranged in one horizontal row of each block 241 and extracts frequency components of the image of 16 pixels in one horizontal row. Since each block 241 includes 16 rows of 16 pixels, the similar feature vector calculation unit 171 performs the frequency of 16 images by performing horizontal (horizontal) DFT processing for each block 241. The component will be extracted.

  Then, the similar feature vector calculation unit 171 accumulates (adds) the frequency components of the image obtained as a result of applying the DFT processing to each block 241 in the horizontal direction (lateral) for each frequency. The similar feature vector calculation unit 171 calculates the maximum value from eight lower frequency components excluding the DC component among the values obtained by integrating the results of applying the horizontal (horizontal) DFT processing to each block 241. Extract ingredients. In this case, when the maximum value is less than the predetermined threshold value, the processing result of the block 241 is discarded.

  The similar feature vector calculation unit 171 accumulates the maximum value for each block 241 in this way for each of the eight frequencies for the image, thereby obtaining the frequency of the maximum value for the eight frequencies as shown in FIG. A horizontal component histogram is generated. The scale of the horizontal component histogram indicates the frequency of the image, and the frequency of the horizontal component histogram indicates the number (frequency) at which the frequency component is maximum.

  As described above, the similar feature vector calculation unit 171 generates a vertical component histogram and a horizontal component histogram for an image.

  For example, in step S42, the similar feature vector calculation unit 171 extracts a color histogram, a vertical component histogram, and a horizontal component histogram as features for calculating the degree of similarity between the two images from the features of the two images. To do.

  Returning to FIG. 10, in step S43, the color feature extraction unit 172 of the similar feature amount extraction unit 162 of the image analysis unit 131 applies the color feature extraction process to the image, and the process ends. Through the color feature extraction process, a degree of association indicating the degree to which the image is recalled by a predetermined color name is extracted from the image based on the color of the pixel of the image. Details of the color feature extraction processing will be described later with reference to the flowchart of FIG.

  As described above, in step S34, the image analysis unit 131 analyzes the image recorded in the image holding unit 140, and extracts the feature of the image.

  In step S35, the metadata generation unit 133 generates image metadata including the image features extracted in step S34. In step S <b> 36, the entry generation unit 134 generates entries for the main image 201 and the reduced image 202. The entry generation unit 134 adds (stores) the generated entry to the content database 141 and the similar feature database 142 in association with the metadata generated in step S35. The content database 141 and the similar feature database 142 record metadata including image features extracted by the server 13.

  In step S <b> 37, the transmission control unit 138-1 causes the communication unit 79 to enter metadata including the extracted image features in the content database 111 and the similar feature database 112 of the digital still camera 11. That is, in step S37, the transmission control unit 138-1 sends the metadata generated in step S35 to the digital still camera 11 to the communication unit 79 together with instructions for filling the content database 111 and the similar feature database 112. Send it. The reception control unit 109 of the digital still camera 11 causes the communication unit 47 to receive the metadata and a command to fill in the content database 111 and the similar feature database 112, and then to the metadata, the content database 111, and the similar feature database 112. The content database 111 and the similar feature database 112 are supplied. When the content database 111 and the similar feature database 112 receive an entry command, the content database 111 and the similar feature database 112 record metadata including image features extracted by the server 13.

  As described above, the content database 141 and the similar feature database 142 and the content database 111 and the similar feature database 112 record the same metadata including the image features extracted in the server 13.

  In step S38, the transmission control unit 138-1 and the reception control unit 138-1 of the server 13 cause the communication unit 79 to disconnect from the digital still camera 11, and the process ends.

  Note that the server 13 can execute the same processing as the backup processing shown in the flowchart of FIG. 9 on the image captured by the mobile phone 12 with respect to the mobile phone 12.

  As shown in FIG. 14, when an image taken with the digital still camera 11 or the mobile phone 12 is backed up to the server 13-1 or the server 13-2, the server 13-1 or the server 13-2 is backed up. The image is analyzed to extract the feature of the image, and the metadata 261 describing the extracted feature of the image is written back to the digital still camera 11 or the mobile phone 12.

  FIG. 15 is a diagram illustrating a specific example of the metadata 261 describing the characteristics of the extracted image related to the main image 201 and the reduced image 202.

  The metadata 261 is described in, for example, an XML (eXtensible Mark-up Language) method.

  Between the <photo> tag and the </ photo> tag, information indicating the relationship between the main image 201 and the reduced image 202 and information indicating the characteristics of the main image 201 and the reduced image 202 are arranged.

  Between the <guid> tag and the </ guid> tag, a content ID which is specific information for specifying the main image 201 and the reduced image 202 related to the metadata 261 is arranged. For example, the content ID is 128 bits. The content ID is common to the main image 201 and the reduced image 202 obtained by reducing the main image 201.

  Between the <FullImgPath> tag and the </ FullImgPath> tag, the path of the file storing the main image 201 as image data and the file name of the file storing the main image 201 as image data are arranged. The Between the <CacheImgPath> tag and the </ CacheImgPath> tag, the path of the file storing the reduced image 202 as image data and the file name of the file storing the reduced image 202 as image data are arranged. The

  In 2003: 03: 31 06:52:32 placed between the <TimeStamp> tag and the </ TimeStamp> tag, the main image 201 was taken at 6:52:32 on March 31, 2003 It is a time stamp indicating that.

  Between the <Faceinfo> tag and the </ Faceinfo> tag, information related to the face image included in the main image 201 and the reduced image 202 specified by the content ID is arranged.

  1 arranged between the <TotalFace> tag and the </ TotalFace> tag indicates that the number of face images included in the main image 201 or the reduced image 202 specified by the content ID is one. That is, the value arranged between the <TotalFace> tag and the </ TotalFace> tag indicates the total number of face images included in the main image 201 or the reduced image 202 specified by the content ID.

  Information about one face image is arranged between the <FaceEntry> tag and the </ FaceEntry> tag. Since the total number of face images in the metadata 261 illustrated in FIG. 15 is 1, a set of <FaceEntry> tags and </ FaceEntry> tags are arranged.

  The value arranged between the <x> tag and the </ x> tag indicates the horizontal position of the face image in the main image 201 or the reduced image 202 specified by the content ID. In FIG. 15, 0.328767 arranged between the <x> tag and the </ x> tag sets 0.0 as the left end of the main image 201 or the reduced image 202 and 1.0 as the right end of the main image 201 or the reduced image 202. In this case, the horizontal position of the right end of the face image is 0.328767.

  The value arranged between the <y> tag and the </ y> tag indicates the position in the vertical direction of the face image in the main image 201 or the reduced image 202 specified by the content ID. In FIG. 15, 0.204082 arranged between the <y> tag and the </ y> tag sets the upper end of the main image 201 or the reduced image 202 to 0.0 and sets the lower end of the main image 201 or the reduced image 202 to 1.0. In this case, the vertical position of the upper end of the face image is 0.204082.

  That is, the normalized horizontal position of the face image is placed between the <x> tag and the </ x> tag, and the face image is placed between the <y> tag and the </ y> tag. The normalized vertical position of the image is placed.

  A value arranged between the <width> tag and the </ width> tag indicates the width (lateral size) of the face image in the main image 201 or the reduced image 202 specified by the content ID. In FIG. 15, 0.408163 arranged between the <width> tag and the </ width> tag is 0.408163 when the width of the main image 201 or the reduced image 202 is 1.0. It shows that.

  Arranged between the <height> tag and the </ height> tag indicates the height (vertical size) of the face image in the main image 201 or the reduced image 202 specified by the content ID. In FIG. 15, 0.273973 arranged between the <height> tag and the </ height> tag indicates that the height of the face image is 0.273973 when the height of the main image 201 or the reduced image 202 is 1.0. Indicates that

  That is, the normalized width of the face image is placed between the <width> tag and the </ width> tag, and the face image normalization is placed between the <height> tag and the </ height> tag. The height is arranged.

  The value arranged between the <roll> tag and the </ roll> tag indicates the roll angle of the face in the face image. In FIG. 15, 0.000000 arranged between the <roll> tag and the </ roll> tag indicates that the face roll angle in the face image is 0.000000 degrees.

  The value arranged between the <pitch> tag and the </ pitch> tag indicates the pitch angle of the face in the face image. In FIG. 15, 0.000000 arranged between the <pitch> tag and the </ pitch> tag indicates that the face pitch angle in the face image is 0.000000 degrees.

  The value arranged between the <yaw> tag and the </ yaw> tag indicates the face yaw angle in the face image. In FIG. 15, 0.000000 arranged between the <yaw> tag and the </ yaw> tag indicates that the face yaw angle in the face image is 0.000000 degrees.

  Here, the roll angle is a movement angle around the front-rear axis (x-axis) indicating the position of the face in the front-rear direction. The pitch angle is a movement angle around the horizontal axis (y axis) indicating the position of the face in the left-right direction. The yaw angle is a movement angle around the vertical axis (z axis) indicating the vertical position of the face.

  Between the <Similarityinfo> tag and the </ Similarityinfo> tag, a book specified by the content ID used when obtaining the degree of similarity between the master image 201 specified by the content ID or the reduced image 202 and another image The feature amounts of the image 201 and the reduced image 202 are arranged.

  In the example shown in FIG. 15, between the <Similarityinfo> tag and the </ Similarityinfo> tag, the degree of association indicating the degree to which the main image 201 or the reduced image 202 is recalled by a predetermined color name, and the frequency of the color or image A feature amount for calculating the degree of similarity such as a component is arranged.

  Between the <ColorInfo> tag and the </ ColorInfo> tag, the main image is extracted based on the pixel color of the main image 201 or the reduced image 202 extracted from the main image 201 or the reduced image 202 specified by the content ID. A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by a predetermined color name is arranged.

  Between the <ColorWhite> tag and the </ ColorWhite> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the color of the pixel of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by the color name of white is arranged. In FIG. 15, 0 arranged between the <ColorWhite> tag and the </ ColorWhite> tag has a relevance level of 0 indicating the degree to which the main image 201 or the reduced image 202 is recalled by a white color name. It shows that.

  Between the <ColorBlack> tag and the </ ColorBlack> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the color of the pixel of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by the color name of black is arranged. In FIG. 15, 0 arranged between the <ColorBlack> tag and the </ ColorBlack> tag has a relevance degree of 0 indicating the degree to which the main image 201 or the reduced image 202 is recalled by a black color name. It shows that.

  Between the <ColorRed> tag and the </ ColorRed> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the pixel color of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by the color name of red is arranged. In FIG. 15, 0 arranged between the <ColorRed> tag and the </ ColorRed> tag has a relevance of 0 indicating the degree to which the main image 201 or the reduced image 202 is recalled by a red color name. It shows that.

  Between the <ColorYellow> tag and the </ ColorYellow> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the color of the pixel of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by the color name of yellow is arranged. In FIG. 15, 0 arranged between the <ColorYellow> tag and the </ ColorYellow> tag has a relevance of 0 indicating the degree to which the main image 201 or the reduced image 202 is recalled by a yellow color name. It shows that.

  Between the <ColorGreen> tag and the </ ColorGreen> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the pixel color of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by a color name of green is arranged. In FIG. 15, 12 arranged between the <ColorGreen> tag and the </ ColorGreen> tag has a relevance of 0.12 indicating the degree to which the main image 201 or the reduced image 202 is recalled by a green color name. Indicates that That is, here, the degree of association is recorded in% (percent) notation.

  Between the <ColorBlue> tag and the </ ColorBlue> tag, the main image extracted from the main image 201 or the reduced image 202 specified by the content ID based on the pixel color of the main image 201 or the reduced image 202 A degree of association indicating the degree to which 201 or the reduced image 202 is recalled by the color name of blue is arranged. In FIG. 15, 0 arranged between the <ColorBlue> tag and the </ ColorBlue> tag has a relevance of 0 indicating the degree to which the main image 201 or the reduced image 202 is recalled by the color name being yellow. It shows that.

  Between the <VectorInfo> tag and the </ VectorInfo> tag, the main image specified by the content ID for obtaining the degree of similarity between the main image 201 specified by the content ID or the reduced image 202 and another image. Features for 201 or reduced image 202 are placed.

  Between one set of <VectorInfo> tag and </ VectorInfo> tag, one feature of the main image 201 or the reduced image 202 specified by the content ID is arranged. In the example of the metadata 261 in FIG. 15, three sets of <VectorInfo> tag and </ VectorInfo> tag are described.

  Between each <VectorInfo> tag and </ VectorInfo> tag, a <method> tag and a </ method> tag, and a <vector> tag and a </ vector> tag are arranged. Between the <method> tag and the </ method> tag, a feature method for determining the degree of similarity is described. Between the <vector> tag and the </ vector> tag, the amount of the feature is described. Is done. The feature amount described between the <vector> tag and the </ vector> tag is a vector.

  In FIG. 15, the Color Feature arranged between the <method> tag and the </ method> tag between the top <VectorInfo> tag and the </ VectorInfo> tag is the next <vector> tag and The feature quantity arranged between the </ vector> tags is a color feature quantity. The color feature amount is, for example, a feature amount indicated by the color histogram described with reference to FIG.

  In FIG. 15, the Texture Feature arranged between the <method> tag and the </ method> tag between the <VectorInfo> tag and the </ VectorInfo> tag from the second top is the next <vector>. The feature amount arranged between the tag and the </ vector> tag indicates the feature amount of the pattern. The feature amount of the pattern is, for example, a feature amount indicated by a frequency component histogram including a vertical component histogram and a horizontal component histogram described with reference to FIGS. 12 and 13.

  As a whole, the metadata 261 is stored in the content database 111 and the similar feature database 112 in the digital still camera 11, and is stored in the content database 141 and the similar feature database 142 in the server 13. That is, the metadata 261 is appropriately divided, and the digital still camera 11 stores a part thereof in the content database 111, the remaining part is stored in the similar feature database 112, and is stored in the content database 111 in the server 13. The same part as the stored part is stored in the content database 141, and the same part as the part stored in the similar feature database 112 is stored in the similar feature database 142.

  FIG. 16 is a diagram illustrating a configuration of (parts of) metadata stored in the content database 111 or the content database 141.

  The metadata stored in the content database 111 or the content database 141 includes a content ID, a shooting time, a path name, a file name, a group ID, information on a face image included in the image (hereinafter referred to as face image information), It consists of a label ID and a comment.

  The content ID is an ID unique to the image and identifies the image. The main image 201 and the reduced image 202 are specified by the content ID. The content ID is a property that is a GUID, and is represented by a character string type. The shooting time indicating the date and time when the image was shot is expressed in coordinated universal time and local time. The shooting time expressed in Coordinated Universal Time is a property that is UTCdate and is expressed in the date type. The shooting time represented in Coordinated Universal Time is the same as the shooting time (UTC (Universal Coordinated Time)) written in the Date Time Original of the EXIF data.

  The shooting time expressed in local time is a property that is date, and is expressed in a date type. The photographing time represented by the local time is the same as the photographing time (Local time) written in Date Time Original of the EXIF data.

  The path name indicates the directory name (not including the file name) of the file of the main image 201 such as ms / DCIM / XXXXX /. The path name is a property that is path, and is expressed as a string type.

  The file name indicates the name of a file that stores the main image 201 that is image data, such as DSC00001.JPG. The file name is a property that is DCFname and is represented by a string type.

  The path name and file name of the reduced image 202 indicate the directory name and file name of the file of the reduced image 202 such as /DATA/EVENTIMAGE/000000000001.JPG. The path name and file name of the reduced image 202 are set as a property of vgaCachePath and are expressed in a character string type.

  The group ID is data for specifying the group to which the image belongs. The images are classified into a desired group by the user. The group ID specifies a group into which images are classified. For example, it is possible to create a group for each event (an event or event such as a trip or athletic meet) in which an image is taken, and classify the images taken at that event into groups corresponding to the event.

  The group ID is a property that is groupID, and is expressed by a numeric type.

  For example, the face image information may be a landscape image (image without a face), a small number of person images (images with 1 to 5 faces), or a large number of person images (6 or more people). Image showing the face of). For example, face image information of 1 indicates that the image is a landscape image, face image information of 2 indicates that the image is a small number of person images, and face image information of 3 indicates that the image is an image. Indicates that it is a portrait of a large number of people. The face image information is a property which is faceExistence and is expressed by a numerical value type.

  The face image information may indicate the number of face images included in the image, the position of the face image in the image, the size of the face image, or the orientation of the face in the face image.

  The label ID indicates a label attached to the image. The label ID is a property that is labels, and is expressed as a numeric array type.

  A comment is a property that is a comment and is represented by a string type.

  The protected state indicates the state of protection of the image such as deletion and addition. The protect state is a property that is protect and is expressed by a logical data type.

  The exchange / import flag indicates that the image has been exchanged or has been imported. The exchange / import flag is a property which is exchangeOrImportFlag, and is represented by a logical data type.

  A meta enable flag that is true indicates that the server 13 has generated metadata for the image. The meta enable flag is a property that is metaEnableFlag, and is represented by a logical data type.

  A backup flag that is True indicates that the image has been backed up by the server 13. The backup flag is a property which is backUpFlag and is expressed by a logical data type.

  FIG. 17 is a diagram illustrating the structures of metadata (parts) stored in the content database 111 and metadata (parts) stored in the similar feature database 112.

  The content database 111 stores content items for each image. The content item is a part of the metadata 261.

  For example, the content item 281-1 corresponds to one image specified by the stored content ID, the content ID, the path name and file name of the main image 201 (Path in FIG. 17), and the reduced image 202. The content item 281-2 corresponds to other images, and includes a content ID, a path name of the main image 201, and a path name and file name, a group ID, a shooting time represented by local time, and face image information. The file name (Path in FIG. 17), the path name and file name of the reduced image 202, the group ID, the shooting time expressed in local time, face image information, and the like.

  Hereinafter, when there is no need to distinguish between the content item 281-1 and the content item 281-2, they are simply referred to as the content item 281.

  The similar feature database 112 stores similar feature items for each image. The similar feature item includes data of a part other than the part constituting the content item 281 in the data constituting the metadata 261. However, the similar feature item includes a content ID.

  For example, the similar feature item 282-1 corresponds to the content item 281-1 specified by the stored content ID, that is, corresponds to one image specified by the stored content ID, and the content ID , A color histogram, and a histogram of frequency components.

  The color histogram indicates the frequency for each of the 32 colors of the image and is a property that is a histogram. The frequency component histogram is composed of a vertical component histogram and a horizontal component histogram, and indicates the frequency of the maximum value of the frequency component with respect to eight frequencies in each of the vertical direction and the horizontal direction of the image, and is a property which is a texture. .

  Similarly, for example, the similar feature item 282-2 corresponds to the content item 281-2 specified by the stored content ID, that is, corresponds to one image specified by the stored content ID. , Content ID, color histogram, and frequency component histogram.

  Hereinafter, when there is no need to distinguish between the similar feature item 282-1 and the similar feature item 282-2, they are simply referred to as the similar feature item 282.

  As described above, the similar feature database 112 stores the similar feature item 282 corresponding to the content item 281 stored in the content database 111.

  FIG. 18 is a diagram illustrating the structure of the similar feature item 282. The similar feature item 282 includes an item 291, an item 292-1 to an item 292-32, and an item 293. The item 291 includes a content ID, a pointer indicating the items 292-1 to 292-32, and a pointer indicating the item 293. The pointers indicating the items 292-1 to 292-32 correspond to the color histogram. A pointer indicating the item 293 corresponds to a histogram of frequency components.

  Each of the items 292-1 to 292-32 is the frequency of the color histogram, that is, each of the colors represented by L * a * b * and the ratio of each color occupied in the image (for example, The number of pixels for each of the 32 colors). The item 292-1 is a color represented by L * a * b *, and indicates the first color of the 32 colors and the ratio occupied by the first color in the image. The item 292-2 is a color represented by L * a * b *, and indicates the second color of the 32 colors and the ratio occupied by the second color in the image.

  The items 292-3 to 292-32 are colors represented by L * a * b *, respectively, and the third to thirty-second colors of the thirty-two colors and the third color The ratios of the thirty-second to thirty-second colors are indicated in the image.

  That is, the items 292-1 to 292-32 indicate a color histogram of one image as a whole. The color histogram can also be expressed as a color feature vector Cv. The color feature vector Cv is also expressed as Cv = {(c1, r1),..., (C32, r32)}. Here, each of (c1, r1) to (c32, r32) represents a ratio occupied in any one of the 32 colors represented by any of c1 to c32.

  Item 293 shows a vertical component histogram and a horizontal component histogram. Each of the vertical component histogram and the horizontal component histogram shows eight frequencies.

  The frequency component histogram formed by combining the vertical component histogram and the horizontal component histogram can also be expressed as a frequency component vector Tv. The frequency component vector Tv is also expressed as Tv = {(t1,1), ..., (t8,1), (t9,1), ..., (t16,1)}. Here, each of (t1,1) to (t16,1) indicates the maximum number (frequency) of frequency components represented by any of t1 to t16.

  Next, an image acquisition process of the server 13 for acquiring an image from the Web server 15-1, the Web server 15-2, or other devices will be described with reference to a flowchart of FIG. Hereinafter, a case where an image is acquired from the Web server 15-1 will be described as an example.

  In step S61, the transmission control unit 138-2 and the reception control unit 138-2 of the server 13 cause the communication unit 80 to acquire the main image 201 from the Web server 15-1 via the network 14.

  For example, in step S61, the transmission control unit 138-2 and the reception control unit 138-2 cause the communication unit 80 to connect to the Web server 15-1 via the network 14. Then, the transmission control unit 138-2 causes the communication unit 80 to transmit a transmission request for the main image 201 to the Web server 15-1 via the network 14. Since the Web server 15-1 transmits the requested master image 201 via the network 14, the reception control unit 138-2 sends the master image 201 transmitted from the Web server 15-1 to the communication unit 80. To receive. The reception control unit 138-2 supplies the main image 201 acquired by reception to the image holding unit 140.

  In step S <b> 62, the reduced image generation unit 132 generates a reduced image 202 from the received main image 201. For example, the reduced image generation unit 132 generates the reduced image 202 by thinning pixels from the main image 201. Alternatively, the reduced image generation unit 132 generates the reduced image 202 by using the average value of the pixel values of a plurality of adjacent pixels of the main image 201 as the pixel value of one pixel corresponding to the plurality of pixels. To do.

  The reduced image generation unit 132 supplies the generated reduced image 202 to the image holding unit 140.

  In step S <b> 63, the image holding unit 140 records the received main image 201 and the reduced image 202 generated by the reduced image generation unit 132.

   Note that the reduced image generation unit 132 may read the main image 201 from the image holding unit 140 and generate the reduced image 202 from the read main image 201.

  In step S <b> 64, the image analysis unit 131 analyzes the image recorded in the image holding unit 140. The image analysis processing in step S64 is the same as the processing described with reference to the flowchart of FIG.

  In step S65, the metadata generation unit 133 generates image metadata including the image features extracted in step S64. In step S <b> 66, the entry generation unit 134 generates entries for the main image 201 and the reduced image 202. The entry generation unit 134 adds (stores) the generated entry to the content database 141 (and the similar feature database 142) in association with the metadata generated in step S65.

  In step S67, the transmission control unit 138-1 and the reception control unit 138-1 cause the communication unit 79 to connect to the digital still camera 11.

  In step S <b> 68, the search unit 137 selects the reduced image 202 to be brought out to the digital still camera 11 from the reduced images 202 recorded in the image holding unit 140 based on the data transmitted from the digital still camera 11. . The search unit 137 reads the reduced image 202 selected from the image holding unit 140 and supplies the read reduced image 202 to the transmission control unit 138-1.

  In step S <b> 69, the transmission control unit 138-1 causes the communication unit 79 to transmit the selected reduced image 202 to the digital still camera 11.

  In step S70, the transmission control unit 138-1 performs processing similar to that in step S37, with the metadata of the reduced image 202 transmitted to the content database 111 and the similar feature database 112 of the digital still camera 11 to the communication unit 79. Then, the metadata including the extracted image features is entered.

  In step S72, the transmission control unit 138-1 and the reception control unit 138-1 of the server 13 cause the communication unit 79 to disconnect from the digital still camera 11, and the process ends.

  As illustrated in FIG. 20, the server 13-1 or the server 13-2 acquires the master image 201 from the Web server 15-1, the Web server 15-2, or other devices via the network 14 and acquires the master image 201. When the main image 201 is recorded, the server 13-1 or the server 13-2 generates a reduced image 202 from the main image 201, analyzes the main image 201, and extracts features of the main image 201. Then, the server 13-1 or the server 13-2 writes the reduced image 202 in the digital still camera 11 or the mobile phone 12 together with the metadata 261 describing the characteristics of the extracted main image 201.

  Next, search processing in the digital still camera 11 will be described with reference to the flowchart of FIG. In step S <b> 81, the search unit 107 selects metadata to be used for search from the metadata recorded in the content database 111 or the similar feature database 112. For example, the search unit 107 is recalled as the metadata used for the search based on the signal from the input unit 49 according to the user's operation based on the shooting time or shooting condition, information on the face image, and a predetermined color name. Or a feature for calculating a degree of similarity such as a color or a frequency component of an image.

  In step S81, the search unit 107 selects a search range of the main image 201 or the reduced image 202 recorded in the image holding unit 110 based on a signal from the input unit 49 according to a user operation. To do.

  In step S82, the search unit 107 acquires a search start instruction as a signal supplied from the input unit 49 according to the user's operation.

  In step S <b> 83, the search unit 107 sequentially reads the metadata 261 of the main image 201 or the reduced image 202 in the search range from the content database 111 or the similar feature database 112.

  In step S84, the search unit 107 determines whether or not the metadata 261 exists, that is, whether or not the metadata 261 is null, and if it is determined that the metadata 261 exists, In step S85, the search unit 107 generates search result display control data from the metadata 261.

  For example, in step S85, the distance calculation unit 121 of the search unit 107 selects a selected image (standard) based on metadata that is a vector indicating characteristics for calculating the degree of similarity such as a color or a frequency component of an image. The vector distance is calculated from the metadata that is the vector for the image and the metadata that is the vector for the image in the search range, and the search result display control data that is the vector distance is generated.

  The shorter the vector distance, the more similar the images are. Therefore, by using the search result display control data, which is the vector distance, a more similar image is read and the images are similar. They can be displayed in order.

  For example, in step S85, the search unit 107 compares the input threshold value with the relevance level based on metadata that is the relevance level indicating the degree recalled by a predetermined color name, and exceeds the input threshold value. Search result display control data indicating the relevance is generated.

  By using the search result display control data indicating that the degree of relevance is greater than or equal to the input threshold, only images with a high degree of recollection by the color name, that is, images containing many colors of the color name are read out. , Only an image including many colors of the color name can be displayed.

  Alternatively, for example, the search unit 107 calculates the distance between the input threshold value and the degree of association based on the metadata that is the degree of association indicating the degree recalled by a predetermined color name. Generate result display control data.

  By using the search result display control data, which is the distance between the input threshold value and the degree of association, an image including a desired color name in a desired amount is read, and the desired color name in a desired amount An image including it can be displayed.

  Note that the search result display control data includes the content ID, thereby identifying the main image 201 or the reduced image 202 corresponding to the search result display control data.

  In step S <b> 86, the search unit 107 stores the generated search result display control data in the search result holding unit 115.

  In step S87, the search unit 107 determines whether or not the processing of all the main images 201 or reduced images 202 in the search range has been completed, and performs the processing of all the main images 201 or reduced images 202 in the search range. When it is determined that the search has not ended, the process returns to step S83, and the search unit 107 reads the metadata 261 of the next main image 201 or reduced image 202 in the search range from the content database 111 or the similar feature database 112, The above processing is repeated.

  If it is determined in step S84 that the metadata 261 does not exist, that is, if it is determined that the metadata 261 is null, the process returns to step S83, and the search unit 107 selects the content database 111 or similar feature. The metadata 261 of the next main image 201 or reduced image 202 in the search range is read from the database 112, and the above-described processing is repeated.

  If it is determined in step S87 that the processing of all the main images 201 or reduced images 202 in the search range has been completed, the process proceeds to step S88, and the display control unit 106 controls the search result display control from the search result holding unit 115. Read data. In step S89, the display control unit 106 reads the main image 201 or the reduced image 202 from the image holding unit 110 based on the search result display control data, displays the main image 201 or the reduced image 202 on the monitor 40, and The process ends.

  For example, when search result display control data that is a vector distance indicating a feature for calculating the degree of similarity such as a color or a frequency component of an image is generated in step S85, the display control unit 106 in step S89 Then, the main image 201 or the reduced image 202 is displayed on the monitor 40 in the order similar to the reference image.

  For example, when search result display control data indicating that the relevance indicating the degree recalled by the predetermined color name is equal to or greater than the input threshold value is generated in step S85, the display control unit is determined in step S89. 106 displays the main image 201 or the reduced image 202 including many colors of the color name on the monitor 40.

  Further, for example, when search result display control data that is the distance between the degree of association indicating the degree recalled by the predetermined color name and the input threshold value is generated in step S85, the display control unit 106 in step S89. Causes the monitor 40 to display the main image 201 or the reduced image 202 including a desired amount of the color of the desired color name.

  The mobile phone 12 executes the same process as the search process described with reference to the flowchart of FIG. The server 13 executes processing similar to the search processing described with reference to the flowchart of FIG.

  As a result, as shown in FIG. 22, the search of the master image 201 based on the metadata 261 stored in the content database 141 and the similar feature database 142, for example, in the server 13-1 or the server 13-2. Similarly, in the digital still camera 11 or the mobile phone 12, the reduced image 202 can be searched based on the metadata 261 stored in the content database 111 and the similar feature database 112.

  Next, more specific search processing by the digital still camera 11 will be described.

  FIG. 23 is a flowchart illustrating another example of search processing by the digital still camera 11. In step S101, the display control unit 106 causes the monitor 40 to display the reduced image 202 in time series. That is, in step S <b> 101, the image holding unit 110 displays a reduced image 202 in a predetermined range according to a signal from the input unit 49 according to the user's operation among the recorded reduced images 202. To supply. Also, the content database 111 supplies the shooting control metadata to the display control unit 106 among the metadata 261 of the reduced image 202 within a predetermined range supplied to the display control unit 106. Then, the display control unit 106 causes the monitor 40 to display the reduced images 202 in time series in the order of shooting based on the shooting time.

  For example, as illustrated in FIG. 24, the display control unit 106 causes the monitor 40 to display the reduced images 202 in time series in the order of shooting for each group specified by the group ID. The square on the right side of FIG. 24 shows one reduced image 202, and the numbers in the squares indicate the order of shooting. That is, for example, the display control unit 106 causes the monitor 40 to display the reduced images 202 in the raster scan order in the order of shooting for each group.

  In step S101, the image holding unit 110 may display the clustered image on the monitor 40.

  Here, a case will be described as an example where images p1 to p12 photographed at respective timings of times t1 to t12 are targeted for clustering. For example, conditions A and B are set as conditions for defining clusters, and one of the images p1 to p12 is defined by condition A among them. Here, the condition A is a condition for defining a cluster having a low particle size (coarse), and the condition B is a condition for defining a cluster having a particle size higher than the condition A (fine). For example, the event name “wedding” is set in the cluster defined by the condition A.

  The cluster in which the event name “wedding” is set is defined, for example, because the degree of variation in the time interval of the shooting time of each of the images p1 to p12 is smaller than a certain threshold value. .

  Also, according to the condition B, one cluster is defined from the images p1 to p3 of the images p1 to p12, and one cluster is defined from the images p4 to p7. One cluster is defined from the images p8 to p12.

  “Church Ceremony” is set for the cluster composed of the images p1 to p3, “Reception” is set for the cluster composed of the images p4 to p7, and “Secondary Party” is set for the cluster composed of the images p8 to p12.

  In the cluster in which the event name “Church Ceremony” is set, the image p3 and the images p1 to p3 constituting the cluster have the same degree of variation in the time interval of the photographing time, whereas the image p3, Next (next on the time axis), the time interval with the image p4 which is the first image among the images p4 to p7 which is a group of images with a similar degree of variation in the time interval of the photographing time is relatively large. This is defined in part because it is determined that the frequency of shooting has changed.

  Also, in the cluster in which the event name “Reception Party” is set, the degree of variation in the time interval of the shooting time of each of the images p4 to p7 constituting the event is close, while the image p7 and the next In addition, the time interval with the image p8 which is the first image among the images p8 to p12, which is a group of images having a close variation in the time interval of the shooting time, is relatively large, and the frequency of shooting changes at that portion. It is stipulated because it was judged that there was.

  In the cluster in which the event name “secondary party” is set, the degree of variation in the time interval of the photographing time of each of the images p8 to p12 constituting the cluster is close, whereas the image p12 is photographed next. This is defined by the fact that the time interval with the first image in a group of images with similar time interval variations is relatively large, and it is determined that there has been a change in the frequency of shooting at that portion.

  The event names of “wedding”, “ceremony at church”, “banquet”, and “second party” are manually set by the user, for example.

  As described above, a plurality of conditions are set as conditions for clustering the same target image, and clusters having different granularities are defined based on the respective conditions.

  The images included in each cluster defined as described above are presented to the user in a form having a hierarchical structure.

  In step S101, the image holding unit 110 divides the display area for each date on the monitor 40, and sets the date of the divided area to the predetermined area so that the date when the image is captured matches. The reduced image 202 may be displayed. That is, in step S101, the image holding unit 110 may display the reduced image 202 by calendar display.

  In step S <b> 102, the search unit 107 selects one reduced image 202 from the reduced images 202 displayed on the monitor 40 based on a signal from the input unit 49 according to the user's operation.

  In this case, as shown in FIG. 24, when any one of the reduced images 202 displayed in time series is selected, the display control unit 106 highlights or selects the selected reduced image 202. The edge of the reduced image 202 is highlighted.

  In this case, as shown in FIG. 25, when any one of the reduced images 202 displayed in time series is selected, the display control unit 106 enlarges the selected reduced image 202 and displays it on the monitor 40. You may make it display.

  In step S103, the search unit 107 executes a process for searching for similar images.

  FIG. 26 is a flowchart for explaining the details of the similar image search processing corresponding to step S103. In step S131, the search unit 107 acquires a signal from the input unit 49 according to the user's operation, thereby performing a similar search by selecting an item of “similar search” in the menu displayed on the monitor 40. Get instructions.

  In step S132, the search unit 107 acquires a search start instruction by acquiring a signal from the input unit 49 according to the user's operation.

  In step S133, the search unit 107 reads a similar feature vector corresponding to the content ID of the reduced image 202 selected in step S102 from the similar feature database 112. Here, the similar feature vector is the color feature vector Cv or the frequency component vector Tv.

  In step S134, the search unit 107 reads a similar feature vector corresponding to the content ID of one reduced image 202 in the search range from the similar feature database 112.

  In this case, when the similar feature vector that is the color feature vector Cv is read in step S133, the similar feature vector that is the color feature vector Cv is read in step S134. When the similar feature vector that is the frequency component vector Tv is read in step S133, the similar feature vector that is the frequency component vector Tv is read in step S134.

  In step S135, the search unit 107 calculates the distance between the similar feature vector of the reduced image 202 in the search range and the similar feature vector of the selected reduced image 202.

  Here, the color feature vector Cv1 = {(c1_1, r1_1),..., (C32_1, r32_1)} and the color feature vector Cv2 = {(c1_2, r1_2),. c32_2, r32_2)} and distance as an example, the calculation of distance will be described.

  First, the concept of ground distance dij = d (c1i, c2j) is introduced. The ground distance dij represents the distance between the elements of the color feature vector. In this example, since the Euclidean distance between the two colors (distance in the triaxial space of L * a * b *), dij = ‖c1i−c2j It is expressed as ‖.

  Then, the EMD (Earth Movers Distance) between the color feature vector Cv1 and the color feature vector Cv2 associates the color feature vector Cv1 with the supply location, the color feature vector Cv2 with the demand location, and dij with the unit transportation cost, respectively. , Using the solution of the transport problem to determine the flow F = {Fji} from the color feature vector Cv1 to the color feature vector Cv2.

・・・（１）
このとき、

とされる。 That is, the EMD is obtained by the equation (1) by dividing the optimum value of the transportation problem (the minimum value of the total transportation cost) by the number of flows and normalizing it.

... (1)
At this time,

It is said.

  The EMD obtained by Expression (1) is the distance between the color feature vector Cv1 and the color feature vector Cv2.

  The distance of the frequency component vector Tv is obtained in the same manner as the distance of the color feature vector Cv.

・・・（２） Note that the weight Wc is determined with respect to the distance of the color feature vector Cv, and the weight Wt is determined with respect to the distance of the frequency component vector Tv, so that the final distance is obtained from the equation (2). Good.

... (2)

  Even if the user determines the weight Wc and the weight Wt, the weight Wc and the weight Wt may be fixed. For example, more specifically, the weight Wc and the weight Wt may be set to 0.5, respectively, and the final distance may be an average of the distance of the color feature vector Cv and the distance of the frequency component vector Tv.

  For vector distance calculation, Y. Rubner, C. Tomasi, and LJ Guibas.A Metric for Distributions with Applications to Image Databases.Proceedings of the 1998 IEEE International Conference on Computer Vision, Bombay, India, January 1998, pp. The example using EMD (Earth Movers Distance) described in 59-66 was explained, but it is not limited to this. For example, Euclidean distance and Hausdorff distance, as well as Norihiro Kobayakawa, Hoshimori, “Wavelet transform. "Interactive similar image search system used", "December issue of computer science magazine" (December 1, 1999), published by Kyoritsu Shuppan Co., Ltd., pages 30-41, Kuni Kim, Kunihiko Kaneko, Akifumi Makinouchi, Atsuko Ueno, “Design, Implementation and Performance Evaluation of Similar Image Retrieval System Based on Self-Organizing Feature Map”, “Technical Report of IEICE Vol.100 No.31”, (May 2000) 2), The Institute of Electronics, Information and Communication Engineers Line, may be used techniques such as those described in the literature such as page 9 to page 16.

  In step S <b> 136, the search unit 107 stores the distance in the similarity result database 113 in association with the image in the search range. For example, in step S136, the search unit 107 stores the distance in the similarity result database 113 together with the content ID of the image in the range to be searched.

  FIG. 27 is a diagram illustrating the metadata stored in the content database 111 and the similar feature database 112 and the structure of the distance stored in the similar result database 113.

  In FIG. 27, the database record 301-1 corresponds to the content item 281-1 and the content item 281-1, and the database record 301-2 corresponds to the content item 281-2 and the content item 281-2.

  That is, the database record 301-1 and the database record 301-2 each include a content ID, a similar feature vector, a path name and file name of the main image 201, a group ID, a shooting time, and other properties.

  The distance record 302 is stored in the similarity result database 113 and includes a content ID and a distance from the selected image. The distance record 302 is related to the database record 301-1 and the database record 301-2 by the content ID.

  Hereinafter, when it is not necessary to distinguish the database record 301-1 and the database record 301-2 from each other, they are simply referred to as the database record 301.

  The distance in the distance record 302 is a property that is distance.

  The time group record 303 is stored in the time group database 114, and includes a group ID unique to the group (for specifying the group) and an array of content IDs for specifying images belonging to the group specified by the group ID. Consists of. An array of content IDs in the time group record 303 is a property that is PhotoIdArray.

  As shown in FIG. 28, the records of the content database 111, the similarity result database 113, and the time group database 114 are related to each other. The content database 111 and the similar feature database 112 (not shown) store one or a plurality of database records 301, the similarity result database 113 stores one or a plurality of distance records 302, and the time group database 114 stores them. Stores one or more time group records 303.

  Returning to FIG. 26, in step S137, the search unit 107 determines whether or not the processing has been completed for all the images in the search range. If it is determined that the processing has not been completed, the processing returns to step S134. The similar feature vector corresponding to the content ID of the next reduced image 202 in the search range is read from the similar feature database 112, and the above-described processing is repeated.

  If it is determined in step S137 that the processing has been completed, the process proceeds to step S138, and the search unit 107 reads the distance associated with the image in the range to be searched from the similar feature database 112. For example, in step S138, the search unit 107 reads the distance from the similar feature database 112 together with the content ID that specifies the image in the search range.

  In step S139, the search unit 107 sorts the images in the search range in the order of similarity based on the distance read in step S138, and the process ends. For example, in step S139, the search unit 107 sorts the images in the search range in the order of similarity by sorting the content IDs that specify the images in the search range in the order of distance.

  Returning to FIG. 23, in step S104, the display control unit 106 causes the monitor 40 to display the reduced images 202 in the similar order. That is, in step S104, the display control unit 106 reads the reduced image 202 from the image holding unit 110, and causes the monitor 40 to display the reduced image 202 in the similar order sorted in step S139.

  For example, as illustrated in FIG. 29, the display control unit 106 causes the monitor 40 to display the reduced images 202 similar to the reduced image 202 selected in step S102 in the similar order. For example, the display control unit 106 displays the reduced image 202 (key image in FIG. 29) selected in step S102 at the upper left of the display area of the monitor 40, and a reduced image similar to the key image in the right area thereof. 202 are displayed in a raster scan order in a similar order. A square on the right side of FIG. 29 shows one reduced image 202, and alphabets in the square indicate a similar order.

  In step S <b> 105, the search unit 107 selects one reduced image 202 from the reduced images 202 displayed on the monitor 40 based on a signal from the input unit 49 according to the user's operation.

  For example, as shown in FIG. 29, when the reduced image 202 with the alphabet B is selected from the reduced images 202 displayed in the raster scan order in a similar order on the monitor 40, the selected image is selected. The reduced image 202 is highlighted or the edge is highlighted, and the display control unit 106 enlarges and displays the selected reduced image 202 below the key image in the display area of the monitor 40.

  In step S106, the search unit 107 determines whether or not to cancel based on the signal from the input unit 49 according to the user's operation. If it is determined not to cancel, the search unit 107 proceeds to step S107. It is determined whether or not to decide.

  In step S107, if it is determined to be determined, the process proceeds to step S108, and the search unit 107 acquires the group ID of the selected reduced image 202 from the content database 111 in step S105. That is, the search unit 107 reads out the metadata 261 specified by the content ID of the selected reduced image 202 from the content database 111 in the process of step S105, and is selected from the read metadata 261. By extracting the group ID that identifies the group to which the reduced image 202 belongs, the group ID of the selected reduced image 202 is acquired.

  In step S <b> 109, the search unit 107 reads out the reduced image 202 belonging to the group specified by the acquired group ID from the image holding unit 110. More specifically, the search unit 107 searches the time group record 303 in the time group database 114 with the acquired group ID. The search unit 107 reads from the time group database 114 an array of content IDs that specify images belonging to the group specified by the group ID from the time group record 303 including the same group ID as the acquired group ID. Then, the search unit 107 reads the reduced image 202 specified by the content ID that is an element of the read content ID array from the image holding unit 110. The search unit 107 supplies the read reduced image 202 to the display control unit 106.

  In step S110, the display control unit 106 causes the monitor 40 to display the read reduced image 202 in time series, and the process ends.

  In step S110, the display control unit 106 may display the clustered image on the monitor 40, or may display the reduced image 202 by calendar display.

  If it is determined in step S107 that the determination is to be made, the process returns to step S104 and the above-described processing is repeated.

  If it is determined in step S106 to cancel, the process returns to step S101 and the above-described processing is repeated.

  In the processing from step S101 to step S110, the image selection state is maintained until the next image is selected in step S102 or step S105. In step S101, step S104, or step S110, the image is displayed and the edge of the selected image is highlighted so that the user can identify the selected image. A selection is shown.

  That is, the state transitions between the time-series display state and the display state in the similar order while maintaining the image selection state.

  In this way, an image taken at a time close to the time when an image similar to the predetermined image is taken is immediately displayed, or an image taken at a time close to the time when the predetermined image is taken. An image similar to can be displayed immediately. Further, it is possible to search for an image so that the images are traced in order depending on whether the images are similar or taken at a close time.

  Even in the digital still camera 11 with a small display screen, by combining the search of the time axis and the similarity search effectively, the concept of similarity between images and the concept of time, which are dominant elements of human memory, It is possible to search and browse images according to the conditions.

  In addition, the distance indicating similarity is merely a similarity based on a statistical method, search omissions occur, and images that are considered to be similar according to human sense may not be searched. Even if a search failure occurs, a list of images of events that are close to each other is displayed, so that it is possible to reach an image that can be regarded as similar from a human perspective.

  If images of events and events that are repeated every year, such as images of cherry blossoms, images of fireworks, and images of barbecue, are taken every year, similar images are searched and displayed in chronological order. Since it can be changed, images of similar events (events) can be displayed in chronological order, and can be used as an album for recalling memories.

  The digital still camera 11 may search for the main image 201 by the processing shown in the flowchart of FIG.

  According to the search processing in the flowchart of FIG. 23, for example, as shown in the upper side of FIG. 30, first, the reduced image 202 is displayed on the monitor 40 in time series for each group. For example, when the reduced image 202 (key image) to which the alphabet A is added is selected from the reduced images 202 displayed in time series, the edges of the reduced image 202 to which the alphabet A is added are emphasized. Displayed.

  When the reduced image 202 (key image) to which the alphabet A is added is selected and a similar image search process is executed, the reduced image 202 similar to the reduced image 202 to which the alphabet A is added is searched. Then, they are displayed on the monitor 40 in a similar order.

  In this case, the key image which is the reduced image 202 to which the alphabet A is added is enlarged and displayed on the monitor 40.

  When the reduced image 202 to which the alphabet B is added is selected from the reduced images 202 displayed in a similar order, the key image that is the reduced image 202 to which the alphabet B is added is enlarged on the monitor 40. Displayed.

  When the reduced image 202 similar to the reduced image 202 to which the alphabet A is added is displayed on the monitor 40 in the order of similarity, when canceled, the state returns to the state of displaying the reduced image 202 in time series.

  When the reduced image 202 to which the alphabet B is added is selected from the reduced images 202 displayed in a similar order and the determination key is pressed, the reduced image 202 to which the alphabet B is added belongs to the group to which the alphabet belongs. The reduced image 202 to which it belongs is displayed on the monitor 40 in time series. In this case, the edge of the reduced image 202 to which the alphabet B is added is highlighted.

  When the reduced images 202 are grouped according to the shooting date, the reduced image 202 having a date close to the date when the reduced image 202 to which the alphabet B is added is recorded on the monitor 40 in time series. Is displayed.

  Next, search processing in the server 13 will be described. FIG. 31 is a flowchart for explaining search processing by the server 13. In step S161, the display control unit 136 of the server 13 causes the output unit 77, which is a display, to display the main image 201 in time series. That is, in step S161, the image holding unit 140 displays the main image 201 in a predetermined range according to the signal from the input unit 76 according to the user's operation among the recorded main images 201. To supply. In addition, the content database 141 supplies the shooting control metadata to the display control unit 136 among the metadata 261 of the main image 201 within a predetermined range supplied to the display control unit 136. Then, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 in a time series in the order of shooting based on the shooting time.

  For example, as shown on the right side of FIG. 32, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 in time series in the order of shooting (displayed in time axis). For example, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 in the order in which the images were captured for each group.

  In step S162, the search unit 137 selects one master image 201 from the master images 201 displayed on the output unit 77, which is a display, based on a signal from the input unit 76 according to the user's operation. select.

  In step S163, the search unit 137 executes a process for searching for similar images. The similar image search process in step S163 is executed by the search unit 137 instead of the search unit 107, but the other points are the same as the process described with reference to the flowchart of FIG. The detailed explanation is omitted.

  In step S164, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 in a similar order. That is, in step S164, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 in the sorted similar order.

  For example, as shown on the left side of FIG. 32, the display control unit 136 causes the output unit 77, which is a display, to display the main image 201 similar to the main image 201 selected in step S162 in the order of similarity.

  In step S165, the search unit 137 selects one master image 201 from the master images 201 displayed on the output unit 77, which is a display, based on a signal from the input unit 49 according to the user's operation. select.

  In step S166, the search unit 137 determines whether to display in time series based on the signal from the input unit 49 according to the operation of the user. For example, the search unit 137 determines whether or not to display in time series based on a signal from the input unit 76 in response to a click on the switching button 351 or the switching button 352 displayed on the output unit 77 that is a display. judge.

  For example, when the switch button 351 displayed on the output unit 77 that is a display and instructing display in time series order is clicked, in step S166, it is determined to display in time series. If so, the procedure proceeds to step S167.

  In step S167, the search unit 137 acquires the group ID of the selected master image 201 from the content database 141. That is, the search unit 137 reads the metadata 261 specified by the content ID of the selected master image 201 from the content database 141, and the group to which the selected master image 201 belongs from the read metadata 261. The group ID of the selected master image 201 is acquired by extracting the group ID that identifies

  In step S168, the search unit 137 reads the main image 201 belonging to the group specified by the acquired group ID from the image holding unit 140. More specifically, the search unit 137 searches the time group record 303 in the time group database 144 with the acquired group ID. The search unit 137 reads from the time group database 144 an array of content IDs that specify images belonging to the group specified by the group ID from the time group record 303 including the same group ID as the acquired group ID. Then, the search unit 137 reads the main image 201 specified by the content ID that is an element of the read content ID array from the image holding unit 140. The search unit 137 supplies the read main image 201 to the display control unit 136.

  In step S169, the display control unit 136 causes the output unit 77, which is a display, to display the read main image 201 in time series. For example, in step S169, the display control unit 136 causes the output unit 77, which is a display, to display the read main image 201 in time series for each group.

  In step S170, the search unit 137 selects one master image 201 from the master images 201 displayed on the output unit 77, which is a display, based on a signal from the input unit 76 according to the user's operation. select.

  In step S171, the search unit 137 determines whether or not to display in time series based on the signal from the input unit 49 according to the user's operation. For example, the search unit 137 determines whether or not to display in time series based on a signal from the input unit 76 in response to a click on the switching button 351 or the switching button 352 displayed on the output unit 77 that is a display. judge.

  For example, when the switching button 352 for instructing the display in the similar order displayed on the output unit 77, which is a display, is clicked, since it is determined in step S171 that the display is performed in the similar order, it is determined that the display is performed in time series. If so, the procedure returns to step S163 and repeats the above-described processing.

  Further, for example, when the switch button 351 displayed on the output unit 77 that is a display and instructing the display in time series is clicked, it is determined in step S171 that the images are not displayed in the similar order. When it is determined not to do so, the procedure returns to step S167 and repeats the above-described processing.

  In step S166, for example, when the switching button 352 displayed on the output unit 77 that is a display and instructing the display in the order of similarity is clicked, it is determined that the display is not performed in time series, so the procedure is step S163. Returning to FIG.

  Thus, for example, according to the click of the switching button 351 or the switching button 352 displayed on the output unit 77 that is a display, the display in the similar order and the time-series display can be arbitrarily switched.

  Next, the extraction of the degree of association in the server 13 will be described.

  The digital still camera 11, the mobile phone 12, and the server 13 search for an image using the color name and the degree of association with the color name as image features. The server 13 extracts the degree of association with a predetermined color name from the image as one of the features of the image.

  Here, the degree of association with the color name means the degree to which an image is recalled by a specific color name. In other words, the degree of association refers to a ratio in which a color that can be assumed to be a specific color name is included in a certain image.

  Here, the color names are, for example, red, blue, yellow, white, black, green, and the like.

  FIG. 33 is a block diagram illustrating an example of the configuration of the color feature extraction unit 172 that extracts the degree of association with a color name. The color feature extraction unit 172 includes an image input unit 401, a “red” association degree extraction unit 402, a “blue” association degree extraction unit 403, a “yellow” association degree extraction unit 404, and an extraction feature recording unit 405.

  Note that the “red” relevance extraction unit 402, the “blue” relevance extraction unit 403, and the “yellow” relevance extraction unit 404 are examples, and an arbitrary number of relevances for extracting relevance for an arbitrary color. A degree extracting unit is provided. That is, the relevance degree extraction unit is prepared for each color name.

  Hereinafter, a case where a “red” association degree extraction unit 402, a “blue” association degree extraction unit 403, and a “yellow” association degree extraction unit 404 are provided will be described as an example.

  The image input unit 401 acquires, from the image holding unit 140, the main image 201 that is a target for extracting the degree of association. In addition, the image input unit 401 receives the color name, the “red” association degree extraction unit 402, the “blue” association degree extraction unit 403, or the “yellow” association degree extraction unit 404 from the association degree extraction unit correspondence holding unit 145. Correspondence information indicating the correspondence of is acquired.

  As shown in the example of FIG. 34, the association information recorded in the association degree extraction unit correspondence holding unit 145 includes a “red” association degree extraction unit 402 that extracts a color name and a degree of association with the color name. Information for specifying either the “blue” relevance extraction unit 403 or the “yellow” relevance extraction unit 404 is arranged. For example, in the example of the correspondence information shown in FIG. 34, the correspondence between the color name “red” and the “red” association degree extraction unit 402 is shown, and the color name “blue” and the association “blue”. The correspondence with the degree extraction unit 403 is shown, and the correspondence between the color name “yellow” and the “yellow” association degree extraction unit 404 is shown.

  Based on the correspondence information, the image input unit 401 converts the main image 201 acquired from the image holding unit 140 into a “red” association degree extraction unit 402, a “blue” association degree extraction unit 403, and a “yellow” association degree extraction unit. 404 is supplied.

  The “red” relevance degree extraction unit 402 extracts, from the main image 201 supplied from the image input unit 401, a relevance degree indicating the degree to which the main image 201 is recalled by a color name that is red. The “red” association degree extraction unit 402 supplies the extraction feature recording unit 405 with the association degree that is extracted from the main image 201 and indicates the degree that is recalled by the color name that is red.

  The “blue” relevance extraction unit 403 extracts, from the main image 201 supplied from the image input unit 401, a relevance indicating the degree to which the main image 201 is recalled by a blue color name. The “blue” association degree extraction unit 403 supplies the extraction feature recording unit 405 with the association degree that is extracted from the main image 201 and indicates the degree that is recalled by the blue color name.

  The “yellow” relevance degree extraction unit 404 extracts, from the main image 201 supplied from the image input unit 401, a relevance degree indicating the degree to which the main image 201 is recalled by a color name that is yellow. The “yellow” association degree extraction unit 404 supplies the extraction feature recording unit 405 with the association degree that is extracted from the main image 201 and indicates the degree that is recalled by the color name that is yellow.

  The extracted feature recording unit 405 is recalled by the color name that is red supplied from each of the “red” association degree extraction unit 402, the “blue” association degree extraction unit 403, and the “yellow” association degree extraction unit 404. The extracted feature storage unit associates the relevance level indicating the level, the relevance level indicating the level recalled by the blue color name, and the relevance level indicating the level recalled by the yellow color name with the main image 201. 146 to record.

  For example, in this case, as illustrated in FIG. 35, the extraction feature holding unit 146 has a content ID that identifies the master image 201 and a relevance level that indicates the degree recalled by the color name that is red, and a color name that is blue The degree of association indicating the degree recalled by the color name and the degree of association indicating the degree recalled by the color name being yellow are recorded.

  In the above-described example, an example in which the main image 201 recorded in the image holding unit 140 is input from the image input unit 401 is shown. However, the image is not limited to the main image 201, and the reduced image 202 or the reduced color image 221 is displayed. May be input to the reduced image 202 or the color-reduced image 221. Further, instead of the image, the above-described color histogram associated with the image for which each degree of association is to be extracted is input from the image input unit 401, and each degree of association extraction unit (for example, “red” degree of association extraction unit) is input. 402, the “blue” relevance extraction unit 403, and the “yellow” relevance extraction unit 404) may extract each relevance from the color histogram.

  FIG. 35 is a diagram illustrating a logical structure of the degree of association recorded in the extracted feature holding unit 146. In the example shown in FIG. 35, the extraction feature holding unit 146 corresponds to the content ID “000” and is extracted from the main image 201 identified by the content ID “000”, which is 0.80 in red. Relevance indicating the degree recalled by a color name, 0.00, relevance indicating the degree recalled by a blue color name, and the degree recalled by a color name yellow of 0.10 Record the relevance indicating. In addition, the extracted feature storage unit 146 is recalled by a color name of red, which is 0.00, extracted from the main image 201 identified by the content ID of 001, corresponding to the content ID of 001. The degree of association indicating the degree of color, the degree of association indicating 0.25, the degree of recalling by the color name of blue, and the degree of association indicating the degree of recall of the color name being 0.20 are recorded. To do. Further, the extracted feature holding unit 146 is recalled by the color name of red, which is 0.15, extracted from the main image 201 identified by the content ID of 002, corresponding to the content ID of 002. Relevance indicating the degree to which the color name is 0.05, relevance indicating the degree that is recalled by the color name that is blue, and relevance indicating the degree that is recalled by the color name that is 0.00 that is yellow To do.

  The extracted feature recording unit 405 is recalled by the color name that is red supplied from each of the “red” association degree extraction unit 402, the “blue” association degree extraction unit 403, and the “yellow” association degree extraction unit 404. Are related to the main image 201 as metadata 261. The association degree indicating the degree recalled by the blue color name and the association degree indicating the degree recalled by the yellow color name are associated with the master image 201 as metadata 261. And recorded in the similar feature database 142.

  The degree of association may be stored in a predetermined area of the main image 201 that is EXIF data.

  The search unit 137 searches the main image 201 using the color name and the degree of association with the color name as a feature of the main image 201. In this case, for example, the search unit 137 includes a search condition input unit 421 and a condition matching unit 422.

  The search condition input unit 421 inputs a search condition for the degree of relevance based on a signal from the input unit 76 according to the user's operation. The search condition input unit 421 supplies the search condition for the degree of relevance to the condition matching unit 422.

  The condition collation unit 422 collates the search condition supplied from the search condition input unit 421 with the relevance recorded in the extracted feature holding unit 146. The condition matching unit 422 stores, in the search result holding unit 147, the content ID corresponding to the degree of relevance that satisfies the search condition as a result of the matching.

  FIG. 36 is a flowchart for explaining the details of the color feature extraction processing corresponding to step S43. In step S <b> 201, the image input unit 401 inputs the main image 201, which is an image whose relevance is to be extracted, from the image holding unit 140. Further, the image input unit 401 inputs correspondence information from the association degree extraction unit correspondence holding unit 145.

  In step S202, the image input unit 401 inputs a color name. In step S203, the image input unit 401, based on the correspondence information, the “red” association degree extraction unit 402, the “blue” association degree extraction unit 403, or the “yellow” association degree extraction unit corresponding to the input color name. One of 404 is specified.

  For example, in step S203, when the color name “red” is input in step S202, the image input unit 401 specifies the “red” association degree extraction unit 402 based on the correspondence information.

  The image input unit 401 supplies the input main image 201 to any one of the identified “red” relevance extraction unit 402, “blue” relevance extraction unit 403, and “yellow” relevance extraction unit 404.

  In step S204, any one of the “red” relevance extraction unit 402, the “blue” relevance extraction unit 403, and the “yellow” relevance extraction unit 404 specified in step S203 executes the relevance extraction process. . Details of the association degree extraction process will be described later.

  The extracted degree of association is supplied to the extracted feature recording unit 405.

  In step S205, the extracted feature recording unit 405 causes the extracted feature holding unit 146 to record the extracted degree of association as a color feature vector in association with the main image 201 from which the degree of association is extracted.

  In step S206, the image input unit 401 determines whether or not the color name is the end, that is, whether or not all the color names have been extracted from the main image 201, and determines that the color name is not the end. If the color name has not been extracted, there is a degree of association with the color name that has not yet been extracted. Therefore, the process returns to step S202, the next color name is input, and the above-described processing is repeated.

  If it is determined in step S206 that the color name is the end, that is, it is determined that the relevance level has been extracted from the main image 201 for all color names, the process ends.

  FIG. 37 is a flowchart for explaining an example of details of the relevance level extraction process when the “red” relevance level extraction unit 402 is identified in step S203, corresponding to step S204 of FIG.

  In step S221, the “red” association degree extraction unit 402 clears the built-in counter. In step S <b> 222 that is first executed, the “red” association degree extraction unit 402 inputs the color of the first pixel among the pixels of the main image 201, that is, the pixel value. In step S223, the “red” association degree extraction unit 402 calculates a position in the color space corresponding to the color of the pixel.

  In step S224, the “red” association degree extraction unit 402 determines whether or not the calculated position on the color space is in the subspace corresponding to the color name that is red.

  Here, the position in the color space that is calculated corresponding to the color of the pixel will be described.

  For example, the pixel value of each pixel of the main image 201 is expressed in RGB. In this case, the pixel value includes an R value, a G value, and a B value. The RGB color space is a space in which the R axis, the G axis, and the B axis are orthogonal to each other, as shown in FIG. One position in the RGB color space is determined by one pixel value.

  In the RGB color space, it is difficult to represent the position of a color that a human recognizes as a color of a predetermined color name in one area (it is difficult to express).

  Therefore, consider representing the color of a pixel at a position in the L * a * b * space. As shown in FIG. 39, the L * a * b * space is expressed by the L * axis, the a * axis, and the b * axis that are orthogonal to each other. In the L * a * b * space, the luminance increases as L * that is the value in the L * axis direction increases, and the luminance decreases as L * decreases. When L * is constant, the saturation decreases as it approaches the L * axis.

  One pixel value determines one position in the L * a * b * space.

  In the L * a * b * space, the position of a color that a human recognizes as a color having a predetermined color name can be expressed by one area. A region including a color position that a human recognizes as a color having a predetermined color name is referred to as a subspace. The subspace is, for example, a region having a spread in the L * a * b * space.

  First, examples of subspaces for white and black will be described.

  FIG. 40 is a diagram illustrating an example of a white subspace and a black subspace. The white subspace 441 is an ellipse in which one axis of the ellipsoid coincides with the L * axis, and the graphic center is the uppermost position in the L * a * b * space (the maximum value on the L * axis). The space inside the ellipse that coincides with the L * a * b * space. The white sub-space 441 is a space showing a color with low saturation and high luminance. The color indicated by the position in the sub space 441 is recognized as white by humans.

  The black subspace 442 is an ellipsoid in which one axis of the ellipsoid coincides with the L * axis, and the graphic center is the lowest position in the L * a * b * space (the minimum value on the L * axis). The space inside the ellipse that coincides with the L * a * b * space. The black sub-space 442 is a space that shows a color with low saturation and low luminance. The color indicated by the position in the sub space 442 is recognized as black by humans.

  Next, examples of subspaces for red, yellow, green, and blue will be described.

  Since red, yellow, green, and blue are chromatic colors, from the L * a * b * space, the region inside the saturation boundary 461 shown in FIG. 41, the region below the luminance lower limit boundary 462, and the luminance upper limit The region above the boundary 463 is excluded. A region inside the saturation boundary 461 indicates a color with low saturation. The saturation boundary 461 is provided at a position where the saturation of the color indicated by the inner region is low and the color is not recognized by humans as red, yellow, green, or blue.

  The region below the lower luminance limit boundary 462 indicates a color with low luminance. The luminance lower limit boundary 462 is provided at a position where the luminance of the color indicated in the lower region is low and the color is not recognized by humans as red, yellow, green, or blue.

  A region above the luminance upper limit boundary 463 indicates a color with high luminance. The luminance upper limit boundary 463 is provided at a position where the luminance of the color indicated by the upper region is high and the color is not recognized by humans as red, yellow, green, or blue.

  Accordingly, a space obtained by excluding the area inside the saturation boundary 461, the area below the luminance lower limit boundary 462, and the area above the luminance upper limit boundary 463 from the L * a * b * space is indicated by the space. The color consists of positions that are recognized by humans as red, yellow, green, or blue.

  FIG. 42 shows a space obtained by excluding the area inside the saturation boundary 461, the area below the luminance lower limit boundary 462, and the area above the luminance upper limit boundary 463 from the L * a * b * space. As described above, it is perpendicular to the plane composed of the a * axis and the b * axis, and is divided at a radial boundary centered on the L * axis. For example, when the L * a * b * space is viewed from the upper side of the L * axis, the green subspace 481 includes the upper boundary of the negative a * axis and the lower boundary of the negative a * axis. The space on the a * axis side surrounded by The color indicated by the position in the sub space 481 is recognized as green by humans.

  Further, when the L * a * b * space is viewed from the upper side of the L * axis, the blue subspace 482 includes a boundary on the right side of the negative b * axis and a boundary on the left side of the negative b * axis. The space on the b * axis side surrounded by. The color indicated by the position in the subspace 482 is recognized by humans as blue.

  Similarly, for example, when the L * a * b * space is viewed from the upper side of the L * axis, the red subspace 483 includes the upper boundary of the positive a * axis and the lower side of the positive a * axis. The space on the a * axis side surrounded by the boundary on the side. The color indicated by the position in the subspace 483 is recognized as red by humans. For example, when the L * a * b * space is viewed from the upper side of the L * axis, the yellow subspace 484 includes the right boundary of the positive b * axis and the left boundary of the positive b * axis. The space on the b * axis side surrounded by. The color indicated by the position in the subspace 484 is recognized by humans as yellow.

  That is, in step S223, the “red” association degree extraction unit 402 calculates a position in the L * a * b * space corresponding to the color of the pixel. In step S224, the “red” association degree extraction unit 402 determines whether the calculated position in the L * a * b * space is in the subspace 483 corresponding to the color name that is red. To do. That is, in step S224, the “red” relevance extraction unit 402 determines whether or not the pixel color is a color that is recognized by humans as red.

  If it is determined in step S224 that the calculated position in the L * a * b * space is within the subspace 483 corresponding to the color name that is red, the pixel color is recognized as red by humans. In step S225, the “red” relevance extraction unit 402 increments the counter by 1, and the procedure proceeds to step S226.

  If it is determined in step S224 that the calculated position in the L * a * b * space is not within the subspace 483 corresponding to the color name that is red, the color of the pixel is not recognized by human beings as red. Since it is a color, skip step S225 and do not increment the counter, and the procedure proceeds to step S226.

  In step S226, the “red” association degree extraction unit 402 determines whether or not the pixel is the end, that is, whether or not the process has been applied to all the pixels of the main image 201, and determines that the pixel is not the end. If YES in step S222, the color of the next pixel among the pixels of the main image 201, that is, the pixel value of the next pixel is input, and the above-described processing is repeated.

  If it is determined in step S226 that the pixel is the end, that is, it is determined that the processing has been applied to all the pixels of the main image 201, the process proceeds to step S227, and the “red” relevance extraction unit 402 determines the number of counters (value ) Is divided by the number of pixels of the main image 201. As a result, in the main image 201, a ratio in which a color that can be assumed to be red is included is obtained.

  In step S228, the “red” relevance degree extraction unit 402 sets the result of division as red relevance, supplies the relevance degree to the extracted feature recording unit 405, and ends the process.

  The subspace in the L * a * b * space has been described as an example. However, the color space is not limited to the L * a * b * space, and a color space in which a color of a predetermined color name is expressed by one region, The degree of association may be obtained based on the subspace.

  In the relevance extraction process described with reference to FIG. 37, a binary determination is made as to whether or not the color for each pixel is inside the subspace. It may be possible to reflect the degree of relevance whether it is close to the boundary (whether it is just the boundary).

  Next, the association degree extraction process in this case will be described.

  FIG. 43 is a flowchart for explaining another example of the details of the relevance level extraction process when the “red” relevance level extraction unit 402 is identified in step S203, corresponding to step S204 of FIG. In step S241, the “red” association degree extraction unit 402 clears the stored association degree. In step S <b> 242 that is first executed, the “red” association degree extraction unit 402 inputs the color of the first pixel among the pixels of the main image 201, that is, the pixel value. In step S243, the “red” association degree extraction unit 402 calculates a position in the color space corresponding to the color of the pixel.

  In step S224, the “red” association degree extraction unit 402 calculates a certainty factor belonging to the subspace corresponding to the color name for the calculated position in the color space. That is, in step S224, the “red” association degree extraction unit 402 calculates a certainty factor belonging to the subspace 483 corresponding to the color name that is red for the calculated position in the color space.

  The certainty factor is an index value that continuously changes from 1 to 0 from the inner side to the outer side of the sub space, indicating whether it is close to the center of the sub space or the boundary of the sub space.

  For example, in step S224, if the calculated position in the color space is closer to the center of the subspace 483, the “red” association degree extraction unit 402 calculates a certainty factor closer to 1, and calculates the calculated color space. When the position is closer to the boundary of the subspace 483, a certainty factor closer to 0 is calculated.

  In step S245, the “red” association degree extraction unit 402 adds the certainty factor to the association degree. In step S246, the “red” association degree extraction unit 402 determines whether or not the pixel is the end, that is, whether or not the process has been applied to all the pixels of the main image 201, and determines that the pixel is not the end. If YES in step S242, the color of the next pixel among the pixels of the main image 201, that is, the pixel value of the next pixel is input, and the above-described processing is repeated.

  If it is determined in step S226 that the pixel is the end, that is, it is determined that the process has been applied to all the pixels of the main image 201, the degree of association is supplied to the extraction feature recording unit 405, and the process is performed. finish.

  When the degree of relevance is calculated based on the certainty level, the degree of relevance closer to the human sense can be obtained. In particular, even when the image includes many colors close to the boundary of the subspace, a more accurate degree of association can be obtained.

  The process of step S224 in the relevance extraction process described with reference to FIG. 37 is a two-class classification problem as to whether or not a pixel color is determined to be a color of a specific color name, and various pattern recognition methods. Can be replaced.

  Next, the association degree extraction process in this case will be described.

  FIG. 44 is a flowchart for explaining another example of the details of the relevance level extraction process corresponding to step S204 of FIG. 36 when the “red” relevance level extraction unit 402 is specified in step S203. Since the processing of step S261 and step S262 is the same as the processing of step S221 and step S222 of FIG. 37, respectively, description thereof is omitted.

  In step S263, the “red” association degree extraction unit 402 recognizes the color of the pixel.

  For example, in step S <b> 263, the “red” association degree extraction unit 402 recognizes the color of the pixel using a neural network. Pattern recognition using a neural network is described in, for example, Junichiro Toriwaki, recognition engineering-pattern recognition and its application, Corona Co., etc.

  In the case of pattern recognition, a plurality of judgment data indicating whether or not the color of a specific color value (L *, a *, b *) is a color of a specific color name is collected in advance by hand, and the collected judgment Based on the data, neural network learning is performed to generate parameters necessary for identification.

  FIG. 45 is an example of determination data indicating whether the color is blue. In the example of the determination data in FIG. 45, for example, the color specified by L * being 0.02, a * being 0.04, and b * being 0.10 is not blue, but 0.72. The color specified by a certain L *, a * that is 0.00, and b * that is 0.12 is blue, L * that is 0.28, a * that is −0.02, and 0. .15 indicates that the color specified by b * is not blue.

  According to the neural network, it is determined whether the color of a pixel is a color of a specific color name according to the parameters generated in this way.

  The pattern recognition method may be any method that can determine whether the color of a pixel is a color of a predetermined color name, and may be any method such as SVM (Support Vector Machine).

  In step S264, the “red” association degree extraction unit 402 determines whether the pixel color belongs to red as a result of recognition. When it is determined in step S224 that the color of the pixel belongs to red, the process proceeds to step S265, the “red” relevance extraction unit 402 increments the counter by 1, and the procedure proceeds to step S266.

  If it is determined in step S264 that the color of the pixel does not belong to red, step S265 is skipped and the procedure proceeds to step S266 without incrementing the counter.

  Since the processing from step S266 to step S268 is the same as the processing from step S226 to step S228 in FIG. 37, description thereof will be omitted.

  Further, the certainty factor may be obtained by a pattern recognition technique.

  FIG. 46 is a flowchart for explaining another example of the degree of association degree extraction process corresponding to step S204 of FIG. 36 when the “red” degree of association extraction unit 402 is identified in step S203. Since the process of step S281 is the same as the process of step S241 of FIG. 43, the description thereof is omitted. The processes in step S282 and step S283 are the same as the processes in step S262 and step S263 of FIG.

  In step S284, the “red” association degree extraction unit 402 calculates a certainty factor for determining that the color name belongs as a recognition result. That is, in step S284, the “red” association degree extraction unit 402 calculates a certainty factor for determining that the color of the pixel belongs to red as a result of recognition. For example, a value input to the output layer of the neural network can be used as the certainty factor.

  Since the processing of step S285 and step S286 is the same as the processing of step S245 and step S246 of FIG. 43, the description thereof is omitted.

  Note that, when “blue” relevance extraction unit 403 is identified in step S203, or when “yellow” relevance extraction unit 404 is identified in step S203, corresponding relevance extraction processing corresponding to step S204 in FIG. The details of are different in that the “blue” relevance degree extraction unit 403 or the “yellow” relevance degree extraction unit 404 executes processing or subspace instead of the “red” relevance degree extraction part 402, but other points are The processing is the same as that described with reference to FIG. 37, FIG. 43, FIG. 44, or FIG.

  FIG. 47 is a flowchart for explaining search processing. In step S <b> 311, the search condition input unit 421 acquires a search condition for the degree of association based on a signal from the input unit 76 according to the user's operation. The search condition input unit 421 supplies the search condition for the degree of relevance to the condition matching unit 422.

  For example, as shown in FIG. 48, a GUI (Graphical User Interface) image is displayed on the output unit 77 which is a display. In the example shown in FIG. 48, the slide bar 491 operated by the user specifies the granularity (threshold value) for each color name, which is a search condition. When the check box 492 corresponding to the color name is checked by the user, the granularity for the color name specified by the slide bar 491 of the color name is acquired as a search condition in step S311.

  For example, when a black check box 492, a red check box 492, and a green check box 492 are checked, the black granularity specified by the black slide bar 491, the red slide bar 491, In step S311, the red granularity and the green granularity designated by the green slide bar 491 are acquired as search conditions.

  When the AND search radio button 493 is turned on, the logical product of the granularity for each color name specified by the slide bar 491 is the final search condition, and the OR search radio button 494 is turned on. The logical sum of the granularity for each color name designated by the slide bar 491 is the final search condition.

  More specifically, for example, in step S <b> 311, the search condition input unit 421 determines that (“red”> 0.5) AND (“blue” ≧ 0.3) AND (“green” <0.1) or the like. The search condition indicated by the logical expression for a plurality of color names is acquired.

  For example, when the user wants to search for an image showing a blue sky, the user inputs a search condition of “blue” ≧ 0.3. In step S311, the search condition input unit 421 reads “blue” ≧ 0.3. Get the search condition that is.

  For example, when the user wants to search for an image of strawberry picking, the user inputs a search condition of (“red”> 0.1) AND (“green” ≧ 0.3), and in step S311 The search condition input unit 421 acquires search conditions that are (“red”> 0.1) AND (“green” ≧ 0.3).

  It should be noted that the color names in the search condition need not be all defined color names (relationship extraction units are prepared), that is, the color names in the search condition are defined. It may be a part of the color name or one color name.

  In addition, a numerical value may be directly input and acquired for each color name.

  In step S <b> 312, the condition matching unit 422 acquires the color feature vector of the main image 201 to be searched from the extracted feature holding unit 146.

  In step S313, the condition matching unit 422 determines whether or not the acquired color feature vector matches the search condition. For example, in step S313, the condition matching unit 422 selects the color name element corresponding to the check box 492 that is checked and the color specified by the slide bar 491 from among the elements of the acquired color feature vector. The granularity of the name is compared, and if the element of the color name of the color feature vector is equal to or greater than the designated granularity, it is determined that the color feature vector matches the search condition.

  For example, when the logical product of the granularity for each color name is the final search condition, in step S313, the condition matching unit 422 determines all the elements of the color name corresponding to the checked check box 492. When the element of the color name of the color feature vector is equal to or greater than the specified granularity, it is determined that the color feature vector matches the search condition. For example, when the logical sum of the granularity for each color name is the final search condition, in step S313, the condition matching unit 422 determines whether any of the color name elements corresponding to the check box 492 being checked. If the color name element of the color feature vector is equal to or greater than the specified granularity, it is determined that the color feature vector matches the search condition.

  If it is determined in step S313 that the acquired color feature vector matches the search condition, the process proceeds to step S314, and the condition matching unit 422 corresponds to the color feature vector acquired in step S312 in the search result holding unit 147. A content ID for specifying the main image 201 is added, and the process proceeds to step S315.

  If it is determined in step S313 that the acquired color feature vector does not match the search condition, the process of step S314 is skipped, and the process proceeds to step S315 without adding the content ID to the search result holding unit 147.

  In step S315, the search condition input unit 421 determines whether or not the image is the end, that is, whether or not all the images have been searched, and the image is not the end, that is, has searched for all the images yet. If it is determined that it is not, the process returns to step S312 to acquire the color feature vector of the next main image 201, and the above-described processing is repeated.

  If it is determined in step S315 that the image is the end, that is, all the images have been searched, the process ends.

  As a result of this processing, the search result holding unit 147 stores the content ID that identifies the main image 201 that satisfies the search conditions.

  FIG. 49 is a diagram illustrating an example of the master image 201 identified by the content ID stored in the search result holding unit 147 and displayed on the output unit 77 serving as a display. For example, when the green check box 492 is checked and the granularity is designated by the green slide bar 491, as shown in the upper left of FIG. 49, the main image 201 containing a lot of green is displayed on the output unit 77 which is a display. Is displayed. Further, for example, the green check box 492 is checked, the granularity is designated by the green slide bar 491, the red check box 492 is checked, the granularity is designated by the red slide bar 491, and the AND search radio button 493 is selected. When turned on, as shown in the upper right of FIG. 49, the main image 201 containing a lot of green and red is displayed on the output unit 77 which is a display.

  For example, when the blue check box 492 is checked and the granularity is specified by the blue slide bar 491, as shown in the lower left of FIG. 49, the main image 201 containing a lot of blue is displayed on the output unit 77 which is a display. Is displayed. Also, for example, the blue check box 492 is checked, the granularity is designated by the blue slide bar 491, the white check box 492 is checked, the granularity is designated by the white slide bar 491, and the AND search radio button 493 is selected. When turned on, as shown in the lower right of FIG. 49, the main image 201 including a large amount of blue and white is displayed on the output unit 77 that is a display.

  It is easy for the user to guess what color the desired image contains and how much the desired image can be retrieved.

  Furthermore, the search can be performed again by changing the granularity to an arbitrary granularity such as expanding or narrowing the conditions according to the search result. Thereby, a desired image can be retrieved more easily.

  In this way, it is possible to intuitively search for an image from the color image or atmosphere of the image possessed by the user.

  Since a search condition combining various conditions can be determined for the entire set of images, a search result that is an image can be taken out at an arbitrary granularity during the search.

  A color feature vector composed of the degree of relevance is extracted in advance, and the image can be retrieved by comparing with the degree of relevance or by a logical operation, so that the image can be retrieved quickly.

  Since the degree of association can be expressed by a numerical value having a relatively small number of digits, the data amount of the color feature vector can be further reduced. Therefore, the capacity of the recording space required for recording the color feature vector may be relatively small.

  In addition, although the digital still camera 11 and the mobile phone 12 were mentioned as an example of an apparatus, it is not restricted to this, The apparatus should just handle an image and may be a portable player or a viewer.

  As described above, when the metadata of the image is recorded, the image can be searched in the device. In addition, the device captures an image, records information related to the image as data of a predetermined structure in association with the image, and controls transmission of the image to the image processing device. Controls the reception of images transmitted from the camera, extracts the features of the received image, associates the features extracted from the image with the image, records them as data with the same structure as the device, and sends them to the device with the features When the transmission of the image is controlled, a desired image can be easily retrieved with a device having a relatively small processing capability.

  Further, when image metadata is recorded, an image can be searched for in the device. Also, the features of the image are extracted, the features extracted from the image are related to the image and recorded as data of a predetermined structure, and the information related to the image is recorded as data having the same structure as the structure When transmission to a device having a characteristic to be controlled is controlled, a desired image can be easily searched for in a device having a relatively small processing capability.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 2 or FIG. 3, a recording medium for recording a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc- (Including Read Only Memory), DVD (Digital Versatile Disc), magneto-optical disk), or removable media 82 which is a package medium made of semiconductor memory or the like, or ROM 72 in which a program is temporarily or permanently stored Or it is comprised by the hard disk etc. which comprise EEPROM46 and the memory | storage part 78. FIG. The program is stored in the program recording medium via a local area network, the Internet, or a digital satellite via the communication unit 47, the communication unit 48, the communication unit 79, or the communication unit 80, which is an interface such as a router or a modem, as necessary. It is performed using a wired or wireless communication medium such as broadcasting.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  11 Digital Still Camera, 12 Mobile Phone, 13 Server, 14 Network, 31 Shooting Lens, 32 Aperture, 33 Imaging Device, 34 Analog Signal Processing Unit, 35 A / D Converter, 36 Digital Signal Processing Unit, 37 MPU, 38 Memory, 40 monitor, 41 compression / decompression unit, 43 memory card, 46 EEPROM, 47 communication unit, 48 communication unit, 49 input unit, 71 CPU, 72 ROM, 73 RAM, 76 input unit, 77 output unit, 78 storage unit, 79 communication Unit, 80 communication unit, 82 removable media, 101 shooting control unit, 102 reduced image generation unit, 103 metadata generation unit, 104 entry generation unit, 105 recording control unit, 106 display control unit, 107 search unit, 108 transmission control unit , 09 reception control unit, 110 image holding unit, 111 content database, 112 similar feature database, 113 similar result database, 114 time group database, 115 search result holding unit, 121 distance calculation unit, 131 image analysis unit, 132 reduced image generation unit 133 metadata generation unit, 134 entry generation unit, 135 recording control unit, 136 display control unit, 137 search unit, 138-1 transmission control unit, 138-2 transmission control unit, 139-1 reception control unit, 139-2 Reception control unit, 140 image holding unit, 141 content database, 142 similar feature database, 143 similar result database, 144 time group database, 145 association degree extracting unit correspondence holding unit, 146 extraction feature Holding unit, 147 search result holding unit, 151 distance calculation unit, 161 face image detection unit, 162 similar feature amount extraction unit, 171 similar feature vector calculation unit, 172 color feature extraction unit, 201 main image, 202 reduced image, 261 meta Data, 401 image input unit, 402 “red” relevance extraction unit, 403 “blue” relevance extraction unit, 404 “yellow” relevance extraction unit, 405 extraction feature recording unit, 421 search condition input unit, 422 condition collation unit

Claims (12)

Feature extraction means for extracting features of the image;
An association means for relating the feature extracted from the image by the feature extraction means to the image;
Transmitting means for transmitting the characteristics of the image to a device;
The device records features of the image transmitted by the transmitting means, retrieves images associated with the features, and presents the retrieved images to the device by the transmitting means. An image processing apparatus comprising: a transmission control unit that controls transmission of features. The image processing apparatus according to claim 1, further comprising receiving means for receiving the image transmitted from the device. The image processing apparatus according to claim 2, wherein the feature is information relating to a face. The image processing apparatus according to claim 2, wherein the characteristic is information regarding a color. The image processing device
Extract image features,
Associating features extracted from the image with the image;
Send the image features to the device,
The device records features of the transmitted image, retrieves images associated with the features, and controls transmission of the image features to the device to present the retrieved images. Processing method. Receiving means for receiving features extracted from the image and related to the image by the server;
Recording means for recording the features received by the receiving means;
Search means for searching for an image associated with the feature;
An information processing apparatus comprising: presentation means for presenting an image retrieved by the retrieval means. The information processing apparatus according to claim 6, further comprising an image capturing unit that captures an image. The information processing apparatus according to claim 7, wherein the characteristic is information related to a face. The information processing apparatus according to claim 7, wherein the characteristic is information regarding a color. Information processing device
The server receives the features extracted from the image,
Record the received features;
Search for an image associated with the feature;
An information processing method for presenting searched images. Receiving means for receiving features extracted from the image and related to the image by the server;
Recording means for recording the features received by the receiving means;
Search means for searching for an image associated with the feature;
A program that causes a computer to function as presentation control means for controlling presentation of an image searched by the search means. A server and an image processing device,
The server
Feature extraction means for extracting features of the image;
An associating means for associating the feature with the image;
Transmission means for transmitting the characteristics to the image processing apparatus,
The image processing apparatus includes:
Means for receiving said features;
Recording means for recording the features received by the receiving means;
Search means for searching for an image associated with the feature;
An image processing system comprising: presentation means for presenting an image retrieved by the retrieval means.

Images