JP5414930B2 - Image playback device - Google Patents

Description

  The present invention relates to an image reproduction device.

  In recent years, digital still cameras and digital video cameras having a function of transmitting captured moving images or still images to an external device wirelessly or by wire have been realized. For example, there is a mobile phone with a built-in digital camera and data communication function. In addition, access points for data communication by wireless LAN can be used in various places.

  It is known to recognize an object included in a moving image or a still image and generate character code data indicating its characteristics. By using this, it is possible to search for an image including a desired object, or to search for an image having similar contents using the image as a search key.

  In many cases, extraction of an object included in an image requires a high level of data processing capability, and this is imposed on the server (feature recognition server). Patent Document 1 describes a system that transmits an image captured by a camera of a mobile terminal to a server and receives character data that describes an object included in the image, that is, object information from the server.

JP 2003-323620 A

  Patent Document 1 does not describe a method for effectively displaying object information received from a feature recognition server to a user when the user wants to display image data on the display panel of the mobile terminal body. Also, the display when the feature recognition server fails in the search is not considered.

  In view of the above problems, an object of the present invention is to present an image reproduction apparatus that effectively displays an image and information on an object included in the image on a display unit.

  An image reproduction apparatus according to the present invention is an image reproduction apparatus capable of displaying metadata on an object included in a captured image and including metadata including first and second information of the object on a display unit. Thumbnail display control means for displaying a list of thumbnail images on the display section, selection means for selecting a thumbnail image from the thumbnail images displayed in the list, and a captured image corresponding to the selected thumbnail image, Image display control means for displaying a photographed image larger than the thumbnail image on the display section, recording control means capable of recording the metadata corresponding to the photographed image to be displayed on the recording section, and the list display Includes a thumbnail image corresponding to the recorded image with the corresponding metadata. The first information included in the metadata is superimposed and displayed on the display unit without superimposing the second information to be displayed, and the corresponding metadata is a thumbnail image corresponding to the captured image not recorded. Is displayed on the display unit without superimposing the metadata, and when the image display control means displays the photographed image, the selected thumbnail image is superimposed with the first information. In the case of a thumbnail image, the metadata corresponding to the photographed image corresponding to the thumbnail image, and the first and second information included in the recorded metadata are superimposed on the photographed image and displayed. And a superimposed display control means for displaying on the screen.

  According to the present invention, it is possible to effectively display information on an object included in image data on a display unit.

It is a schematic block diagram of one Example of this invention. It is a schematic block diagram of a server according to the present embodiment. It is an example of the structure and content of the database 20b. It is a thumbnail display example. It is an example of a full screen display. It is an operation | movement flowchart of a present Example. It is an example of the content of the database 20b after metadata acquisition. This is an example in which metadata is overlaid and displayed in thumbnail display. This is an example in which metadata is displayed as an overlay in full screen display. It is an operation | movement flowchart of a present Example with respect to a moving image. It is the example of the content of the database 20b with respect to a moving image. It is an example of metadata display in a full screen display.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a schematic configuration block diagram of a digital video camera 10 according to an embodiment of the present invention, and FIG. 2 is a schematic configuration block diagram of a feature recognition server 50 that cooperates with the digital video camera 10. However, the solid line indicates the flow of image data, and the broken line indicates a control signal.

  The configuration of the digital video camera 10 will be described. The digital video camera 10 includes an imaging unit 12, a compression device 14, a recording / reproducing device 16, a metadata management device 18, an HDD (Hard Disk Drive) 20, a display control device 22, a display device 24, a communication device 26, and a system control device 30. , An operation device 32 and a memory 34 are provided.

  The imaging unit 12 includes an optical lens, an imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), an autofocus mechanism, a zoom mechanism, an A / D converter, and signal processing (for example, gamma correction, color balance). Adjustment, luminance / color separation), etc., and converts an optical image of a subject into an electrical signal and outputs a digital image signal.

  The recording / reproducing device 16 records and reproduces the image data from the imaging unit 12 on the HDD 20. The recording / reproducing apparatus 16 also generates a thumbnail image of the image data recorded on the HDD 20 and records it on the HDD 20. Note that the recording / reproducing apparatus 16 compresses and decompresses the image data as necessary. In this embodiment, both moving images and still images can be recorded on the HDD 20. MPEG (Moving Picture Experts Group phase) 2 and Motion JPEG are known as compression methods for moving images, and JPEG 2000 is known as a compression method for still images.

  The HDD 20 also records an object information database 20b that stores metadata indicating the characteristics of the recorded image 20a. When the image data is recorded in the HDD 20, the metadata management device 18 generates a metadata record corresponding to the image data in the object information database 20b.

  The object information database 20b corresponds to, as basic elements, unique identification information (ID), a reply flag (Flag) indicating that the server 50 has not been received, the server 50 has been received, and the server 50 has failed to search. It consists of a file path (FilePath) to image data, a name (Name) indicating the name of the found object, and detailed information (MoreInfo).

  FIG. 3 shows a configuration and example contents of the object information database 20b. In the reply flag, “0” indicates that the server 50 has not been received, “1” indicates that it has been received from the server 50, and “−1” indicates that the server 50 has failed to search. At the stage where the image data is recorded on the HDD 20, the reply flag is “0”. In the example illustrated in FIG. 3, the corresponding image is a still image, six pieces of image data are recorded in the HDD 20, and none of the feature data is received from the server 50. Although details will be described later, the metadata management device 18 of the recording / reproducing device 16 receives the metadata from the server 50, stores it in the database 20b of the HDD 20, and updates the reply flag according to the reception status.

  Although the HDD 20 is exemplified as the recording medium, it may be a removable medium such as a volatile semiconductor memory or a recordable optical disk.

  The display control device 22 causes the display device 24 to display the image according to the image data from the recording / reproducing device 16. The display control device 22 generates a menu image signal of a character-based interface, that is, a CUI (Character-based User Interface) or a graphic-based interface, that is, a GUI (Graphical User Interface), and applies the generated menu image signal to the display device 24. . In particular, in this embodiment, the display control device 22 applies metadata information to the display device 24. The display device 24 is, for example, an LCD (liquid crystal display panel).

  The communication device 26 can transmit and receive data to and from the server 50 in accordance with IEEE (Institute of Electrical and Electronic Engineers) 1394, IEEE 802.11, USB (Universal Serial Bus), etc. In this embodiment, HTTP (Hyper An Internet protocol such as Text Transfer Protocol (FTP) or File Transfer Protocol (FTP) is used.

  The system control device 30 controls the entire video camera 10 in accordance with the operation of the operation device 32. The operation device 32 includes a moving image shooting switch, a release switch (SW1, SW2), a playback switch, and a stop switch, and the user operates the system to control moving image shooting, still image shooting, playback, stop, and the like. 30. The operation device 32 includes a menu key, a cross key, a determination key, and an enlargement key, and the user uses these to instruct the system control device 30 to display, select, determine, and enlarge a thumbnail image. To do.

  The memory 34 includes a ROM (Read Only Memory) that stores programs and fixed parameters of the system control device 30, a RAM (Random Access Memory) that stores temporary data and serves as a work memory of the system control device 30, and the like. Yes.

  The configuration of the server 50 will be described with reference to FIG. The communication device 52 communicates with the camera 10 and receives image data (moving image data or still image data) from the camera 10. The recording / reproducing device 54 records and reproduces the image data received by the communication device 52 on a hard disk (HDD) 56. The recording / reproducing device 58 reproduces the image data requested for feature extraction by the camera 10 from the image data recorded in the HDD 56 and supplies the image data to the image analyzing device 58. If there is no need to permanently store the received image data, the HDD 56 is simply used as a temporary storage for image analysis.

  The image analysis device 58 analyzes luminance or color distribution or pattern from the image data reproduced from the HDD 56 and extracts features from the analysis result. In other words, the image analysis device 58 is a feature extraction device. The collation device 60 collates the feature obtained by the image analysis device 58 with the feature database 62, and acquires the name of the image subjected to image analysis and the detailed description text from the feature database 62. The feature database 62 stores features extracted by image analysis of the image analyzer 58, object names, and detailed descriptions for various images or objects.

  The collation device 60 supplies the collation result with the feature database 62 to the metadata generation device 64. The metadata generation device 64 generates metadata in a predetermined format that accommodates the verification result of the verification device 60. The generated metadata is transmitted by the communication device 52 to the camera 10 that is the request source. The metadata management device 18 of the camera 10 stores the metadata transmitted from the server 50 in the HDD 20 and updates the reply flag.

  A basic operation of imaging, recording and reproduction of the camera 10 will be described. In the camera mode, the imaging unit 12 outputs image data corresponding to the optical image of the subject. Output image data from the imaging unit 12 is written in a video memory (VRAM) (not shown). The display control device 22 drives the display device 24 according to the image data stored in the video memory. Thereby, the user can confirm the subject and the composition. In the recording mode, the recording / reproducing device 16 further records the image data from the imaging unit 12 on the HDD 20. In this way, the image data of the image to be shot is stored in the HDD 20.

  In the playback mode, the recording / playback apparatus 16 reads image data of a specified image from the HDD 20. With respect to the read image data, the display control device 22 drives the display device 24 in accordance with the image data read from the HDD 20 by the recording / reproducing device 16. As a result, the reproduced image is displayed on the screen of the display device 24.

  In this embodiment, it is possible to select a list display of thumbnail images of still images recorded in the HDD 20 and a full screen display of one designated image. FIG. 4 shows an example of a screen displaying a list of thumbnails of six images, and FIG. 5 shows an example of displaying one designated image on a screen for shaking. For example, when a desired one is selected in the thumbnail display as shown in FIG. 4 and the enlargement key of the operation device 32 is operated, one designated image is displayed in a full screen as shown in FIG. The

  Next, with reference to FIGS. 6 to 9, a characteristic operation of the present embodiment for a still image, that is, an operation for displaying metadata overlay will be described. 6 shows an operation flowchart, FIG. 7 shows an example of contents of the object information database 20b stored in the HDD 20, FIG. 8 shows an example of thumbnail display, and FIG. 9 shows an example of full screen display.

  The system control device 30 determines whether or not the photographing switch has been pressed (S10). When the photographing switch is pressed (S10), a still image is photographed and the photographed image is recorded on the HDD 20 by the recording / reproducing device 16 (S11). At the same time, as described above, the metadata management device 18 generates a new metadata record for the still image data in the database 20b (S12). At this time, the new record of metadata consists only of an ID, an update flag with a value “0”, and a file path to image data. The captured image data is also transmitted to the server 50 via the communication device 26 (S13).

  The server 50 determines whether image data has been received by the communication device 52 (S20). When the image data is received (S20), the recording / reproducing device 54 records the received image data in the HDD 56 (S21). Image data is read from the HDD 56 in the order of recording and sent to the image analyzer 58. The image analyzer 58 analyzes the luminance or color distribution or pattern of the image data and extracts the features of the image data (S22).

  The extracted features are sent from the image analysis device 58 to the collation device 60, and the collation device 60 collates the features of the image data with the feature database 62 and searches the feature database 62 for records that match the features (S23). The metadata generation device 64 generates metadata that matches the image data from the camera 10 from the search result (S24). The generated metadata is transmitted to the camera 10 via the communication device 52 (S25). If the image analysis (S22) or search (S23) has failed, the failure is transmitted to the camera 10 (S25).

  The communication device 26 of the camera 10 stands by for receiving metadata (S30). When the metadata is received from the server 50 (S30), the server management device 18 stores each piece of information of the received metadata in the corresponding field of the corresponding record in the object information database 20b of the HDD 20 (S31). At this time, the reply flag is updated to a value “1” indicating that it has been received.

  FIG. 7 shows an example of the contents of the object information database 20b updated in this way. In the example illustrated in FIG. 6, metadata is obtained for the image data of ID1, ID2, and ID4, the metadata search has failed for the image data of ID3, and the images of ID5 and ID6 are detected. For data, it is waiting to receive metadata. The contents of the name field (Name) and the detail field (MoreInfo) are supplemented with information from the server 50.

  When metadata is acquired as shown in FIG. 6 and images recorded in the HDD 20 are displayed as a list of thumbnails, the character data in the name field of the metadata are indicated by reference numerals 80, 82, 84, 86 in FIG. As shown in FIG. 6, the thumbnail images are displayed in a superimposed manner. For an image whose reply flag is “−1”, as shown by reference numeral 86, the character “???” indicating unknown is superimposed on the thumbnail image, ie, is overlaid. It is obvious that a specific symbol or figure may be used instead of the character “???”.

  When the user selects a specific image on the thumbnail screen shown in FIG. 8 and presses the enlarge key of the operation device 32, a full screen display as shown in FIG. 9 is displayed. In this full screen display, the name field and the detail field are displayed. As shown by reference numeral 90 in FIG. 9, the character data is superimposed on the reproduced image, that is, displayed as an overlay.

  As described above, in this embodiment, the display mode is changed according to the presence or absence of the search result of the object information by the server 50, so that the user can check on the screen whether the object information has been acquired or the search has failed. .

  This embodiment can be applied to moving images. FIG. 10 shows an operation flowchart of this embodiment for moving images.

  The system control device 30 determines whether or not the moving image shooting switch has been pressed (S110). When the moving image shooting switch is pressed (S110), shooting of a moving image is started, and each shot frame image is sequentially recorded on the HDD 20 by the recording / reproducing device 16 (S111). At the same time, as described above, the metadata management device 18 generates a new metadata record for the moving image data on the database 20b (S112). At this time, the new record of metadata consists only of an ID, a reply flag with a value “0”, and a file path to image data.

  The moving image data recorded in the HDD 20 is transmitted to the server 50 via the communication device 26 while the moving image is being shot or after the moving image is shot (S113).

  The server 50 determines whether image data has been received by the communication device 52 (S120). When the image data is received (S120), the recording / reproducing device 54 records the received image data in the HDD 56 (S121). Image data is read from the HDD 56 in the order of recording and sent to the image analyzer 58. The image analyzer 58 analyzes the luminance or color distribution or pattern of the image data and extracts the features of the image data (S122).

  The extracted features are sent from the image analysis device 58 to the collation device 60. The collation device 60 collates the features of the image data with the feature database 62, and searches the feature database 62 for records that match the features (S123). . The metadata generation device 64 generates metadata that matches the image data from the camera 10 from the search result (S124).

  Steps S121 to S124 are executed for all the frames of the moving image (S125). For frames that fail image analysis or fail to retrieve from the database 62, image analysis of the next frame is immediately executed. When the end of the moving image is reached (S125), the generated metadata is transmitted to the video camera 10 via the communication device 52 (S126). If finally valid metadata cannot be obtained, a failure is transmitted to the camera 10 (S126).

  If the user of the camera 10 designates a frame for requesting metadata in the moving image, the processes of steps S121 to S124 are executed for the designated frame.

  The communication device 26 of the camera 10 stands by for receiving metadata (S130). When the metadata is received from the server 50 (S130), the server management device 18 stores the received metadata in the corresponding record in the object information database 20b of the HDD 20 (S131). At this time, the reply flag is updated to a value “1” indicating that it has been received.

  FIG. 11 shows an example of the contents of the object information database 20b in the case of a moving image. A field (Time) indicating the frame position of the object information is added. The Time field 35:05:00 for ID1 indicates that the record is metadata for a frame of 35 minutes 05 seconds 00 from the start of imaging. A number may be indicated. With respect to this metadata, in the full screen display, as shown in FIG. 12, the data of the Time field is also displayed as an overlay on the screen.

DESCRIPTION OF SYMBOLS 10 Video camera 12 Imaging unit 16 Recording / reproducing apparatus 18 Metadata management apparatus 20 HDD
20a: Image data 20b Object information database 22 Display control device 24 Display device 26 Communication device 50 Server 52 Communication device 54 Recording / playback device 56 HDD
58 Image analysis device 60 Collation device 62 Feature database 64 Metadata generation device

Claims (1)

An image playback apparatus capable of displaying metadata on an object included in a captured image and including metadata including first and second information of the object on a display unit,
Thumbnail display control means for displaying a plurality of thumbnail images in a list on the display unit;
Selecting means for selecting a thumbnail image from the thumbnail images displayed in the list;
Image display control means for displaying on the display unit a photographed image corresponding to the selected thumbnail image and larger than the thumbnail image;
A recording control unit capable of recording the metadata corresponding to the captured image to be displayed in a recording unit;
When the list is displayed, the corresponding metadata is included in the metadata without superimposing the second information included in the metadata for the thumbnail image corresponding to the recorded photographed image. While the first information is superimposed and displayed on the display unit, the corresponding metadata is displayed on the display unit without superimposing the metadata on the captured image that is not recorded. When the image display control means displays a photographed image, when the selected thumbnail image is a thumbnail image on which the first information is superimposed, it corresponds to the photographed image corresponding to the thumbnail image. Superimposing the first and second information included in the recorded metadata to be displayed on the display unit by superimposing the captured image on the captured image And 示制 your means,
An image reproducing apparatus comprising:

Images