JP2010287974A - Mobile phone and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile phone, or the like, capable of automatically providing a moving image that is easy for a user to view, in real time during photographing, on the basis of a reference image without applying load on the user by imparting a tag. <P>SOLUTION: The mobile phone including a moving image data storage section and a moving image data expanding section includes a reference image feature storage section wherein reference images, feature quantities of the reference images, and keywords are stored in association with each other; a feature quantity calculating section which calculates the feature quantity of an image included in moving image data; a feature quantity comparing section which compares the feature quantity calculated by the feature quantity calculating section, with the feature quantities stored in the reference image feature storing section; and a tag information preserving section which generates and preserves tag information including a keyword of a reference image resembling the image contained in the moving image data, on the basis of the comparison result of the feature quantity comparing section. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention creates a moving image data from an image input from the image input unit and a sound input from the audio input unit, and stores the moving image data in the moving image storage, and the moving image data from the moving image storage. The present invention relates to a mobile phone or the like provided with a moving image data expansion unit that reads out and expands an image and sound and outputs the image to an image output unit and the sound to an audio output unit.

  Various image processing apparatuses that capture and reproduce images and moving images are known. Here, even in devices such as mobile phones, it has become possible to shoot high-quality moving pictures for a long time due to the low cost of memory media and the incorporation of high-pixel cameras. .

  When shooting for a long time, it is necessary to perform operations such as fast-forwarding in order to find a scene to be seen in particular, which takes time. In video software sold on DVD, etc., chapters are divided in advance, and if you jump by specifying chapters, you can see the target scene immediately, but videos shot with mobile phones etc. Later, it was necessary to move the data to a personal computer, etc., and work with special editing software or by setting the scene breaks by themselves. Therefore, the difficulty of using the video shooting function by the user has been one of the reasons why the user cannot use the video shooting function while having the ability to shoot for a long time.

  Here, in the case of a recording device for TV and the like, there is a device that automatically divides by CM or the like, but there are a method for detecting silence and a method for detecting a change of monaural / stereo sound, and in shooting with a mobile phone. Was difficult to use.

  Here, a metadata providing apparatus and a metadata providing method that can be applied even when the classified image is a moving image are disclosed (for example, see Patent Document 1). For example, a feature is provided that extracts feature values from still image data collected from the Internet, compares them with feature values extracted from each frame data (image) of moving images, and assigns keywords related to near still image data as metadata is doing.

JP 2008-217701 A

  By using the above-described method, it is possible to perform a similar image search for each frame of a moving image and to add metadata to each frame. Generally, however, a process for calculating a feature vector of each image and a feature vector are used. The similar image search takes a large amount of processing and takes time, and in particular, the larger the image size, the more time it takes. Thus, there remains a problem of how to apply it during recording. In particular, a dedicated recording device such as a video camera is relatively easy to implement because it uses dedicated hardware for handling large-sized movies, but it is used in common with other functions such as a telephone, such as a mobile phone. It was difficult to realize it under the restriction that a general-purpose and small CPU had to be used.

  Also, assuming that all video frames have been given metadata, in order to present a list of scene changes, i.e., scenes where something unique is first seen, to the user during playback, all metadata must be entered once. However, as long as the same image is continuously captured in the video, metadata corresponding to the number of frames is added, which causes a problem that it takes time to read.

  In view of the above-described problems, the present invention aims to automatically convert a reference image to a reference image in real time during shooting even when shooting a high-resolution moving image with a mobile phone having relatively low processing capability. An object of the present invention is to provide a mobile phone or the like that can provide a video that can be easily viewed by the user by attaching a tag without imposing a burden on the user.

  In view of the above-described problems, the mobile phone of the present invention includes a moving image data storage unit that creates moving image data from an image input from the image input unit and audio input from the audio input unit and stores the moving image data in the moving image storage. A mobile phone comprising: a moving image data reading unit that reads moving image data from the moving image storage, expands an image and sound, outputs an image to an image output unit, and outputs sound to an audio output unit; A reference image feature storage unit that stores a reference image, a feature amount of the reference image, and a keyword in association with each other, a feature amount calculation unit that calculates a feature amount of an image included in the moving image data, and the feature amount calculation unit An image included in the moving image data based on a comparison result of the feature amount comparison unit that compares the feature amount calculated by the feature amount stored in the reference image feature storage unit and the feature amount comparison unit In A tag information storage unit to store generates tag information including the keyword of the reference image to be similar, characterized in that it comprises a.

  In the mobile phone of the present invention, the tag information storage unit stores the start time of moving image data of an image similar to the reference image in addition to the tag information.

  In the mobile phone of the present invention, the tag information storage unit stores the end time of moving image data of an image similar to the reference image in addition to the tag information.

  In the mobile phone of the present invention, the tag information storage unit stores the scene information corresponding to the keyword corresponding to the reference image in the tag information.

  In addition, the mobile phone of the present invention further includes a reproduction instruction unit that instructs the moving image data expansion unit to perform reproduction based on tag information based on the moving image data.

  In the mobile phone of the present invention, the reproduction instruction unit identifies a scene including a keyword from the tag information, and reproduces the moving image data expansion unit so that the scene is continuously reproduced. It is characterized by giving instructions.

  Further, in the mobile phone of the present invention, the moving image data expansion unit, when instructed to fast-forward from the reproduction instruction unit, based on the tag information, only the first frame of each tag or a frame of several seconds from the top of each tag Only, the moving image data is expanded and output in order according to the time series.

  The program of the present invention includes a moving image data storage function for creating moving image data from a video data and sound data in a computer equipped with an image output device and an audio output device, and storing the moving image data in a moving image storage; A program that reads moving image data, expands an image and sound, and realizes a moving image data expansion function that outputs the image to the image output device and the sound to the sound output device. A reference image feature storage function for storing a feature quantity and a keyword in association with each other, a feature quantity calculation function for calculating a feature quantity of an image included in the moving image data, and a feature quantity calculated by the feature quantity calculation function Based on the comparison result of the feature amount comparison function for comparing the feature amount stored by the reference image feature storage function and the feature amount comparison function, Characterized in that to realize the tag information stored function to generate and save the tag information including the keyword of the reference image similar to the image included in the video data.

  According to the mobile phone of the present invention, it has the reference image feature storage unit that stores the reference image, the feature amount of the reference image, and the keyword in association with each other, and calculates the feature amount of the image included in the moving image data. Then, the calculated feature amount is compared with the feature amount stored in the reference image feature storage unit. Then, based on the comparison result, tag information including a keyword of a reference image similar to the image included in the moving image data is generated and stored. Therefore, tag information is generated based on the reference image stored in the reference image feature storage unit without the photographer being conscious, and the moving image data that the user can reproduce using the tag information during reproduction. Can be provided.

  In addition, the mobile phone of the present invention can reproduce only a scene including the keyword from the moving image data by designating the keyword by the user.

It is a figure for demonstrating the function structure of the mobile telephone in this embodiment. It is a figure for demonstrating the function part used when the mobile telephone in this embodiment image | photographs moving image data. It is a figure for demonstrating the function part used when the mobile telephone in this embodiment reproduces | regenerates moving image data. It is a figure for demonstrating an example of the data structure of reference image characteristic DB in this embodiment. It is a figure for demonstrating an example of the data structure of the moving image storage in this embodiment. It is a figure for demonstrating an example of the tag information in this embodiment. It is a figure for demonstrating an example of the tag information generation process in this embodiment. It is a figure for demonstrating an example of the tag preservation | save process in this embodiment. It is a figure showing the time sequence of the moving image data in this embodiment. It is a figure for demonstrating an example of the screen display in this embodiment. It is a figure showing the time sequence of the moving image data in this embodiment.

  The best mode for carrying out the present invention will be described below with reference to the drawings. In the present embodiment, as an example, a case will be described in which the present invention is applied to a mobile phone 1 capable of high-resolution moving image shooting.

[Function configuration]
First, the functional configuration of the mobile phone 1 will be described. FIG. 1 shows the entire functional configuration of the mobile phone 1, and FIG. 2 shows functional units used when recording a moving image on the mobile phone 1. FIG. 3 is a diagram mainly illustrating functional units used when reproducing a moving image.

  As shown in FIG. 1, the mobile phone 1 includes a control unit 10, a communication unit 20, an image input / output unit 22, a camera unit 24, a display unit 26, an image processing unit 30, and an audio input / output unit 40. The microphone 42, the speaker 44, the sound processing unit 50, the storage unit 60, the moving image processing unit 70, the image analysis unit 80, and the reproduction instruction unit 90 are connected via a bus.

  The control unit 10 is a functional unit that controls the mobile phone 1, and is configured by a processing device such as a CPU, for example. The control unit 10 implements each function by reading and executing various programs stored in the storage unit 60.

  The communication unit 20 is a functional unit for the mobile phone 1 to communicate with a base station, and can communicate voice data and each data via the network (for example, a mobile phone network) via the communication unit 20.

  The image input / output unit 22 is a functional unit that processes an input image and outputs the processed image to each functional unit. For example, it is a functional unit that processes an image input from the camera unit 24 or outputs an image in a display form suitable for the display unit 26.

  Here, it is assumed that the image input / output unit 22 has an image conversion function for appropriately converting an image having an appropriate size according to an output device and a function unit and outputting the image. For example, when the input image data (for example, image data with a resolution of XGA) is output to the display unit 26 (for example, a display unit with VGA as a display capability), the display size of the display unit 26 (with a resolution of VGA). Image data) is appropriately converted and output.

  In the present embodiment, the image data output from the image input / output unit 22 to the feature amount calculation unit 82 is output as image data output to the display unit 26. This reduces the processing load by using display image data (that is, image data having a resolution lower than that of the captured data) in the process of extracting feature quantities having the next largest processing load after image encoding. can do.

  The camera unit 24 is a functional unit for photographing a subject, a landscape, and the like, and is configured by, for example, a CCD. An image captured by the camera unit 24 is output to the image input / output unit 22.

  The display unit 26 is a functional unit that displays various information in the mobile phone 1 and displays an image or a moving image. For example, it is configured by a liquid crystal display, an organic EL display, or the like.

  The image processing unit 30 is a functional unit that encodes and decodes image data in a predetermined format in the mobile phone 1. The image processing unit 30 includes an image encoder unit 32 that encodes image data and an image decoder unit 34 that decodes image data.

  The voice input / output unit 40 is a functional unit that processes input voice and outputs the processed voice to each functional unit. For example, it is a functional unit that processes sound input from the microphone 42 or outputs it to the speaker 44. In this embodiment, the speaker 44 is used as an audio output unit. However, it is needless to say that the audio output unit may be composed of other audio output devices such as headphones.

  The voice processing unit 50 is a functional unit that encodes and decodes voice data in a predetermined format in the mobile phone 1. The audio processing unit 50 includes an audio encoder unit 52 that encodes audio data, and an audio decoder unit 54.

  The storage unit 60 is a functional unit for storing the setting state of the mobile phone 1 and storing temporary image data. Here, the storage unit 60 is configured by any storage device such as a semiconductor memory, a hard disk drive, or an optical disk drive, for example. Further, various data and programs for operating the mobile phone 1 are also stored. The control unit 10 executes various control processes by reading and executing the control program stored in the storage unit 60.

  Further, the storage unit 60 has areas for the reference image feature DB 62 and the moving image storage 64.

  An example of the data structure of the reference image feature DB 62 is shown in FIG. The reference image feature DB 62 stores a file path of the reference image, a keyword corresponding to the reference image, and feature amount data of the reference image in association with each other.

  Here, even if the subject is the same, there is a difference in the feature amount of the image depending on the shooting conditions and the like, so that a higher identification rate can be obtained by registering a plurality of images corresponding to the same keyword. The reference image may be registered in advance by the manufacturer of the mobile phone, or may be registered later by the user. In addition, since mobile phones often have a telephone directory function that can register images corresponding to person names, these may be used as reference images. Furthermore, in still image shooting with a mobile phone, position information can be embedded using GPS, and this may be used.

  The moving image storage 64 is a storage area in which moving image data and tag information corresponding to the moving image data are stored. For example, as shown in FIG. 5, tag information (for example, “<taglist> ˜”) is stored in association with moving image data (for example, “recorded100.mp4”).

  Here, the tag information will be described with reference to FIG. First, moving image data is specified by a filename tag (T1), and a tag in the moving image data is specified by a tag id tag (T100). A range (for example, T10 or T12) surrounded by the tag id tag is one tag.

Details of each tag information are as follows.
ref-img tag (T102): Path information for accessing the reference image is recorded.
thumbnail tag (T104): Path information related to the thumbnail image is recorded.
keyword (T106): A keyword for the content is recorded.
start-time tag (T108): The start time of the content is recorded.
end-time tag (T110): The end time of the content is recorded.

  In this embodiment, the XML format is used as an example of the tag information, but it is needless to say that the tag information may be stored in another format.

  The moving image processing unit 70 combines the input image data and audio data to encode the moving image data into a predetermined format such as the Mpeg4 format, or conversely, the read moving image data is converted into image data and audio data. This is a functional unit that outputs data separately. The moving image processing unit 70 includes a moving image data storage unit 72 that encodes and stores moving image data, and a moving image data expansion unit 74 that expands the moving image data and separates (decompresses) it into image data and audio data.

  The image analysis unit 80 is a functional unit that analyzes various images and reads and saves tag information from the moving image storage 64. The image analysis unit 80 includes a feature amount calculation unit 82, a feature amount comparison unit 84, a tag storage unit 86, and a tag reading unit 88.

  The feature amount calculation unit 82 is a functional unit that calculates the feature information of the input image data for one frame. In this embodiment, the feature information calculation method is not particularly limited. For example, a method using the R, G, and B color histograms in the RGB method as feature information, a method using the Y-only histogram in the YUV method as feature information, and the like. Is more suitable.

  Note that the data input to the feature amount calculation unit 82 is display image data output from the image input / output unit 22. Note that captured image data (that is, high-resolution image data) may be used depending on the processing performance of the feature amount calculation unit 82.

  The feature amount comparison unit 84 is a functional unit for calculating the degree of similarity by comparing the feature information calculated by the feature amount calculation unit 82 with the feature information acquired from the reference image feature DB 62.

  The tag storage unit 86 performs recording based on the reference image information determined by the feature amount comparison unit 84 to have a certain degree of similarity and the time stamp information of the frame data subjected to the feature calculation processing by the feature amount calculation unit 82. The tag information of the moving image data inside is created and stored in the moving image storage 64 in association with the moving image data.

  The tag reading unit 88 reads the tag information included in the moving image storage 64 and outputs the tag information to the moving image data expansion unit 74 and the image input / output unit 22.

  The reproduction instructing unit 90 is hardware and software having a role of designating a position so that a user who views the tag information displayed on the display unit 26 reproduces from a favorite position of a moving image. As the hardware, a method using a touch panel integrated with the display unit 26 may be used, or a method using a key of a mobile phone, a keyboard connected to the mobile phone, or the like may be used.

[When recording movies]
Next, each functional unit will be described focusing on the functional units used when recording a moving image and reproducing the moving image. First, a case where a moving image is recorded will be described.

  The camera unit 24 and the microphone 42 are a photographing device and a voice input device attached to the mobile phone 1, respectively.

  An image (moving image) input from the camera unit 24 is output to the image input / output unit 22 and output to the display unit 26, the image encoder unit 32, and the feature amount calculation unit 82 as image data. The feature amount calculation unit calculates a feature amount based on the image input from the image input / output unit 22. Then, the feature amount of the reference image stored in the reference image feature DB 62 is compared with the feature amount calculated by the feature amount calculation unit 82 to recognize a similar reference image (ie, tag).

  Here, image data encoded in a predetermined format is transmitted from the image input / output unit 22 by the image encoder unit 32.

  The voice input from the microphone 42 is changed to voice data by the voice input / output unit 40. Further, the audio encoder unit 52 encodes the data into a predetermined format and outputs the encoded data to the moving image data storage unit 72.

  Further, the moving image data combining unit sets the image data output from the image encoder unit 32 and the audio data output from the audio encoder unit 52 as a moving image file. The moving image file is stored in the moving image storage 64 in the moving image storage 64 together with the tag information stored by the tag storage unit 86.

[Playback function]
Next, a case where a moving image is reproduced will be described. FIG. 3 is a system diagram showing an embodiment for realizing a function of reproducing a moving image on the mobile phone 1 in the present embodiment.

  When the reproduction instruction unit 90 instructs the user to reproduce a moving image, the moving image data expansion unit 74 reads the corresponding moving image data and tag information from the moving image storage 64.

  The moving image data read from the moving image storage 64 is output to the moving image data expansion unit 74, and the tag information is output to the moving image data expansion unit 74 and the image input / output unit 22 via the tag reading unit 88.

  The moving image data expansion unit 74 outputs the input moving image data separately into image data and audio data. Specifically, the image data is output to the image decoder unit 34 and the audio data is output to the audio decoder unit 54.

  Then, the image decoder unit 34 decodes the image and outputs it to the display unit 26 via the image input / output unit 22. Also, the audio decoder 54 decodes the audio and outputs it to the speaker 44 via the audio input / output unit 40.

[Tag information generation processing]
Next, a tag information generation procedure at the time of moving image shooting will be described with reference to FIG. In the mobile phone 1, an image serving as a reference image and a keyword corresponding to the image are registered in advance in the reference image feature DB 62. Here, a plurality of reference images can be registered, and a feature amount is calculated by the feature amount calculation unit 82 before shooting and registered in the reference image feature DB 62.

When shooting is started, all feature quantity data is first read from the reference image feature DB 62 into the feature quantity comparison unit 84 (step S10). Specifically, if n pieces of feature data are registered, they are read as variables F _{r1 to} F _rn . When reading is completed, input of an image from the camera unit 24 is started (step S12).

  The image input / output unit 22 outputs an image to the image encoder unit 32 according to the shooting parameters. Further, in order to make it easy for the user to understand what the image provided to the image encoder unit 32 is, the image is also output to the display unit 26. Further, the image input / output unit 22 includes a buffer that can always acquire the latest image data, and notifies the feature amount calculation unit 82 when data for one frame is accumulated.

  In the case of a portable terminal, since the resolution of the display unit 26 is generally smaller than the size that can be photographed, the resolution of the image output to the display unit 26 is coarser than the image output to the image encoder unit 32. Therefore, in this embodiment, an image to be displayed on the display unit 26 is used instead of an image to be actually encoded as a feature extraction target.

That is, the same data as the image to be displayed on the display unit 26 is stored in the buffer of the image input / output unit 22. The feature amount calculation unit 82 can acquire the data at any timing as long as it is ready to acquire the data. However, if a new image is input from the camera before the data is acquired, the image input / output is performed. The data is overwritten in the unit 22. When notified that the data has accumulated in the buffer, the feature amount calculation unit 82 receives data for one frame (step S14), and calculates the feature amount data. Here, the calculated feature quantity is stored in F _i which means the feature quantity data of the i-th frame (step S16).

この算出方法では、ヒストグラムのすべての要素が同じ値の時にはＳ_ｒｎ＝０となり、もっとも類似している画像であると判断でき、Ｓ_ｒｎが大きいほど類似していないと判断することができる。類似度の算出は読み込んである参照画像データの特徴量データの数だけ算出するので、ｎ個読み込んであればＳ_ｒ１〜Ｓ_ｒｎに格納される。 Next, the feature amount comparison unit 84 calculates the similarity between F _i and F _{r1 to} F _rn (step S18). There are various calculation methods for calculating the degree of similarity, depending on the calculation method of feature amount data. For example, when a histogram of only Y in the YUV method is used as feature amount data, the difference between each element is added over all elements. Similarity can be obtained. That is, each element _F i of _{F i,} F r1 _{to F rn} each element _F r1 _{to F rn} of, if the number of elements is m, the similarity between the feature amount data of _{F i} and n-th reference image S _rn is obtained by the following equation.

In this calculation method, when all the elements of the histogram have the same value, S _rn = 0, so that it can be determined that the images are most similar, and as S _rn is large, it can be determined that they are not similar. Since the similarity is calculated by the number of feature amount data of the read reference image data, if n are read, they are stored in S _{r1 to} S _rn .

  Then, among the calculated similarities, those having a value (threshold value) below a certain value, that is, a value that can be determined to have a value above a certain value are extracted and extracted as tag information candidates (step S20). This constant threshold value may be the same value for all reference images, or may be set individually for each reference image.

  Subsequently, tag storage processing (step S22) is executed. If the recording is not finished, the process is repeated from step S14. In other cases, the process is finished.

[Tag save processing]
Subsequently, the tag storing process executed in step S22 of FIG. 7 will be described with reference to FIG.

  The tag storage unit 86 records the frame tag information by repeating steps S50 to S58 for each of the image data determined to be similar to a certain level in step S20 of FIG.

  First, paying attention to one similar reference image data, it is determined whether or not the reference image data is determined to be similar in the previous frame (step S50). If it is determined that the image is similar in the previous frame, the tag information candidate is skipped (step S50; Yes → step S60).

  On the other hand, if it is not determined to be similar in the previous frame, it is added as tag information to the tag list (step S52). What is added here is the tag tag in FIG. 5 and its id (for example, T100). Next, path information for accessing the reference image and keyword information of the reference image are recorded in the tag information added in step S52 (step S54). These correspond to the ref-img tag (for example, T102) and the keyword (for example, T106) in FIG. Next, in S44, time stamp information of the input frame data is recorded (step S56). This corresponds to the start-time tag (for example, T108) in FIG.

  Next, the frame data is recorded as a thumbnail image and recorded as information for access (step S58). This is information for easily presenting the viewer with a list of scenes during playback of a moving image. Of course, if there is no such use, this procedure is unnecessary. For example, the frame may be decoded based on the start time information of the tag information and presented to the viewer.

  As described above, when steps S50 to S58 are repeated, tag information based on the reference image data newly extracted as the tag information candidate is added. Thereafter, when the processing is completed for all the tags (step S60; Yes), the frame data is added to the tag information in which the end time is not recorded among the tag information that has not been changed in the series of processing from S50 to S60. Is recorded as the end time (step S62). This corresponds to the end-time tag (for example, T110) in FIG. This means that the scene determined to be similar to a certain reference image has ended. Note that the tag storage processing per frame ends here.

  Thus, the range from the start time to the end time for one tag is referred to as a “scene”. For example, a range of moving image data corresponding to “mountain” as tag information (keyword) is set as one “scene”.

  FIG. 9 is an image view showing a part of scenes in the moving image data in time series indicated by the tags stored as shown in FIG. In this section, scenes of flowers, people, and autumn leaves are photographed, and there are no scenes corresponding to the other keywords (night view, new installation, snow scene) shown in FIG. In addition, it can be seen that the scenes of flowers and people, and people and autumn leaves are part of the subject at the same time.

  Specifically, for tag id = 1 (frames with a time stamp of 12345 to 13345), a flower is used for tag id = 2 (frames with a time stamp of 13,000 to 16000), and for tag id = 3 (time stamp). Is a frame of 18001 to 19123), and a person is shown for a tag id = 4 (a frame whose time stamp is 18583 to 24210). Here, it can be seen that for the frames with time stamps of 13000 to 13345, both the flower and the person are shown.

  As described above, in the present embodiment, a scene having a plurality of meanings is provided in a condition that the degree of similarity with a reference image is equal to or greater than a threshold value, instead of uniquely determining what kind of scene a certain moving image frame is. Is allowed. By doing this, it is possible to prevent a situation where a person is captured but a flower is also captured and the scene is judged to be a flower, and it cannot be reproduced from the scene where the person appears. I can do it.

  FIG. 10 is an example of a screen for selecting a scene that the viewer wants to see. In FIG. 10A, a scene list is displayed, and a scene can be selected from the list and reproduced. Here, four scenes exemplified as “flower”, “person”, “autumn leaves”, and “person” are arranged in time series. What is displayed is a thumbnail image for visually determining what kind of scene it is, a keyword indicating what scene it is, and a start time.

  Of course, only thumbnail images may be displayed so that a scene can be selected, or when there are many scenes, the screen may be scrolled and displayed. Further, as the thumbnail image, the thumbnail image included in the tag information shown in FIG. 5 may be used, or the moving image may be actually reproduced from the time indicated by the tag and presented as a moving image.

  FIG. 10B shows an example of a method in which scene names are designated and scenes having the same scene name are reproduced together. When “person” is selected in the moving image shown in the example of the embodiment described above, the scenes with tag id = 2 and tag id = 4 can be reproduced continuously. With this same method, for example, it is possible to extract only a scene in which a person is shown and create it as another moving image file.

  Further, as another method of using the tag information, use for fast-forwarding a moving image can be considered. Fast-forwarding so far includes decoding all frames faster than usual (such as double-speed playback), displaying only frames at regular intervals, and only intra-frame encoded frames such as MPEG4. Display methods have been used.

  However, since the display of a certain section is skipped regardless of the content of the moving image, it has been difficult to adjust the position exactly to be viewed. Therefore, by displaying the thumbnail images of the recorded tag information one after another, it becomes possible to resume playback from the position where some scene starts, regardless of the position where fast forward is canceled.

  Also, in the case of fast-forwarding, not only thumbnail images are displayed, but also only a few seconds from the beginning of each tag are displayed continuously, so that the contents can be played back in a form that is easier to grasp (digest playback is performed) You can also FIG. 11 is a diagram showing, in a time series, frames that are played back and frames that are not played back when fast-forwarding a moving image recorded by the method according to the above-described embodiment.

  The video data development unit 74 determines the playback time of each tag in advance (or may be set by the user), and starts this playback operation when there is a fast-forward operation (or digest playback operation) by the user. .

  First, tag information at a time closest to the current reproduction time is acquired, and data for a reproduction time set in advance is reproduced from the start time. Thereafter, the next tag data is acquired, and the operation for reproducing the data for the specified time is repeated.

  When there are a plurality of reference images, a plurality of tag information may exist at the same time. If the time at which a fast-forward operation is performed or the time when reproduction of tag information during fast-forwarding is overlapped with the time of the frame to be reproduced in the next tag information, the corresponding frame is not reproduced. And so on.

[Modification]
The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the embodiment, and the design and the like within the scope not departing from the gist of the present invention are also claimed. include.

  In addition to realizing the functions of the mobile phone described above and other functions in hardware, it can also be realized by loading a computer program having the functions described above into the memory of the computer. This computer program is stored in a recording medium such as a semiconductor memory provided in the mobile phone. Each function described above is realized by being loaded from the recording medium into a memory of a computer built in the mobile phone and executed.

DESCRIPTION OF SYMBOLS 1 Cellular phone 10 Control part 20 Communication part 22 Image input / output part 24 Camera part 26 Display part 30 Image processing part 32 Image encoder part 34 Image decoder part 40 Voice input / output part 42 Microphone 44 Speaker 50 Voice processing part 52 Voice encoder part 54 Audio decoder unit 60 Storage unit 62 Reference image feature DB
64 moving image storage 70 moving image processing unit 72 moving image data storage unit 74 moving image data development unit 80 image analysis unit 82 feature amount calculation unit 84 feature amount comparison unit 86 tag storage unit 88 tag reading unit 90 reproduction instruction unit

Claims (8)

A moving image data storage unit that creates moving image data from an image input from the image input unit and a sound input from the audio input unit and stores the moving image data in the moving image storage; reads out the moving image data from the moving image storage; And a moving image data expansion unit that expands sound and outputs an image to the image output unit and outputs sound to the sound output unit,
A reference image feature storage unit that stores a reference image, a feature amount of the reference image, and a keyword in association with each other;
A feature amount calculation unit for calculating a feature amount of an image included in the moving image data;
A feature amount comparison unit that compares the feature amount calculated by the feature amount calculation unit with the feature amount stored in the reference image feature storage unit;
A tag information storage unit that generates and stores tag information including a keyword of a reference image similar to an image included in the moving image data based on a comparison result of the feature amount comparison unit;
A mobile phone comprising:   The mobile phone according to claim 1, wherein the tag information storage unit stores the start time of moving image data of an image similar to the reference image further included in the tag information.   3. The mobile phone according to claim 2, wherein the tag information storage unit further stores the end time of moving image data of an image similar to the reference image in addition to the tag information.   The mobile phone according to any one of claims 1 to 3, wherein the tag information storage unit stores scene information corresponding to a keyword corresponding to the reference image in tag information.   5. The mobile phone according to claim 1, further comprising: a reproduction instruction unit that instructs the moving image data expansion unit to perform reproduction based on tag information based on the moving image data. phone.   The playback instruction unit identifies a scene including a keyword from the tag information, and issues a playback instruction to the moving image data development unit so that the scene is continuously played back. Item 6. The mobile phone according to Item 5.   The video data expansion unit, in response to a fast-forward instruction from the playback instruction unit, based on the tag information, only the first frame of each tag or only a few seconds of frames from the top of each tag in order according to the time series 6. The mobile phone according to claim 5, wherein data is expanded and output. In a computer equipped with an image output device and an audio output device,
A video data storage function for creating video data from image data and audio data and storing it in the video storage, and reading video data from the video storage, developing images and audio, and outputting the image to an image output device In addition, a program for realizing a video data expansion function for outputting sound to a sound output device,
A reference image feature storage function for storing a reference image, a feature amount of the reference image, and a keyword in association with each other;
A feature amount calculation function for calculating a feature amount of an image included in the moving image data;
A feature amount comparison function for comparing the feature amount calculated by the feature amount calculation function with the feature amount stored by the reference image feature storage function;
A tag information storage function for generating and storing tag information including a keyword of a reference image similar to an image included in the moving image data based on a comparison result of the feature amount comparison function;
A program that realizes

Images