KR20170098139A - Apparatus and method for summarizing image - Google Patents

Abstract

A method for summarizing an image input to an electronic device includes the steps of: obtaining key frames from the image; determining first summary frames among the key frames based on preset criteria; generating multiple pieces of first summary information of the first summary frames; and storing the pieces of first summary information and the first summary frames to the electronic device.

Description

[0001] APPARATUS AND METHOD FOR SUMMARIZING IMAGE [0002]

The present disclosure relates to an image summarizing apparatus and method.

With the development of multimedia technology and network technology, a user can generate an image using the terminal or receive and use an image from another terminal or a service server.

However, as the number of images available to the user increases, it is inconvenient for the user to select the image to be used effectively. However, since the existing summary information providing technology has merely combined a part of existing media, it is difficult for the user to understand the contents of the image with ease. Accordingly, there is a demand for an image summarizing technique that allows the user to efficiently grasp the contents of the image and make the summary information more accessible to the user.

And to provide a method and apparatus for summarizing images.

According to one aspect of the present disclosure, there is provided a method of summarizing an image input to an electronic device, the method comprising: obtaining key frames from the image; Determining first summary frames of the key frames based on a predetermined criterion; Generating a plurality of first summary information of the first summary frames; And storing the first summary frames and the plurality of first summary information in the electronic device.

According to two aspects of the present disclosure, there is provided an electronic device for summarizing an input image, comprising: obtaining key frames from the image; determining first summary frames of the key frames based on a predetermined criterion; A control unit for generating a plurality of first summary information of the summary frames; And a memory for storing the first summary frames and the plurality of first summary information; And an electronic device.

Three aspects of the present disclosure are a computer-readable recording medium having recorded thereon a program for executing the method of the first aspect in a computer.

1 is a block diagram of a user device used for image summarization, in accordance with one embodiment.
2 is a block diagram illustrating components of a user device, in accordance with one embodiment.
3 is a flow diagram illustrating a process for generating first summary frames using key frames, according to one embodiment.
4 is a flow diagram illustrating a process of first summary frames based on video navigation using a user device, in accordance with one embodiment.
5 is a flow diagram illustrating a process of first summary frames based on an action summary search using a user device, in accordance with one embodiment.
6 is a flow diagram illustrating a process for using first summary frames for instantaneous memory, in accordance with one embodiment.
7 is a flow diagram illustrating a process for using first summary frames to optimize storage space, according to one embodiment.
8 is a diagram for explaining providing summarized frames by summarizing images input to an electronic device according to an embodiment.
9 is a flow diagram of a method for generating summary information of summary frames, according to one embodiment.
10 is a flow diagram of a method for representing an example of displaying an image from a selected first summary frame, according to one embodiment.
11 is a diagram illustrating an example of displaying an image from a selected first summary frame, according to one embodiment.
12 is a flowchart illustrating a method of searching for an image, according to an embodiment.
FIG. 13 is a diagram for explaining a method of searching for an image matching a playback period of an image, according to an embodiment.
14 is a diagram illustrating an example of selecting a portion of a first summary frame, according to one embodiment.
15 is a flowchart illustrating a method for generating a master summary for a plurality of images, according to an embodiment.
16 is a diagram illustrating an example illustrating a method for generating a master summary for a plurality of images, according to one embodiment.
17 is a flow diagram of a method for displaying an image from a playback position of a selected summary frame, according to one embodiment.
18 is a flowchart illustrating a method of storing a part of an input image according to an embodiment.
19 is a diagram illustrating an example of selecting a storage method of an input image according to an exemplary embodiment.
20 is a block diagram of an electronic device, in accordance with one embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "comprising ", it means that it can include other elements as well, without departing from the other elements unless specifically stated otherwise.

Also, in this specification, key frames are images inserted at regular time intervals in an image, and summary frames are determined frames among key frames, and the amount of change of images among the key frames is relatively May be many frames. The summary frames may be key frames.

Also, in this specification, the image displayed on the electronic device may include a state in which the image is being reproduced or stopped.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Figure 1 is a block diagram of a user device used for image summarization. User device 101 may be an electronic device capable of storing data in at least one format. User device 101 may include at least one means for storing and capturing media in the at least one of the above-described formats. In addition, the user device 101 may store data in local memory, storage-based clouds, or both. In addition, the user device 101 may include at least one means for reproducing media content to a user. In addition, the user device 101 may support at least one option for the user to interact with the user device 101 to manage data. User device 101 may be a smart phone, a tablet, a personal digital assistant (PDA), and / or the like.

2 is a block diagram illustrating components of a user device, in accordance with one embodiment. The user device 101 includes an input / output interface 201, a video summarization engine 202, a memory module 203, a navigation module 204, a content retrieval module 205, And a master summary generator 206.

The input / output interface 201 is configured to interact with the user device 101 to perform at least one function related to data management, data capture and related activities. The input / output interface 201 may be in a suitable form such as, but not limited to, a keypad and a touch screen display. The input / output interface 201 also provides the user with options to initialize and control functions associated with data capture and management. The input / output interface 201 may be associated with at least one means for capturing media content, or may receive / collect content from an external source. External sources represent things such as the Internet, external hard disks, and the like.

The image summary engine 202 may identify action sequences in the captured image, extract corresponding key frames, and generate summary frames corresponding to the images using the extracted key frames. A 'key frame' may represent a frame representing unique action scenes from the image to be processed. In one example, the image summary engine 202 automatically initializes the summary frames when a new image is collected and stored in the memory module 203. In one embodiment, the video summary engine 202 generates summary frames when receiving input from a user.

The memory module 203 may store media content of different types and other formats in the corresponding media database and may also store media content to other components of the user device 101 when receiving a data request for further processing . In various embodiments, the memory module 203 may be internal or external to the user device 101. Also, the memory module 203 may be fixed size or expandable. In addition, the memory module 203 may store summary frames generated for each image stored in the media database in the same or different database. The memory module 203 may support media content indexing to support fast content retrieval and retrieval.

The navigation module 204 may perform video navigation. The video navigation process may allow the user to quickly access the video with other action sequences. While the video is being played back, the navigation module 204 may identify to the user a bundle of key frames associated with the video based on the summary frames generated and stored for the video in the memory module 203. In addition, the navigation module 204 may collect the input received from the user. The input relates to the selection of a particular keyframe from the keyframes that are part of the bundle being displayed to the user. In addition, the navigation module 204 retransmits a part of the image in which the key frame is displayed to the user.

The content recovery module 205 may collect the search terms from the user. The search term may include at least one portion of at least one type of media file. In one embodiment, the search terms may be generated directly by the user based on the media content shown. For example, while the user is viewing an image, using the appropriate options, the user can select a particular portion of the image and provide the selected portion as a search term. Upon receiving the search term, the content restoring module 205 searches the contents stored in the memory module among the summary images displayed by the image library index as much as possible, and identifies all matching contents. In addition, the content recovery module 205 displays the identified content to the user using the input / output interface 201.

The master summary generator 206 may generate a master summary including summary frames of a plurality of selected images. The master summary generator 206 identifies key frames for the selected images from the summary frames generated for the selected images and generates a master summary for the selected images. In one embodiment, the master summary generator 206 receives user selections related to the images used to generate the master summary. In another example, the master summary generator 206 identifies and selects related content from the memory module 203, and generates a master summary for the selected images. The master summary generator 206 may identify related content based on at least one parameter that is not limited to the period during which the content is created, stored and tagged.

3 is a flowchart showing a process of generating summary frames using key frames. First, the image is selected, and automatically or based on the user's instructions, the image summary engine 202 identifies the frames representing other actions in the selected image (step 302). In addition, the image summary engine 202 extracts the identified frames as key frames corresponding to a specific image (step 304).

After identifying the key frames, the image summary engine 202 generates summary frames from the identified key frame (s) based on one or more predetermined criteria. In one example, the predetermined criteria may be an interest score. The video summary engine 202 determines the interest of the extracted key frames as an interest score (step 306). In one example, the score of interest is determined based on at least one criterion preset by the user.

In work, 'interest' can be determined based on the amount of 'new information' in the keyframe to be considered. Assuming at time T, the M-th key frame is being processed and a dictionary consisting of N-key frames (represented by spatio-temporal features) is available. The M-th key frame is compared with all the contents of the dictionary using a predetermined matching criterion, and the number of matches (N) is identified. If 'N' exceeds the predefined threshold 'T', the interest score of the M-th key frame is set to 'high'. Also, the M-th key frame is added to the dictionary by removing the already existing key frame from the dictionary, and the dictionary is updated accordingly. In one example, the key frame that best matches the rest of the key frame in the dictionary is selected to be removed. In one example, the dictionary is updated based on the interest score of the key frame. For example, an interest score of a new key frame to be considered is compared with an interest score of a key frame having a minimum interest score among all pre-existing key frames. If the score of interest of the new key frame is high, the dictionary is updated by replacing the existing key frame with the new key frame. If the value of N is smaller than the threshold value T, the score of interest of the M-th key frame is set to 'low', and the M-th key frame is not added in advance.

Also, the determined score of interest is compared to the threshold of interest. The threshold of interest is predetermined and preset. If the determined interest score equals or exceeds the threshold, the corresponding key frame is selected to generate summary frames. In addition, using the selected key frame, summary frames are generated (step 310). In method 300, various actions may be performed in a different order or simultaneously, in the order presented. Also, in some embodiments, some of the actions shown in FIG. 3 may be omitted.

4 is a flow chart illustrating processing of summary frames based on video navigation, using a user device. While the selected image is being played back, the navigation module 204 identifies the key frames associated with the image, based on the summary frames generated and stored for the image in the memory module 203. In one embodiment, only key frames with high interest scores are selected and the selected key frames are displayed as a bundle of key frames (step 402). The user can select at least one keyframe from the displayed bundles using the appropriate user interface.

The navigation module 204 receives input according to a user selection of a particular key frame (step 404) and identifies a particular part of the image to be reproduced (step 406). The selected key frame is selected from the image to be reproduced. The navigation module 204 navigates / retransmits to the user a selected portion of the image (step 408). In method 400, various actions may be performed in a different order, or simultaneously, in the order presented. Also, in some embodiments, some of the actions shown in FIG. 4 may be omitted.

5 is a flow diagram illustrating the processing of summary frames based on an action summary search using a user device. At user device 101, content retrieval module 205 may collect a search term from a user (step 502) and the search term may include at least one portion of at least one type of media file. For example, if a user wants to search all images in a media library index, the search term may be part of any image. For example, while viewing a video file, the user can select a particular portion of the video using the appropriate options provided by the content recovery module 205 and the input / output interface 201, Can be provided.

Upon receiving the search term, the content recovery module 205 extracts all the key frames from the search term (step 504), and compares the extracted key frame with the image library index (step 506). By comparing the key frames, the content recovery module identifies (step 508) and restores (step 510) all the matched content in the image library. Also, the identified matches are displayed to the user. For example, if the search term image is a penalty kick in a soccer game, the content retrieval module 205 searches to identify all images in the library having at least one similar key frame (indicating penalty kick) .

In method 500, various actions may be performed in a different order or simultaneously, in the order presented. Also, in some embodiments, some of the actions shown in Figure 5 may be omitted.

6 is a flow chart illustrating processing using summary frames for instantaneous storage. A 'moment recall' indicates a feature that allows collection of summary frames matching an input query, and the input query is an image. User device 101 initiates instant memory by collecting images as query inputs (step 602). The user device 101 compares the input query with a database in the storage space in which the summary frames associated with at least one image are stored (step 604).

By comparing the query entered in the database with the summary frames, at least one image query is identified as being matched with the input query. Any suitable image and / or image processing and comparison algorithm may be used to compare the summarized frames with the input query. In various examples, parameters such as timestamps and geographic tags associated with the entered query, as well as summary frames, are considered to identify the match.

If at least one matching is identified, the identified matching is provided as an output, in a suitable format, using at least one suitable interface, in response to the input query (step 608). If no match is found, a preconfigured message indicating that no results were found is displayed to the user using the appropriate interface (step 610).

In method 600, various actions may be performed in a different order or simultaneously, in the order presented. Also, in some embodiments, some of the actions shown in FIG. 6 may be omitted.

7 is a flow chart illustrating the process of using summary frames for storage space optimization. The user may initiate the video recording using the user device 101 (step 702).

The user device 101 may be implemented to monitor the recording of an image and receives at least one trigger of a predefined kind for optimization of the storage space (step 704). For example, the possible storage space may be a set value, that is, less than or equal to a threshold of a preset storage space in the user device 101. Also, it may be at least one of a combination of manual inputs provided by the user, the possible storage space may be less than a threshold value and / or a predefined event by a user.

Upon receipt of at least one trigger for storage space optimization, the user device 101 generates a summary of the dynamically recorded image (step 706) and stores the summary frames instead of the actual image in the corresponding storage space (step 708 ).

In method 700, various actions may be performed in a different order or simultaneously, in the order presented. Also, in some embodiments, some of the actions shown in FIG. 7 may be omitted.

8 is a diagram for explaining providing summary frames by summarizing an image input to an electronic device, according to an embodiment. The electronic device 1000 may analyze the image and determine frames with relatively large changes in the image as summary frames. The electronic device 1000 may display the image 810 and display summary frames with the image 810. If the user selects any of the summary frames, the electronic device 1000 may play the image 810 from the playback position of the selected summary frame. Since the electronic device 1000 determines and provides the main frames of the image 810 to the user, the user can easily search for the desired playback position in the image 810. [ Referring to FIG. 8, the electronic device 1000 can display an input image 810. [0064] FIG. Also, the electronic device 1000 may display summary frames. In addition, the electronic device 1000 may display summary frames along with the input image 810 on the screen, but is not limited thereto.

The electronic device 1000 may obtain key frames and determine summary frames among the obtained key frames. After receiving the user input 820, the electronic device 1000 may determine the summary frames. That is, when the user touches the displayed icon 821, the electronic device 1000 can display the summary frames. In addition, the electronic device 1000 may receive the user input 820 and then display the determined summary frames on the screen of the input image 810.

The electronic device 1000 may receive a user input 830 that selects any of the displayed summary frames

Electronic device 1000 may generate summary information for summary frames. The summary information includes information about the summary frame. For example, the summary information may include an image file name including a summary frame, a playback position, a playback position of a next key frame, and matching information. Summary information may be generated for each summary frame. For example, the summary information C 840 is information on the summary frame C. The summary information C 840 includes an image file name, a playback position, and matching information regarding the summary frame C. The image file name is an identification value of the image 810, and may be displayed, for example, abc.avi. The playback position of the summary frame C indicates the time at which the summary frame C is reproduced in the image 810. [ The matching information may include key point information, place information, and date and time information, and may further include information necessary for searching for the same or similar summary frame.

The electronic device 1000 may include all devices capable of image processing. The electronic device 1000 may be a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a personal digital assistant (PDA), a laptop, a media player, a micro server, , Navigation, kiosks, MP3 players, digital cameras, consumer electronics, and other mobile or non-mobile computing devices. The device may also be a wearable device, such as a watch, eyeglasses, a hair band, and a ring, with communication and data processing capabilities.

9 is a flow diagram of a method for generating summary information of summary frames, according to one embodiment.

In step 910, the electronic device may obtain key frames from the input image. The image input to the electronic device may be an image generated in the electronic device. For example, the input image may be an image captured by a camera of the electronic device. Further, the image input to the electronic device may be an image received by the electronic device from an external server (for example, a cloud server) or an external electronic device. The image input to the electronic device may include key frames. The key frames included in the input image may be a still image of the image. That is, the key frames may be image files. The key frames acquired by the electronic device can be displayed as a thumbnail.

In step 920, the electronic device may determine the summary frames among the key frames based on predetermined criteria. In one embodiment, the predetermined criteria may be the amount of change in particular key frames as compared to other key frames. For example, among key frames, key frames in which pixel values of a full screen are changed over a predetermined threshold value, key frames in which a new object appears, Key frames may be determined as summary frames. The electronic device may limit the number of summary frames determined when determining summary frames among the key frames within a certain playback interval. For example, the electronic device may determine one summary frame among the key frames within a 10 minute interval of the image.

For example, when N key frames are included in the image input to the electronic device, the electronic device uses the key frames to generate one specific key frame (hereinafter referred to as "A key frame & The similarity between N-1 key frames can be compared. The electronic device can compare the similarity between the key frames using the Spatio-Temporal Feature of the key frames. In addition, the electronic device can compare the similarities between the key frames using the feature points of the key frames. Also, the electronic device can compare the similarity between the key frames using at least one of the time information and the place information included in the key frames. As a result of comparing the similarity of the "A keyframes" with the N-1 keyframes, if the amount of change of the "A keyframes " is determined to be greater than or equal to a preset threshold value, the electronic device determines & . The electronic device can determine the summary frames by comparing the similarity between each of the N key frames included in the input image and the remaining N-1 key frames.

In step 930, the electronic device may generate a plurality of summary information of summary frames. The summary information includes a video file name, a playback position, and matching information.

In step 940, the electronic device may store summary frames and a plurality of summary information in memory. The electronic device may link the summary information to the summary frames.

10 is a flow diagram of a method for representing an example of displaying an image from a selected summary frame, according to one embodiment.

While the image is being displayed at step 1010, the electronic device may display summary frames. In one embodiment, the electronic device may display summary frames along with the input image. In addition, the electronic device may display the determined summary frames along with the input image, corresponding to the user input. For example, summary frames may be displayed at the bottom, left, or right portion of the screen.

In another embodiment, if the summary frames are a plurality, the electronic device may display a portion of the summary frames. In addition, the electronic device can display summary frames that are not displayed, corresponding to user input.

In step 1020, the electronic device may receive an input of a user selecting the displayed summary frame. The electronic device may also receive user input to select a plurality of displayed summary frames.

In step 1030, the electronic device may display an image from the playback position of the selected summary frame. The electronic device may display an image corresponding to the playback position of the selected summary frame, but is not limited thereto. When an image is being reproduced on the electronic device, the electronic device can reproduce an image corresponding to the reproduction position of the selected summary frames. Further, when the image is stopped on the electronic device, the electronic device can display a still image of the image corresponding to the playback position of the selected summary frames.

11 is a diagram illustrating an example of displaying an image from a selected summary frame, according to one embodiment.

Referring to FIG. 11, while the input image 1110a is being reproduced on the electronic device 1000, the electronic device 1000 can display summary frames. A plurality of summary frames may be located at the bottom of the screen. The electronic device 1000 may display summary frames, corresponding to the user input 1120. The electronic device 1000 may receive a user's input 1120 that selects one of the displayed summary frames.

The electronic device 1000 may receive the user input 1120 selecting one of the displayed summary frames and then play the input video 1110b from the playback position of the selected summary frame.

12 is a flowchart illustrating a method of searching for an image, according to an embodiment. Referring to FIG. 12, the electronic device can use the summary information of the summary frame to provide a user with an image similar to the image being reproduced.

In step 1210, the electronic device may receive a user input that selects a first location and a second location in a playback interval of the image. The playback section may be located at the bottom of the image in the form of a bar. In one embodiment, the electronic device can receive user input that selects only the first location in the playback interval of the image. When only the first position is selected, the electronic device can automatically determine the start position of the image as the second position. Further, when only the first position is selected, the electronic device can automatically determine the end position of the image as the second position.

Further, in one embodiment, the electronic device may receive a user input that selects a plurality of first locations and a second set of locations.

In one embodiment, instead of selecting the first and second positions in the playback interval of the image, the electronic device may receive user input that directly selects the two first summary frames of the first summary frames included in the image have. For example, the electronic device may receive a user input that selects two first summary frames of the first summary frames displayed with the image. Of the two first summary frames selected by the user, the reproduction position of the first summary frame having the relatively advanced reproduction position is determined as the first position, and the reproduction position of the remaining first summary frame is determined as the second position.

The first summary frames are selected from the frames of the image being reproduced. The second summary frames are selected from the frames of the image stored in the memory. In yet another embodiment, the second summary frames may be selected in an interval that is not specified by the user in the image being played back.

In step 1220, the electronic device may extract, among the first summary frames, first summary frames included between the selected locations. In addition, the electronic device may display the extracted first summary frames. In addition, the extracted first summary frames may include an identification value that is distinguished from the first summary frames that have not been extracted.

Further, in one embodiment, when a plurality of first positions and a second set of positions are received, the electronic device may extract first summary frames included in each set. In addition, the electronic device may display the extracted first summary frames. In addition, the extracted first summary frames may include an identification value that is distinguished from the first summary frames that have not been extracted. In addition, the first summary frames included in each set may include an identification value that is distinct from the first set of summary frames.

In one embodiment, when receiving a user input that directly selects two first summary frames instead of selecting a first position and a second position in a playback interval of the image, the electronic device determines that the two selected first summary frames It is possible to extract first summary frames included between playback positions.

In step 1230, the electronic device may obtain summary information for the extracted first summary frames. And obtain first summary information for each of the first summary frames. In step 1240, the electronic device may obtain a plurality of second summary information from the plurality of images stored in the electronic device. The electronic device may also obtain second summary information in an image that includes first summary frames. The electronic device may obtain second summary information for the frames in the image, excluding the frames included between the first position and the second position. The electronic device may obtain second summary frames from the key frames included in the plurality of images. The electronic device may generate a plurality of second summary information of the second summary frames and then obtain a second plurality of generated summary information. The plurality of second summary information may include the same kind of information as the plurality of first summary information.

In step 1250, the electronic device may retrieve, from among the plurality of second summary information, second summary information that matches the plurality of first summary information. The electronic device can retrieve the second summary information matching the plurality of first summary information using the plurality of first summary information and the matching information included in the plurality of second summary information.

In one embodiment, the electronic device may utilize the vision recognition to retrieve second summary information that matches a plurality of first summary information. The electronic device can match the plurality of first summary information and the plurality of second summary information using the minutia information included in the plurality of first summary information and the plurality of second summary information. The method of performing matching using minutia information includes, but is not limited to, Harris corner, Shi & Tomasi, SIFT DoG, FAST, or AGAST algorithm. In addition, the electronic device can retrieve second summary information that matches a plurality of first summary information using face recognition and region tracking algorithms.

In another embodiment, the electronic device stores second summary information that matches a plurality of first summary information using location information included in the plurality of first summary information, the plurality of second summary information, and date and time information You can search. The electronic device can retrieve a plurality of second summary information including place information that matches the place information included in the plurality of first summary information. In addition, the electronic device can retrieve a plurality of second summary information including date and time information matched with date and time information included in the plurality of first summary information. The place information may be global positioning system (GPS) information of a place where a video including a plurality of first summary information is captured. The date and time information may be the date and time information on which the image including the plurality of first summary information was photographed. However, the place information and the date and time information are not limited thereto.

In one embodiment, the electronic device may receive an input of a user selecting a portion of the region of the first summary frames. If a portion of the first summary frames is selected, the electronic device identifies a plurality of first summary information of the first summary frames corresponding to the selected region, and retrieves second summary information that matches the identified first summary information . For example, the electronic device can search for a plurality of second summary information that match using only the feature point information of the selected region, but is not limited thereto.

In step 1260, the electronic device may display a plurality of images represented by the retrieved second summary information. In one embodiment, the electronic device may display second summary frames corresponding to the retrieved second summary information. The electronic device can display the second summary frames by dividing the screen. For example, the electronic device can divide the screen into 12 areas and display 12 second summary frames. The user can select any of the displayed second summary frames and the electronic device can play back the video including the selected second summary frame. At this time, the electronic device can reproduce an image from the position of the second summary frame.

In one embodiment, the electronic device may display a plurality of second summary information based on a matching value of the plurality of second summary information and the plurality of first summary information. The electronic device may calculate a matching value of the plurality of second summary information. The electronic device may display an image including a plurality of second summary information satisfying a predetermined condition. For example, the larger the matching value, the more the second summary information may have a higher degree of matching with the plurality of first summary information. The electronic device may display an image including a plurality of second summary information having a matching value equal to or greater than a threshold value. Also, for example, the electronic device may preferentially display an image including a plurality of second summary information having a high matching value.

FIG. 13 is a diagram for explaining a method of searching for an image matching a playback period of an image, according to an embodiment.

13, the electronic device 1000 may receive a user input that selects a first location 1310 and a second location 1320 of an image playback interval. The electronic device 1000 may extract first summary frames included between selected locations and the extracted first summary frames may include a separate identification value 1340. [

The electronic device 1000 may obtain a plurality of first summary information 1330 of extracted first summary frames. The plurality of first summary information may include a video file name, a playback position, and matching information. The matching information may include minutia information, time information, and place information.

The electronic device 1000 may search second summary information matching a plurality of first summary information 1330 from a plurality of second summary information stored in the memory 1340. [

14 is a diagram illustrating an example of selecting a portion of a first summary frame, according to one embodiment.

The electronic device 1000 may receive a user input 1430 that selects a partial region 1420 of the first summary frame 1410. [ As shown in FIG. 14, the user can touch and drag the selected area 1420 to select it.

In addition, the electronic device 1000 may obtain a plurality of first summary information corresponding to a portion of the selected first summary frames. The obtained plurality of first summary information may be minutia information on a partial area of the selected first summary frames, but is not limited thereto.

In addition, after the above-described steps are performed, the same steps as steps 540 to 560 of FIG. 5 may be performed, but their descriptions will be omitted for simplicity.

In one embodiment, the electronic device 1000 may use the minutia information of the selected partial area 1420 to identify that the selected partial area 1420 contains a face. The electronic device 1000 may search for an image that includes a frame that matches the identified face. For example, the face recognition algorithm can be used as a method of searching for an image including a frame matching the identified face. The electronic device may detect a face in a selected region 1420, extract the feature of the detected face using the feature point information, and then search second summary information including information matching the extracted face feature.

15 is a flowchart illustrating a method for generating a master summary for a plurality of images, according to an embodiment. Referring to FIG. 15, the electronic device can extract a part of images and combine the extracted images to generate a master summary. The user can reproduce the master summary to view the main portions of the plurality of images in a short time.

In step 1510, the electronic device may obtain a summary frame of the image. In one embodiment, the electronic device obtains a summary frame of a plurality of images. For example, the plurality of images may be images photographed within a period designated by the user, or images selected by the user. Or a plurality of images may be images included in the same folder. Or a plurality of images may be images including the same / similar file name.

In step 1520, the electronic device may extract summary images of the image using the summary frame. In one embodiment, the electronic device extracts the summary images of the plurality of images using the summary frame. The electronic device can extract the images from the reproduction position of each summary frame to the reproduction position of the next key frame to extract the summary images.

In step 1530, the electronic device merges the extracted summary images to generate a master summary. For example, the electronic device may temporally prioritize the summary image of the input image temporally ahead of the master summary.

16 is a diagram illustrating an example illustrating a method for generating a master summary for a plurality of images, according to one embodiment.

Referring to FIG. 16, an electronic device may include a plurality of images 1610 stored in a memory. The electronic device can obtain the summary frames 1630 contained in the image 1620 generated during a specific period from among the plurality of images 1610. [ For example, the user can select images taken during a recent trip, from a plurality of images stored in the electronic device, and the electronic device can acquire a summary frame of images selected by the user.

The electronic device can extract the summary image using the obtained summary frames 1630. [ The electronic device merges the extracted summary images to generate a master summary, thereby providing partial images of interest to the user in the form of a single image file.

17 is a flow diagram of a method for displaying an image from a playback position of a selected summary frame, according to one embodiment. The electronic device 1000 can display summary frames and play back the video from the playback position of the summary frame selected by the user.

The electronic device 1000 may display a plurality of stored summary frames. The electronic device 1000 may display a plurality of summary frames included in one image file. Alternatively, the electronic device 1000 may display summary frames representative of each of a plurality of images. In one embodiment, the electronic device 1000 may display summary frames based on predetermined criteria. For example, the electronic device 1000 may display summary frames in the order in which they are played back in the video. The electronic device 1000 may determine a position at which to display summary frames for a plurality of images in order of date in which a plurality of images are stored.

In addition, the electronic device 1000 may receive an input 1710 of the user selecting the displayed summary frames. The electronic device 1000 plays the video from the playback position 1720 of the displayed summary frame when the user's input 1710 is received.

In addition, electronic device 1000 may obtain a plurality of first summary information of selected summary frames. The plurality of summary information may include, but is not limited to, the playback position information of the summary frames.

Also, the electronic device 1000 can display an image from the playback position 1720 included in the plurality of summary information, but the position at which the image is displayed is not limited thereto.

18 is a flowchart illustrating a method of storing a part of an input image according to an embodiment. Referring to FIG. 18, when the storage space is insufficient, the electronic device can store only a part of the captured image.

In step 1810, the electronic device may determine if the storage space is below a predetermined threshold. The storage space may be, but is not limited to, a memory of the electronic device. If the storage space is not determined to be equal to or less than the preset threshold value, the electronic device can store the entire input image in the storage space.

If it is determined to be equal to or less than the predetermined threshold value, the electronic device can provide notification information to the user. The electronic device may proceed to step 1820 in response to the user's input to the notification information. However, in one embodiment, even if the storage space is not determined to be equal to or less than the predetermined threshold value, the process may proceed to step 1820 corresponding to the user's input.

In step 1820, the electronic device may store summary frames and a plurality of summary information from the input image data. In one embodiment, in response to a user's input to the announcement information, the electronic device may receive an input that stores only summary frames and a plurality of summary information in a storage space. Also, corresponding to the user's input to the announcement information, the electronic device may delete the image data except the summary frames and the plurality of summary information from the storage space.

19 is a diagram illustrating an example of selecting a storage method of an input image according to an exemplary embodiment.

Referring to FIG. 19, a user can take an image using an electronic device. The electronic device may receive a user input that selects the summary frames mode 1910 from among a plurality of image capture modes. The electronic device may store in the storage space the summary frames and the plurality of summary information obtained from the image at the time of imaging, corresponding to the user input selecting the summary frames mode 1910. [ In addition, the electronic device may delete, in the storage space, image data, excluding summary frames and a plurality of summary information, corresponding to a user input selecting the summary frames mode 1910. [

20 is a block diagram of an electronic device, in accordance with one embodiment.

20, an electronic device 2000 according to an embodiment may include a control unit 2100, a display 2200, a communication unit 2300, and a memory 2400. [ However, not all of the components shown in Fig. 20 are essential components of the electronic device 2000. The electronic device 2000 may be implemented by more components than the components shown in Fig. 20, and the electronic device 2000 may be implemented by fewer components than those shown in Fig.

For example, the electronic device 2000 according to one embodiment may further include a user input unit, an output unit, a sensing unit, and an A / V input unit.

The user input unit may mean a means for the user to input data for controlling the electronic device 2000. [ For example, the user input may include a key pad, a dome switch, a touch pad (contact type capacitance type, pressure type resistive type, infrared ray detection type, surface ultrasonic wave conduction type, , Piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

The output unit may output an audio signal, a video signal, or a vibration signal.

The display 2200 can display and output information processed in the electronic device 2000. For example, the display 2200 can display an image input to the electronic device 2000. [

Meanwhile, when the display 2200 and the touch pad have a layer structure and are configured as a touch screen, the display 2200 can be used as an input device in addition to the output device. The display 2200 may be a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display display, and electrophoretic display. And the electronic device 2000 may include two or more displays 2200, depending on the implementation of the electronic device 2000. [ At this time, the two or more displays 2200 may be arranged to face each other using a hinge.

The control unit 2100 can generally control the overall operation of the electronic device 2000. For example, the control unit 2100 can generally control the user input unit, the output unit, the sensing unit, the communication unit, the A / V input unit, and the like by executing the programs stored in the memory 2400. The control unit 2100 may control the user input unit, the output unit, the sensing unit, the communication unit, and the A / V input unit in order to perform the operation of the electronic device 2000 in FIGS.

The steps of FIGS. 1 to 19 described above in this specification can be performed by the electronic device 2000 shown in FIG.

The control unit 2100 may analyze the image and determine frames having a relatively large change in the image as a summary frame. Display 2200 can display an image and display summary frames with the image. When the user receives an input for selecting one of the summary frames in the communication unit 2300, the control unit 2100 can reproduce the video from the playback position of the selected summary frame. Since the control unit 2100 determines the main frames of the video, i.e., the summary frames, and provides the main frames to the user, the user can easily search the desired playback position in the video.

In addition, the control unit 2100 may generate summary information for each of the determined summary frames, and may store summary frames and summary information in the memory 2400. The control unit 2100 can search for an image similar to the inputted image and generate a master summary using the summary frame and the summary information stored in the memory 2400. In addition, Images can be displayed.

21 is a flowchart illustrating a method of displaying an image by an electronic device according to an embodiment. Referring to FIG. 21, an electronic device can search for similar images in one image and provide them to a user.

In step 2110, the electronic device receives a user input for selecting a first position and a second position in a playback interval of the image.

At step 2120, the electronic device obtains first summary information for the frames included between the first and second locations. The first summary information may be information representative of frames included between the first location and the second location. Alternatively, the first summary information may be information on each of the frames included between the first position and the second position.

At step 2130, the electronic device obtains at least one second summary information for frames excluding frames included between the first location and the second location in the image. The electronic device acquires second summary information on an image of a section excluding a section selected by the user in one image. The electronic device can divide an image of a section excluding the section selected by the user into a plurality of sections and obtain second summary information on the frames included in each section.

The first summary information and the second summary information may include a feature, a shape, an arrangement, a movement, and the like of the object included in the image.

In step 2140, the electronic device retrieves, from at least one second summary information, second summary information that matches the first summary information. The electronic device retrieves the second summary information in which the characteristics, shape, arrangement, and movement of the object are most consistent.

The electronic device may determine a summary frame of the image corresponding to the retrieved second summary information. The electronic device can determine the summary frame among the frames included in the image corresponding to the second summary information.

In step 2150, the electronic device displays an image corresponding to the retrieved second summary information. The electronic device can display the first frame of the video corresponding to the second summary information on the full screen or display the first frame on a part of the screen.

In addition, the electronic device may display a summary frame of the image corresponding to the retrieved second summary information. When the user selects the summary frame, the electronic device reproduces the video corresponding to the summary frame. The electronic device can reproduce the image from the summary frame or reproduce the image from the first frame.

When two or more images corresponding to the second summary information are searched, the electronic device can display two or more images in chronological order. The electronic device may display the first frames of two or more images. In addition, the electronic device may display summary frames of two or more images.

Some embodiments may also be implemented in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes any information delivery media, including computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism.

Also, in this specification, the term "part" may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Claims (20)

A method for summarizing an image input to an electronic device,
Obtaining key frames from the image;
Determining first summary frames of the key frames based on a predetermined criterion;
Generating a plurality of first summary information of the first summary frames; And
Storing the first summary frames and the plurality of first summary information in the electronic device;
/ RTI > The method according to claim 1,
The key frames are obtained according to a predetermined time interval,
Wherein the first summary frames are determined based on an amount of change of the key frames among the key frames. The method according to claim 1,
Displaying the first summary frames while the image is being displayed;
Receiving a user input that selects a first summary frame of any of the displayed first summary frames; And
Displaying the image from the playback position of the selected first summary frame;
Further comprising:
Wherein the playback position is included in the first summary information of the selected first summary frame. The method according to claim 1,
Receiving a user input for selecting a first location and a second location in the video playback section;
Extracting, among the first summary frames, first summary frames included between the first location and the second location;
Obtaining first summary information for the extracted first summary frames;
Obtaining a plurality of second summary information from a plurality of images stored in the electronic device;
Searching second summary information matching the plurality of first summary information from the plurality of second summary information; And
Displaying the stored plurality of images including each of the retrieved second summary information;
≪ / RTI > The method according to claim 1,
Receiving a user input for selecting a portion of the stored first summary frames;
Obtaining the plurality of first summary information corresponding to a partial area of the selected first summary frames;
Obtaining a plurality of second summary information from a plurality of images stored in the electronic device;
Searching second summary information matching the plurality of first summary information from the plurality of second summary information; And
Displaying the stored plurality of images including each of the retrieved second summary information;
≪ / RTI > The method according to claim 1,
Obtaining first summary frames of the image;
Extracting summary images of the image using the obtained first summary frames; And
Merging the summary images to generate a master summary;
Further comprising:
Wherein the playback interval is included in the first summary information of the first summary frames. The method according to claim 1,
Displaying the stored plurality of first summary frames on the electronic device;
Receiving an input of a user selecting the displayed first summary frames; And
Displaying the image from the playback position of the selected first summary frame;
Lt; / RTI >
Wherein the playback position is included in the first summary information of the selected first summary frame. The method according to claim 1,
Wherein the storing of the first summary frames and the plurality of first summary information in the electronic device comprises:
Determining whether the storage space in the electronic device is less than or equal to a preset threshold value; And
Storing only the first summary frames and the plurality of first summary information in the electronic device from the data included in the input image if the storage space is determined to be equal to or less than the preset threshold value;
≪ / RTI > An electronic device for summarizing an input image,
Obtaining key frames from the image,
Determining first summary frames of the key frames based on a predetermined criterion,
A controller for generating a plurality of first summary information of the first summary frames; And
A memory for storing the first summary frames and the plurality of first summary information;
. 10. The method of claim 9,
The key frames are obtained according to a predetermined time interval,
Wherein the first summary frames are determined based on an amount of change of the key frames among the key frames. 10. The method of claim 9,
A display for displaying the first summary frames with the image; And
An input for receiving a user input for selecting the displayed first summary frame;
Further comprising:
Wherein the control unit controls the display to display the image from the playback position of the selected first summary frame. 10. The method of claim 9,
Further comprising an input unit for receiving a user input for selecting a first position and a second position of the image playback section,
Wherein the controller extracts first summary frames included between the first position and the second position among the first summary frames,
Acquiring first summary information on the extracted first summary frames,
Acquiring a plurality of second summary information from a plurality of images stored in the electronic device,
Searching second summary information matching the plurality of first summary information from the plurality of second summary information,
And displaying each of the stored plurality of images including each of the retrieved second summary information. 13. The method of claim 12,
Wherein the controller acquires a playback interval of each of the first summary frames,
Acquiring a partial image of the image corresponding to the playback section,
And merging the partial images to generate a master summary. 10. The method of claim 9,
Wherein the controller determines whether the storage space in the electronic device is less than or equal to a preset threshold value,
Wherein the memory stores only the first summary frames and the plurality of first summary information among data included in the input image if the storage space is determined to be equal to or less than the preset threshold value. Memory;
A control unit;
An input unit; And
Display,
Wherein the input unit receives a user input for selecting a first position and a second position in a playback interval of an image,
Wherein the control unit obtains first summary information on the frames included between the first position and the second position, and obtains first summary information on the frames excluding frames included between the first position and the second position Acquiring at least one second summary information for at least one of the first summary information and the second summary information matching the first summary information,
Wherein the display displays an image corresponding to the retrieved second summary information. 16. The method of claim 15,
Wherein the controller determines a summary frame of the image corresponding to the second summary information,
Wherein the display displays the summary frame. 16. The method of claim 15,
Wherein the control unit controls the display to display the at least two images in chronological order when two or more images corresponding to the second summary information are searched. A method of displaying an image in an electronic device,
Receiving a user input for selecting a first position and a second position in a playback interval of an image;
Obtaining first summary information for frames included between the first location and the second location;
Obtaining at least one second summary information on frames in the image excluding frames included between the first location and the second location;
Retrieving, from the at least one second summary information, second summary information matching the first summary information; And
And displaying an image corresponding to the retrieved second summary information. 19. The method of claim 18,
Further comprising determining a summary frame of the image corresponding to the retrieved second summary information,
Wherein the displaying step displays the summary frame. 19. The method of claim 18,
Wherein when displaying two or more images corresponding to the second summary information, the displaying step displays the two or more images in chronological order.

Images