JP2011170424A - Mobile terminal and image classification method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile terminal and an image classification method which allows quick classification of a plurality of images captured by a consecutive photographic functions into groups, in a compact apparatus with the processing capability restricted. <P>SOLUTION: The mobile terminal 1 includes an imaging section 101 and a group calculation section 105. The imaging section 101 generates a captured image. The group calculation section 105 generates the groups for classifying the plurality of captured images generated by the imaging section 101 and calculates a group to which each captured image belongs. When the group is calculated, the group calculation section 105 determines that an inputted second captured image belongs to a first group, or that the second captured image belongs to a newly generated second group, based on a first captured image that is one which the captured image belongs to the first group generated most recently. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a portable terminal and an image classification method, and more particularly to a portable terminal and an image classification method related to a captured image captured by a camera function.

  In recent years, with the widespread use of digital cameras, camera-equipped mobile phones, and the like, users can easily perform camera photography using portable portable devices. Many of these small devices have an advanced photographing function such as a continuous shooting function. As a result, a large number of captured images are stored in a small device, and it is difficult for a user to search for a desired image. Further, there has been a problem that the user needs to select an image to be saved from a large number of captured images.

  The electronic album device described in Patent Document 1 calculates feature amounts of a plurality of stored images, collates the features of each image, and classifies similar images as a common group. By such group classification, the user can refer to images classified for each group. Thereby, the number of images to be selected can be significantly reduced.

  The content classification device described in Patent Document 2 is a device for classifying a plurality of images taken by a digital camera. The content classification device defines rules for classifying a plurality of images, and classifies the input images into groups according to the rules. As an example of the rule, there is a rule that the input image is classified into a group having a feature amount closest to the representative image of each group.

Japanese Patent Laid-Open No. 2002-010196 JP 2004-280254 A

  However, depending on the technique according to the above-mentioned patent document, the following problems occur. As described above, many camera-equipped mobile phones and the like in recent years have a continuous shooting function. In general, small devices such as camera-equipped mobile phones have limited processing capabilities compared to ordinary PCs (Personal Computers). Therefore, it is necessary to quickly process a plurality of images captured by the continuous shooting function in these devices with limited processing capabilities.

  However, in the technique described in Patent Document 1, it is necessary to calculate the feature amount of each image and collate the feature amounts with each other. For this reason, if such processing is performed on a cellular phone or the like with limited processing capability, the processing time is increased and the operability for the user is deteriorated. Further, even with the technique described in Patent Document 2, it is necessary to determine whether or not an input image belongs to each group. Therefore, in the same manner as the technique described in Patent Document 1, the operability of the user is deteriorated in a mobile phone or the like whose processing capability is limited.

  The present invention has been made to solve such a problem, and provides a portable terminal and an image classification method capable of quickly classifying a plurality of images captured by a continuous shooting function into small groups in a small device with limited processing capability. The purpose is to provide.

  One aspect of the portable terminal according to the present invention is an image capturing unit that generates a captured image, and a group that generates a group for classifying a plurality of captured images generated by the image capturing unit and calculates a group to which each captured image belongs. A calculation unit, wherein the group calculation unit receives an input second captured image based on a first captured image that is any one captured image belonging to the first group generated immediately before. It is determined that it belongs to the first group or that the second captured image belongs to a newly generated second group.

  One aspect of the image classification method according to the present invention is an image classification method for generating a group for classifying a plurality of captured images and calculating a group to which each captured image belongs. Based on the first captured image that is one of the images belonging to the group, the input second captured image belongs to the first group, or the second captured image is newly generated. And a grouping step for determining to belong to the second group.

  According to the present invention, it is possible to provide a portable terminal and an image classification method that can quickly classify a plurality of images captured by the continuous shooting function into small groups in a small device with limited processing capability.

1 is a block diagram of a mobile terminal according to a first exemplary embodiment. It is a figure which shows the concept of the adaptability concerning Embodiment 1. FIG. FIG. 3 is a diagram of an image displayed on the display unit according to the first embodiment. 3 is a flowchart showing a processing procedure of the mobile terminal according to the first exemplary embodiment; 3 is a flowchart showing a processing procedure of the mobile terminal according to the first exemplary embodiment; 1 is a block diagram of a mobile terminal according to a first exemplary embodiment.

Embodiment 1
Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the mobile terminal 1 according to the present embodiment. The mobile terminal 1 includes an imaging unit 101, a fitness level assigning unit 102, a buffer 103, a buffer 104, a group calculation unit 105, a reduction processing unit 106, an encoding unit 107, and a manual storage processing unit 110. Prepare. The manual storage processing unit 110 includes a primary storage memory 111, a representative image extraction unit 112, and a display unit 113. The mobile terminal 1 has a connection interface with the flash memory 201.

  The imaging unit 101 is a processing unit including a camera function for capturing a moving image and a still image. The camera function includes a continuous shooting function. Here, the continuous shooting function is a concept including a function of capturing a plurality of images when a shutter button is pressed once and a function of capturing an image at a timing when the shutter button is pressed quickly. . The imaging unit 101 stores the captured image in the buffer 103 or the buffer 104. At this time, the imaging unit 101 inputs a captured image to the buffer specified by the group calculation unit 105.

  Note that the imaging unit 101 may change the output destination of the image depending on whether or not it is a mode for performing continuous imaging. For example, in a mode in which continuous imaging is not performed, the imaging unit 101 may store the captured image in the flash memory 201 as it is.

  The fitness level assigning unit 102 is a processing unit that gives a fitness level to each of the images captured by the imaging unit 101. The degree of conformity is information serving as an index indicating the conformity to a desired condition. For example, the fitness is information such as whether the person to be imaged is smiling or whether it is a high-quality image without camera shake. The fitness level assigning unit 102 analyzes the image output from the imaging unit 101 and gives the fitness level. The fitness level assigning unit 102 can also give other metadata to the image.

  FIG. 2 is a diagram illustrating the concept of an image to which a fitness level is given. The fitness level assigning unit 102 assigns a numerical value indicating the fitness level for each fitness level item to the image captured by the imaging unit 101. For example, a smile index “10” and a good image index “80” are assigned to the image 1 in FIG.

  The buffer 103 and the buffer 104 are storage areas in which images to which the fitness level is attached are stored. One of the buffer 103 and the buffer 104 stores an image (first captured image) used for grouping images (second captured images) to be input later. A group number is assigned to the image (first captured image) used for grouping by a group calculation unit 105 described later.

  The group calculation unit 105 is a processing unit for grouping images captured by the imaging unit 101 (giving a group number to an image). The group calculation unit 105 reads the images from the buffer 103 and the buffer 104, and calculates the similarity between the two images. One of the images read by the group calculation unit 105 is assigned a group number. The similarity is calculated by calculating pixel values of both images and comparing the pixel values. Alternatively, a plurality of pixels may be extracted from both images, an average value of the plurality of pixels may be calculated, and the calculated average values may be compared with each other.

  When the calculated similarity is sufficiently high, the group calculation unit 105 assigns the same number as the group number assigned to one image to the other image that is not assigned a group number. Then, the camera unit 101 is notified of the buffer in which the image with the new group number is stored as the input destination of the next image.

  If the calculated degree of similarity is not high, the group calculation unit 105 assigns a number different from the group number assigned to one image to the other image not assigned a group number. In this example, the group calculation unit 105 assigns a value obtained by adding 1 to the group number assigned to one image to the other image that is not assigned a group number. Then, the camera unit 101 is notified of a buffer different from the buffer in which the image having the new group number is stored as the input destination of the next image.

  Note that a threshold specified by the user can be used to determine whether the similarity is high. For example, if the user sets a high threshold value, the pixel values of both images to be compared are not determined to be similar unless the pixel values of the images to be compared are considerably similar.

  A specific operation example will be described below. The first image captured by the continuous shooting function is stored in the buffer 103. The group calculation unit 105 assigns group number 1 to the first image. In addition, the group calculation unit 105 notifies the imaging unit 101 that the subsequently input image is stored in the buffer 104.

  Subsequently, the captured second image is stored in the buffer 104. The group calculation unit 105 reads images from the buffer 103 and the buffer 104, and calculates the similarity between them. If the similarity between the two is high, the group calculation unit 105 assigns the group number 1 to the second image (the image stored in the buffer 104). The group calculation unit 105 then outputs the image (second image) to the reduction processing unit 106 and the encoding unit 107. In this case, the group calculation unit 105 does not notify the imaging unit 101 because the input destination of the subsequently input image (third image) is not changed.

  Subsequently, the third image is stored in the buffer 104. The group calculation unit 105 reads images from the buffer 103 and the buffer 104, and calculates the similarity between them. If the similarity between the two is low, the group calculation unit 105 assigns a new group number of 2 to the third image (the image stored in the buffer 104). Then, the group calculation unit 105 outputs the image (first image) stored in the buffer 103 to the reduction processing unit 106 and the encoding unit 107. In addition, the group calculation unit 105 notifies the imaging unit 101 that the subsequently input image (fourth image) is stored in the buffer 103.

  As described above, when images belonging to the same group are stored in both buffers, the group calculation unit 105 leaves the previously input image in the buffer. In general, when images are captured by the continuous shooting function, the similarity between the images before and after increases. Therefore, it is possible to accurately determine whether or not the similarity is high by leaving the previously input image in the buffer.

  Note that the processing related to group number assignment by the group calculation unit 105 is not limited to this. For example, the first image and the second image are compared, and a group number is assigned to the second image as described above. Next, the group calculation unit 105 automatically assigns the same group number as the group number assigned to the second image to the third to fifth images. Subsequently, the group calculation unit 105 compares the sixth image with the image remaining in the buffer, and assigns a group number to the sixth image. The group calculation unit 105 automatically assigns the same group number as the group number assigned to the sixth image to the seventh to ninth images. In this way, the processing speed can be improved by reducing the comparison of images.

  The reduction processing unit 106 is a processing unit that generates a thumbnail image of the image to which the group calculation unit 105 has assigned a group. The reduction processing unit 106 outputs the generated thumbnail image to the encoding unit 107.

  The encoding unit 107 encodes the image input from the group calculation unit 105 and the thumbnail image input from the reduction processing unit 106. For example, the encoding unit 107 encodes the image into an image in JPEG (Joint Photographic Experts Group) format. The encoding unit 107 changes the output destination of the encoded image depending on whether or not the mobile phone 1 is in the automatic storage mode.

  The automatic storage mode is an operation mode in which captured images are automatically stored in a storage device such as the flash memory 201. When the automatic save mode is set, the user cannot select an image to be saved. On the other hand, the manual storage mode is an operation mode in which captured images are stored in a temporary memory, and a user selects an image to be stored in the flash memory 201 or the like from images stored in a primary memory. The user can specify in which operation mode the mobile terminal 1 is operated.

  When the automatic save mode is set, the encoding unit 107 stores the encoded image in the flash memory 201. Note that the encoding unit 107 can store an image in an arbitrary storage device without being limited to the flash memory 201. When the manual storage mode is set, the encoding unit 107 outputs the image subjected to the encoding process to the primary storage memory 111.

  The primary storage memory 111 is a storage device in which images are temporarily stored when the manual storage mode is set. The image stored in the primary storage memory 111 is given a group number and fitness information.

  The representative image extraction unit 112 is a processing unit that extracts a representative image representing a group from a plurality of images belonging to the same group based on the fitness information. For example, the representative image extraction unit 112 sets the image having the highest smile index among the images in the group as the representative image of the group. The user can set which fitness level information is used to determine the representative image. Furthermore, the representative image extraction unit 112 can add a plurality of numerical values of the fitness information and determine an image having the highest added value as a representative image.

  The display unit 113 is an information display unit of the mobile terminal 1 and is, for example, a liquid crystal display. The display unit 113 displays thumbnail images of representative images of each group in a list format. Here, the representative images of each group are displayed in the order of imaging time. The order of the imaging times can be easily determined because the group numbers are given in time series. An example of the screen displayed on the display unit 113 is shown in FIG. The list screen displays representative images of each group and imaging information (for example, imaging time, photographer information, etc.). The user can select an arbitrary group with an operation button (not shown).

  For example, when the user selects the group 2, the images of the group 2 are displayed on the display unit 113 in order of high fitness. The user can change the fitness information used to determine the display order. For example, the user can rearrange and display an image group arranged in the order of higher smile index in the order of images with less camera shake. The user selects an image to be saved by operating an operation button or the like. A save flag is set for the image to be saved. The user instructs execution of the saving process. The image for which the save flag is set when the save process is executed is read from the primary storage memory 111 and written to the flash memory 201. Note that the user can instruct the cancellation of the storage flag at any time before the execution of the storage process.

  The user can also instruct the change of the fitness information used for determining the display order after setting the save flag for the image. In this case, a display effect such as greening is applied to the image in which the storage flag is set so that the storage flag is set.

  The flash memory 201 is a secondary storage device used for storing data in the mobile terminal 1. Examples of the flash memory 201 include smart media, compact flash (registered trademark), SD memory card, and the like.

  Next, with reference to the flowchart of FIG. 4, processing when the automatic storage mode of the mobile terminal according to the present embodiment is set will be described. First, the camera function of the mobile terminal 1 is activated (S101). The imaging unit 101 generates a captured image in response to a user's shutter button operation (S102). The fitness level assigning unit 102 reads each captured image and gives the fitness level (S103).

  The imaging unit 101 stores the image to which the fitness level is given in the buffer designated by the group calculation unit 105 (S104). The group calculation unit 105 reads images from the buffer 103 and the buffer 104, and assigns group numbers to the images (S105). The group calculation unit 105 outputs the image assigned with the group number to the reduction processing unit 106 and the encoding unit 107.

  The reduction processing unit 106 generates a thumbnail image of the input image (S106). The reduction processing unit 106 outputs the generated thumbnail image to the encoding unit 107. The encoding unit 107 encodes the image input from the group calculation unit 105 and the thumbnail image input from the reduction processing unit 106 (S107). The encoding unit 107 writes the image subjected to the encoding process in the flash memory 201 (S108). The image written in the flash memory 201 is given a fitness level and a group number.

  Next, with reference to the flowchart of FIG. 5, the process at the time of manual save mode setting of the portable terminal concerning this Embodiment is demonstrated. In FIG. 5, the processes denoted by the same reference numerals as those in FIG. 4 perform basically the same processes as those described above.

  Each processing unit of the mobile terminal 1 assigns a fitness level and a group number to the image captured by the imaging unit 101 (S101 to S105). Then, the reduction processing unit 106 generates a thumbnail image of the image (S106). The encoding unit 107 receives the image and the thumbnail image. The encoding unit 107 encodes these two images (S107).

  The encoding unit 107 stores the encoded image in the primary storage memory 111 (S110). In this way, the image captured by the image capturing unit 101 is stored in the primary storage memory 111. The processes of S102 to S110 are continued until the user instructs the end of imaging (S111).

  When the user has instructed the end of imaging (S111: Yes), the representative image extraction unit 112 extracts a representative image of each group (S112). A list image to be displayed on the display unit 113 is generated by using the extracted representative image information, and the list image is displayed on the display unit 113 (S113).

  When the user instructs to change the fitness information for determining the representative image (S114: Yes), the representative image is extracted and the display screen is regenerated (S112, S113). When the user has not instructed to change the fitness information for determining the representative image (S114: No), the user selects an image to be saved from the list screen (S115). Then, the selected image is written into the flash memory 201 (S116).

  Then, the effect of the portable terminal 1 concerning this Embodiment is demonstrated. As described above, the group calculation unit 105 has a relationship with an image (first captured image) that has been previously assigned a group number, for example, a newly input image (second captured image) by comparison of similarities. The group to which it belongs is determined. Here, the comparison can be performed only by comparing two images, and the processing load of image comparison can be reduced. Therefore, even when continuous imaging is performed, image grouping can be performed quickly. Since image grouping can be performed at high speed, the operability of the user is not impaired even in a portable terminal or the like with limited processing capability.

  Moreover, in the portable terminal 1 concerning this Embodiment, the representative image of each group is extracted according to a fitness. As a result, the user can easily refer to the image having the highest fitness in each group.

  Furthermore, the display unit 113 displays the representative images of each group in the order of imaging time. Therefore, the user can browse images according to time series.

  The present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention. For example, the number of buffers provided in the mobile terminal 1 may be increased according to the processing capability of the mobile terminal 1. By increasing the number of buffers, images belonging to many groups can be held, so that the accuracy of grouping can be improved.

  In addition, the minimum structure of the portable terminal 1 concerning this Embodiment 1 is shown in FIG. The mobile terminal 1 includes an imaging unit 101 and a group calculation unit 105. The imaging unit 101 is a processing unit including a camera function for capturing a moving image and a still image. The camera function includes a continuous shooting function.

  The group calculation unit 105 is a processing unit for grouping images captured by the imaging unit 101 (giving a group number to an image). The group calculation unit 105 appropriately creates groups and classifies each captured image into groups. The group calculation unit 105 uses the captured image (first captured image) belonging to the group generated immediately before to classify the input captured image (second captured image). The group calculation unit 105 determines only whether or not the input captured image belongs to the group generated immediately before. For this reason, the determination process is reduced, and the processing speed of the grouping process is improved.

  A part or all of the above embodiment can be described as in the following supplementary notes, but is not limited thereto.

(Appendix 1)
An imaging unit for generating a captured image;
A group calculation unit that generates a group for classifying the plurality of captured images generated by the imaging unit, and calculates a group to which each captured image belongs,
The group calculation unit is configured such that an input second captured image belongs to the first group based on a first captured image that is any one captured image belonging to the first group generated immediately before. Or a portable terminal that determines that the second captured image belongs to a newly generated second group.

(Appendix 2)
The group calculation unit calculates a similarity between the first captured image and the second captured image, and calculates a group to which the second captured image belongs according to the similarity. The mobile terminal according to attachment 1.

(Appendix 3)
The mobile terminal according to appendix 1 or appendix 2, comprising a first buffer and a second buffer for storing either the first captured image or the second captured image.

(Appendix 4)
A fitness level giving unit that gives fitness level information indicating suitability for a desired condition to each of the captured images generated by the imaging unit,
A supplementary image selection unit that extracts a representative image representing a group based on the fitness information from a plurality of captured images belonging to the same group. The portable terminal described.

(Appendix 5)
The mobile terminal according to any one of supplementary notes 1 to 4, further comprising a display unit that displays a display screen in which the representative images of each group are arranged in order of imaging time.

(Appendix 6)
The similarity is calculated from a difference between a pixel value of the first captured image and a pixel value of the second captured image, according to any one of appendix 2 to appendix 5, wherein the similarity is calculated. Mobile device.

(Appendix 7)
An image classification method for generating a group for classifying a plurality of captured images and calculating a group to which each captured image belongs,
Based on the first captured image, which is any one of the images belonging to the first group generated immediately before, the input second captured image belongs to the first group, or the second An image classification method comprising a grouping step for determining that a captured image belongs to a newly generated second group.

(Appendix 8)
In the grouping step, a similarity between the first captured image and the second captured image is calculated, and a group to which the second captured image belongs is calculated according to the similarity. The image classification method according to appendix 7.

(Appendix 9)
Storing either the first captured image or the second captured image in a first buffer;
9. The image classification method according to appendix 7 or appendix 8, wherein the other captured image not stored in the first buffer is stored in the second buffer.

(Appendix 10)
A fitness level assigning step of giving fitness level information indicating suitability for a desired condition to each of a plurality of captured images;
A supplementary image selection step of extracting a representative image representing a group based on the fitness information from a plurality of captured images belonging to the same group. The image classification method described.

(Appendix 11)
11. The image classification method according to claim 7, further comprising a display step of displaying a display screen in which the representative images of each group are arranged in order of imaging time.

(Appendix 12)
The said similarity is calculated from the difference of the pixel value of a said 1st captured image, and the pixel value of a said 2nd captured image, The additional statement 8 thru | or any one of the additional notes 11 characterized by the above-mentioned Image classification method.

DESCRIPTION OF SYMBOLS 1 Mobile terminal 101 Image pick-up part 102 Suitability provision part 103 Buffer 104 Buffer 105 Group calculation part 106 Reduction process part 107 Encoding part 110 Manual preservation | save processing part 111 Primary preservation | save memory 112 Representative image extraction part 113 Display part 201 Flash memory

Claims (10)

An imaging unit for generating a captured image;
A group calculation unit that generates a group for classifying the plurality of captured images generated by the imaging unit, and calculates a group to which each captured image belongs,
The group calculation unit is configured such that an input second captured image belongs to the first group based on a first captured image that is any one captured image belonging to the first group generated immediately before. Or a portable terminal that determines that the second captured image belongs to a newly generated second group.   The group calculation unit calculates a similarity between the first captured image and the second captured image, and calculates a group to which the second captured image belongs according to the similarity. The mobile terminal according to claim 1.   The mobile terminal according to claim 1, further comprising a first buffer and a second buffer for storing either the first captured image or the second captured image. . A fitness level giving unit that gives fitness level information indicating suitability for a desired condition to each of the captured images generated by the imaging unit,
4. A representative image selection unit that extracts a representative image representing a group from a plurality of captured images belonging to the same group based on the fitness information. The portable terminal according to item.   5. The mobile terminal according to claim 1, further comprising a display unit configured to display a display screen in which the representative images of each group are arranged in order of imaging time.   6. The similarity according to claim 2, wherein the similarity is calculated from a difference between a pixel value of the first captured image and a pixel value of the second captured image. The portable terminal described. An image classification method for generating a group for classifying a plurality of captured images and calculating a group to which each captured image belongs,
Based on the first captured image, which is any one of the images belonging to the first group generated immediately before, the input second captured image belongs to the first group, or the second An image classification method comprising a grouping step for determining that a captured image belongs to a newly generated second group.   In the grouping step, a similarity between the first captured image and the second captured image is calculated, and a group to which the second captured image belongs is calculated according to the similarity. The image classification method according to claim 7. Storing either the first captured image or the second captured image in a first buffer;
9. The image classification method according to claim 7, wherein the other captured image not stored in the first buffer is stored in the second buffer. A fitness level assigning step of giving fitness level information indicating suitability for a desired condition to each of a plurality of captured images;
10. A representative image selection step of extracting a representative image representing a group from a plurality of captured images belonging to the same group based on the fitness information. The image classification method according to item.

Images