JP6951529B2 - Display method of the image pickup device - Google Patents

Description

The present invention relates to an imaging device capable of classifying and organizing a plurality of images taken by a camera, storing them, and easily searching for a desired image.

Digital cameras that convert a subject image into an electrical signal using an electronic device such as a CCD (Charge Coupled Device) and record it in memory are becoming widespread. Recently, it has become common for this digital camera function to be installed in information terminal devices such as mobile phones, smart phones, and tablet terminals.

Further, the storage elements used in digital cameras and the like are increasing in capacity, and a large amount of image data to be photographed is generated, which makes it complicated to classify and organize the image data. In particular, when displaying a large number of images as thumbnail list images, they can be classified based on metadata such as time and position information given to the image data using application software. However, it is necessary to classify a large amount of image data into groups, which is very troublesome and lacks convenience. In order to eliminate such a defect, a method has been proposed in which a plurality of captured images are classified and organized by generating and inserting images that serve as delimiters for a plurality of images.

For example, in Patent Document 1, it is used when the generation date of the previously generated image and the generation date of the image generated this time are different, or when the photographing equipment used when the image was generated last time and the image generated this time are generated. A configuration is described in which a date image or the like is superimposed on an image taken with the shutter closed and inserted as a delimited image when the image is different from the image taken.

Japanese Unexamined Patent Publication No. 2010-50705

In the technique described in Patent Document 1, a delimited image is generated based on a plurality of types of information such as generation date information, photographing equipment information, continuous shooting information, and voice input information. At that time, the priority is not set among the plurality of types of information that are the conditions for generating the delimited image, and they are treated equally. For example, FIG. 5 of Patent Document 1 shows an image (a) in which the generation date information is superimposed, an image (b) in which the generation date information is superimposed, and an image (c) in which the generation date information and the imaging equipment information are superimposed. , In the image (c), the generation date information and the shooting equipment information are treated equally.

It is preferable for the user that there are a plurality of types of delimited image generation conditions because a plurality of search conditions can be selected during an image search. However, if they are on an equal footing, when trying to search under a certain search condition 1, the delimited image for the other search condition 2 becomes annoying, and the usability deteriorates. Further, it is desirable that the image groups that hit the desired search conditions are continuously displayed as one group.

An object of the present invention is to provide an imaging device and a display method thereof that can classify captured images under a plurality of different conditions and easily search for captured images under desired conditions.

The imaging device of the present invention generates and inserts a title image including a display title that classifies the captured image into an imaging unit that captures a subject and a plurality of captured images acquired by the imaging unit. A unit, a storage unit for storing the photographed image and the title image, a display unit for displaying the photographed image and the title image stored in the storage unit in a list, and a plurality of units listed in the display unit. An image search execution unit for searching a desired captured image from the captured image and the title image is provided, and the title generation execution unit is said to have a priority set for each type of display title that classifies the captured image. The feature is that the title image is generated hierarchically.

According to the present invention, since a plurality of captured images are automatically classified and arranged hierarchically according to the priority of the classification conditions, a desired image can be easily and quickly searched.

The external view of the image pickup apparatus which concerns on Example 1. FIG. The block diagram of the image pickup apparatus which concerns on Example 1. FIG. The software block diagram of the image pickup apparatus which concerns on Example 1. FIG. The title image control table according to the first embodiment. The figure which shows the example of the display screen of a title image. The flowchart which shows the imaging operation of Example 1. A flowchart showing a title image generation process. A flowchart showing a title image generation process. Other title image control table. The figure which shows the generation example of other title images. The title image control table according to the second embodiment. The family composition table according to the second embodiment. The flowchart which shows the imaging operation of Example 2. A flowchart showing an image analysis process. A flowchart showing a title image generation process. Other title image control table. The figure which shows the generation example of other title images. The figure which shows an example of the display screen at the time of image search which concerns on Example 3. FIG. The figure which shows the example which changes the priority and rearranges the photographed image.

Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. The embodiments described below are examples for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. It is not limited to the embodiment of.

FIG. 1 shows an external view of the image pickup apparatus of this embodiment. Here, the image pickup device 100 is a case of an information terminal device such as a smart phone, (a) is a front view, (b) is a back view (back surface) view, and the front surface side of (a) is a photographed image or the like. A display unit 121 for displaying various information is arranged. In the imaging device 100 of this embodiment, imaging units (cameras) are provided at three locations. As shown in (a), the third imaging unit 125 is arranged on the same surface (surface) as the display unit 121, and as shown in (b), the first imaging unit 123 and the second imaging unit 124 are the display units. It is arranged on the opposite surface (back surface) from 121. The third imaging unit 125 arranged on the same surface as the display unit 121 is also called an in-camera, and the first imaging unit 123 and the second imaging unit 124 arranged on the surface opposite to the display unit 121 are also called out-cameras. A flash unit 129 is arranged at a position adjacent to the first imaging unit 123 and the second imaging unit 124.

In addition, a voice input unit (microphone) 133, a voice output unit (speaker) 131, an operation key 140k, and a touch panel 140t are arranged as input / output units of the image pickup apparatus 100.

In addition to the smart phone, the image pickup device 100 may be a digital still camera, a mobile phone having a camera function, a tablet terminal, a PDA (Personal Digital Assistants), or a notebook PC (Personal Computer). Further, it may be a video camera capable of shooting a moving image, a portable game machine, or other portable digital device. Therefore, the configuration of the imaging unit (camera) is not limited to the above, and the first imaging unit 123 and the second imaging unit 124 may be integrally configured or the first to third imaging units may be configured according to the function of the information terminal device. It may be integrated and one camera may be arranged on the back side or the display unit 121 side.

FIG. 2A is a block diagram of the image pickup apparatus of this embodiment. The imaging device 100 includes a main control unit 101, a system bus 102, a memory unit 104, a storage unit 110, a video processing unit 120, an audio processing unit 130, an operation unit 140, a communication processing unit 150, a sensor unit 160, and an extended interface unit 170. Consists of.

The main control unit 101 is a microprocessor unit that controls the entire image pickup apparatus 100 according to a predetermined program. The system bus 102 is a data communication path for transmitting and receiving data between the main control unit 101 and each unit in the image pickup apparatus 100.

The memory unit 104 serves as a program area for executing a basic operation program and other application programs, and a temporary storage area for temporarily holding image data and the like as needed when executing various application programs. The memory unit 104 may be integrated with the main control unit 101.

The storage unit 110 stores various setting information of the image pickup device 100, still image data, moving image data, and the like taken by the image pickup device 100. Further, the storage unit 110 stores various application programs executed by the image pickup apparatus 100. The main control unit 101 expands the application program stored in the storage unit 110 into the memory unit 104, and further executes the expanded application program, whereby the image pickup apparatus 100 realizes various functions. The application program can be extended by downloading it from the application server via the Internet.

The storage unit 110 needs to hold the stored information even when the power is not supplied to the image pickup apparatus 100. Therefore, for example, devices such as flash ROMs, SSDs (Solid State Drives), and HDDs (Hard Disc Drives) are used.

The image processing unit 120 includes a display unit 121, a video signal processing unit 122, a first imaging unit 123, a second imaging unit 124, and a third imaging unit 125. The display unit 121 is a display device such as a liquid crystal panel, and provides the image data processed by the video signal processing unit 122 to the user of the image pickup apparatus 100. The video signal processing unit 122 processes the image data captured by the first imaging unit 123, the second imaging unit 124, and the third imaging unit 125.

The voice processing unit 130 includes a voice output unit 131, a voice signal processing unit 132, and a voice input unit 133. The audio output unit 131 is a speaker, and provides the user of the image pickup apparatus 100 with the audio signal processed by the audio signal processing unit 132. The voice input unit 133 is a microphone, which converts a user's voice or the like into voice data and inputs it. The voice input unit 133 is separate from the image pickup device 100, and may be connected to the image pickup device 100 by wired communication or wireless communication.

The operation unit 140 is an instruction input unit for inputting operation instructions to the image pickup apparatus 100, and as shown in FIG. 1, a touch panel 140t arranged on the display unit 121 and an operation key 140k in which button switches are arranged are equivalent. do. The touch panel function may be provided by the display unit 121. As another operation method, the image pickup apparatus 100 may be operated by using a keyboard or the like connected to the expansion interface unit 170 described later. Alternatively, the image pickup apparatus 100 may be operated by using a separate information terminal device connected by wired communication or wireless communication.

The communication processing unit 150 is composed of a LAN (Local Area Network) communication unit 151, a mobile telephone network communication unit 152, and a proximity wireless communication unit 153. The LAN communication unit 151 connects to an access point for wireless communication on the Internet by wireless communication to transmit and receive data. The mobile telephone network communication unit 152 performs telephone communication (call) and data transmission / reception by wireless communication with a base station of the mobile telephone network. The proximity wireless communication unit 153 performs wireless communication when it is in close proximity to a corresponding reader / writer. The LAN communication unit 151, the mobile telephone network communication unit 152, and the proximity wireless communication unit 153 are each provided with a coding circuit, a decoding circuit, an antenna, and the like. Further, an infrared communication unit or the like may be provided.

The sensor unit 160 is a group of sensors for detecting the state of the image pickup device 100. In this embodiment, the GPS (Global Positioning System) receiving unit 161, the acceleration sensor 162, the gyro sensor 163, the geomagnetic sensor 164, and the illuminance sensor 165 , Proximity sensor 166. These sensor groups detect the position, movement, tilt, direction, ambient brightness, proximity status of surrounding objects, and the like of the image pickup apparatus 100.

The expansion interface unit 170 is a group of interfaces for expanding the functions of the image pickup apparatus 100, and in this embodiment, it is configured to include a video / audio interface, a USB (Universal Serial Bus) interface, a memory interface, and the like. The video / audio interface inputs video / audio signals from an external video / audio output device, outputs video / audio signals to an external video / audio input device, and the like. The USB interface connects keyboards and other USB devices. The memory interface connects a memory card or other memory medium to send and receive data.

Needless to say, the configuration of the image pickup apparatus 100 shown in FIG. 2A is for an information terminal device such as a smart phone, and can be appropriately changed according to the function of the target information terminal device.

FIG. 2B is a software configuration diagram of the image pickup apparatus of this embodiment, and shows the software configuration of the memory unit 104 and the storage unit 110.

The storage unit 110 includes various information / data storage areas 110a, and also stores a camera function program 110b, a title generation program 110c, an image analysis program 110d, and an image search program 110e. The various information / data storage areas 110a store various setting information of the image pickup apparatus 100, still image data, moving image data, and the like taken by the image pickup apparatus 100. Further, the camera function program 110b is a program for executing shooting, the title generation program 110c is a program for generating a title image, the image analysis program 110d is a program for analyzing the shot image, and the image search program 110e is. This is a program for searching for captured images.

The memory unit 104 includes a basic operation execution unit 104a that performs basic operations as an information terminal device, various execution units 104b to 104e that perform operations related to an imaging function, and a temporary storage area 104f that temporarily stores captured images and the like. Each of the above programs 110b to 110e stored in the storage unit 110 is expanded in the memory unit 104. Then, the main control unit 101 executes each program to configure the camera function execution unit 104b, the title generation execution unit 104c, the image analysis execution unit 104d, and the image search execution unit 104e, and execute various function operations.

In this embodiment, the various functional operations of the image pickup apparatus are realized by software, but the operations of the various execution units 104b to 104e described above can also be realized by hardware.

Hereinafter, the operation by the image pickup apparatus of this embodiment, particularly the generation of the title image will be described in detail. In this embodiment, the delimited image for classifying a plurality of captured images is referred to as a "title image". The title image is generated including a keyword (display title) for classification. At that time, it is characterized in that priority is set according to the type of display title and title images are generated hierarchically according to the priority.

FIG. 3 is a title image control table in which title image generation rules are defined. The title image control table (T0) is set in advance by the user and stored in various information / data storage areas 110a of the storage unit 110. Here, "year", "season", "month", "day", and "place" are registered as the types of display titles of the title image. The title generation execution unit 104c generates a title image having the display titles when the “year”, “season”, and the like are different among the shooting conditions. The type of display title is not limited to this, and any display title can be added as long as it can be distinguished based on time information and position information (GPS information). The definition of "season" is spring (March-May), summer (June-August), autumn (September-November), and winter (December-February), but the period is arbitrary. Can be set. Further, the "place" is not simply a different place, but is defined as a different place if the previous shooting position and the current shooting position are separated by a predetermined distance or more. In this example, as described in the condition column, it is assumed that the distance is 10 km or more.

In the title image control table, priority flags (= 1, 2, ...) Are set according to the type of display title. Here, the display titles are prioritized in the order of "year", "season", "month", and so on. The title image generation flag defines the case where the title image is generated (flag = 1) and the case where the title image is not generated (flag = 0). In this example, the display title is "day" and the "location" is the title. No image is generated.

Further, the title image composition flag defines a case where the image is combined with another type of title image generated earlier (flag = 1) and a case where the image is not combined (flag = 0). In this example, "year" and "season" are not combined with other types of title images, and "month" is not combined with other types of title images. However, even if its own title image composition flag = 0, if the title image composition flag of another type of title image generated later is 1, it may be combined with the title image generated later. .. The title image composition flag = × mark means not care (irrelevant).

Further, in this embodiment, the design of the generated title image is changed according to the priority of the displayed title. Specifically, as will be described later, the conspicuity of the title image is changed by making at least one of the title image size (aspect ratio), the character size in the image, the background color of the image, and the resolution of the image different. ing. In other words, by displaying the title images hierarchically according to the priority, the user can easily identify the title image with a high priority and quickly search for the captured image belonging to the display title with a high priority. There is.

FIG. 4 is a diagram showing an example of a display screen of a title image. A series of images 10 taken by the image pickup apparatus 100 are displayed as thumbnails on the display unit 121. (A) is a state in which the captured images 10 are arranged in chronological order without a title image and displayed in a list, and the shooting order is in the direction of the arrow, that is, from the left to the right of the screen and from the top to the bottom. .. (B) is a state in which title images 11 to 13 are generated and inserted into the captured image 10 and displayed in a list.

The title image generation rule follows the title image control table (T0) in FIG. 3, and the title image indicating the content of the displayed title each time the "year", "season", and "month" of the shooting conditions change (in this example). White characters) are generated on a black screen and inserted at the delimiter position of the captured image. The title image 11 is inserted at the "year" delimiter, and has the highest priority and the largest image size. The title image 12 is inserted in the "season" delimiter, and the title image 13 is inserted in the "month" delimiter, and the size of the image is reduced according to the priority. When the "season" and the "month" change at the same time, the title image 12 and the title image 13 are combined and displayed.

In this way, by displaying the title images hierarchically by changing the image size according to the priority, the user can quickly search for the captured image belonging to the display title "year" having a high priority.

FIG. 5 is a flowchart showing an imaging operation of the imaging device 100. This imaging operation is controlled by the camera function execution unit 104b and the title generation execution unit 104c configured in the memory unit 104.

When an imaging instruction is given from the operation unit 140 to any of the first imaging unit 123, the second imaging unit 124, and the third imaging unit 125, the imaging process is executed (S201). A captured image is acquired from the image pickup unit (for example, the first image pickup unit 123) instructed by this (S202). Next, the time information and the position information at the time of shooting are acquired from the built-in clock mechanism (not shown) and the GPS receiving unit 161 and used as the metadata of the shot image (S203). The acquired captured image is associated with the metadata and temporarily stored in the temporary storage area 104f of the memory unit 104 (S204).

Next, the variable PRI for internal processing is initially set to 1 (S205). This variable PRI is a parameter of the priority of the title image. The title image control table stored in the storage unit 110 is referred to, and the display title of the priority set in the variable PRI is searched for (S206). For example, in the case of the title image control table (T0) of FIG. 3, the display title "year" is searched for the variable PRI = 1. Next, the title image generation process is executed for the searched display title (for example, "year") according to the setting of the title image control table (S207). The specific content of the title image generation process will be described later, but when the title image generation flag of the target display title = 1 (the title image should be generated) and the shooting conditions do not match the previously shot image. Generate a title image. The generated title image is temporarily stored in the temporary storage area 104f in the memory unit 104. After the title image generation process is completed, it is determined whether or not the processed display title has the lowest priority (S208).

In the determination of S208, when the priority of the processed display title is not the lowest, 1 is added to the variable PRI (S209), and the process returns to S206. Then, the same processing is performed for the display title of the next priority. In the case of the control table (T0) of FIG. 3, the title image generation process is performed for the display title “season” which is the priority of the variable PRI = 2.

In the determination of S208, when the processed display title has the lowest priority, the title image generation process is terminated. Then, the temporarily saved title image is stored in the various information / data storage areas 110a of the storage unit 110 (S210). The number (N) of title images generated in S207 ranges from 0 to the maximum number of priorities set. Next, the image data and metadata temporarily stored in the temporary storage area 104f of the memory unit 104 are stored in the various information / data storage areas 110a of the storage unit 110 (S211). This completes the imaging process.

6A and 6B are flowcharts showing the details of the title image generation process (S207) of FIG. In the title image generation process (S207), a title image is generated for each display title, and here, the display titles "year" and "place" will be described. The same processing is applied to the other display titles "season", "month", and "day", and the description thereof will be omitted.

FIG. 6A is a flowchart showing a title image generation process for the “year”. This is executed following S206 in FIG.
With reference to the title image control table, it is determined whether or not the title image generation flag is set to 1 (S301). When the generation flag is not set to 1, this process ends.

When the generation flag is set to 1, among the images stored in the storage unit 110, the shooting year is read from the metadata of the previously shot image, and the shot image of this time temporarily stored in the memory unit 104 is read. It is determined whether or not it matches the shooting year (S302). When the shooting years match in S302, this process ends.

When the shooting years do not match in S302, a title image having a design according to the priority of the displayed title is generated. That is, the image size (aspect ratio) determined in advance according to the priority is set, a title image including characters indicating the shooting year (year of the Christian era, Japanese calendar year, etc.) is generated, and the image is temporarily stored in the temporary storage area 104f of the memory unit 104. Save (S303). As the background image of the title image, an image showing the zodiac of the year, a calendar, or the like may be used. When the display title is "season", a seasonal image such as a cherry blossom, the sea, autumn leaves, or a snow scene may be used as the background image.

Next, it is determined whether or not the title image composition flag of the title image control table is set to 1 (S304). When the synthesis flag is not set to 1, this process ends. That is, it becomes a single title image without being combined with other title images generated earlier.

When the composition flag is set to 1, it is determined whether or not there is a title image having a higher priority that is generated first and temporarily stored in the memory unit 104 (S305). If there is no title image temporarily saved earlier, this process ends.

When there is a title image temporarily saved earlier, the title image generated earlier and the title image generated this time are combined and temporarily saved in the memory unit 104 (S306). In the compositing process at this time, the title image generated this time is reduced or transformed so that the hierarchical difference from the previously generated upper title image becomes clear. For example, the title image generated this time is designed to be arranged at the bottom of the screen or to reduce the character size of the display title. This completes the title image generation process, and proceeds to S208 in FIG.

FIG. 6B is a flowchart showing a title image generation process targeting the “location”.
With reference to the title image control table, it is determined whether or not the title image generation flag is set to 1 (S311). When the generation flag is not set to 1, this process ends.

When the generation flag is set to 1, the shooting location is read from the metadata of the previously shot image stored in the storage unit 110, and matches the shooting location of the currently shot image temporarily saved in the memory unit 104. It is determined whether or not to do so (S312). In this determination, if the distance between the two shooting locations is less than the distance (10 km in FIG. 3) described in the condition column of the title image control table, it is considered to match. When the photographed image is a moving image obtained by moving the imaging device, a position to be compared as a photographing place such as a shooting start position or a shooting end position may be appropriately determined. When the shooting locations match in S312, this process ends.

When the shooting locations do not match in S312, a title image having a design according to the priority of the display title is generated. That is, the image size (aspect ratio) is set in advance according to the priority, a title image including characters indicating the shooting location (eg, Hakone, sunlight, etc.) is generated, and the image is temporarily stored in the temporary storage area 104f of the memory unit 104. Save (S313). As the background image of the title image, a map image of the place, a landmark image representing the place, or the like may be used. These characters and images can be obtained from the providing server via the Internet from the communication processing unit 150.

Next, it is determined whether or not the title image composition flag of the title image control table is set to 1 (S314). When the synthesis flag is not set to 1, this process ends.

When the composition flag is set to 1, it is determined whether or not there is a title image having a higher priority that is generated first and temporarily stored in the memory unit 104 (S315). If there is no title image temporarily saved earlier, this process ends.

When there is a title image temporarily saved earlier, the title image generated earlier and the title image generated this time are combined and temporarily saved in the memory unit 104 (S316). The synthesis process at this time is the same as the description of S306 in FIG. 6A. Alternatively, when the title image generated this time is an image such as a map or a landmark, the characters of the previously generated title image may be superimposed on the map or the landmark as a background. This completes the title image generation process, and proceeds to S208 in FIG.

7A and 7B are diagrams showing other generation examples of title images. FIG. 7A shows the cases (T1) to (T4) as other examples of the title image control table. Here, the item in which the title image generation flag = 0 is omitted. The control table (T2) has the same contents as the control table (T0) of FIG. FIG. 7B shows an example of a title image generated according to each of these control tables.

As (initial state) of FIG. 7B, when two captured images P1 and P2 are stored in the storage unit 110 together with the metadata, a new imaging process is executed and the captured images are stored in the temporary storage area 104f of the memory unit 104. It is assumed that P3 and metadata are temporarily stored. In this case, the image P2 is the previous image and the image P3 is the current image. Below each image, information on the shooting date and time and the shooting location, which are metadata, is described. At this time, how the title image is generated according to each title image control table of FIG. 7A will be described.

In the case of the control table (T1), the order of priority is "year", "season", and "month", and the title image generation flag is 1 and the title image composition flag is 0. In this case, the shooting date and time of the previous shot image P2 and the shooting date and time of the current shot image P3 do not match in all of "year", "season", and "month". Month title images 23 are generated in that order. At that time, the size of the title image is changed according to the priority. Also, the character size may be changed. Since the composite flags are all 0, the title image is not composited. The three generated title images are inserted immediately before the captured image P3.

In the case of the control table (T2), since the title image composition flag of "month" is set to 1 with respect to the control table (T1), the title images 22 and 23 of the season and the month are combined.

In the case of the control table (T3), the order of priority is "place" and "year", and the title image generation flag is 1 and the title image composition flag is 0. In this case, the distance between the previous shooting location A and the current shooting location B is 100 km, which is 10 km or more as described in the condition column of the control table. Further, since the shooting years are different, the title image 24 of the place and the title image 25 of the year are generated.

In the case of the control table (T4), since the title image composition flag of "year" is set to 1 with respect to the control table (T3), the title images 24 and 25 of the place and the year are combined.

By setting the desired title image control table in this way, it is possible to generate a title image having a hierarchical structure according to the priority order and insert it as a delimiter image. As a result, a plurality of captured images can be classified and arranged according to a desired priority.

In the first embodiment, the title image was generated based on the time information and the position information (GPS information) which are the metadata of the captured image. In the second embodiment, the type of the subject in the captured image is further determined, and the title image is generated based on the determination. Therefore, as the configuration of the image pickup apparatus 100, the image analysis execution unit 104d in FIG. 2B is used, and for example, it is determined whether or not a person or the like exists in the image, and metadata is added. In the following description, the metadata used in the first embodiment is referred to as "metadata 1", and the metadata added in the second embodiment is referred to as "metadata 2".

FIG. 8A is a title image control table for generating a title image in this embodiment. In the title image control table (T5), "family", "person", and "landscape", which are the types of subjects, are added as the types of display titles. Here, the "family" is an image in which a person is shown and the person includes a family. A "person" is an image of a person other than a family member. A "landscape" is an image that does not show a person.

FIG. 8B is a family composition table that defines the conditions for determining the “family” image. The family composition table (T6) consists of a family face image link and a matching number of people. It is determined whether or not the captured image matches the face image of the family's face image link destination, and if the number of matched people is a predetermined number (here, two or more), it is determined to be a family image.

The image analysis execution unit 104d performs person recognition and family recognition (by face recognition) by image analysis, but a known technique using pattern recognition, machine learning, or the like (for example, the Viola-Jones method) may be used. , In particular, the method of realizing it is not limited. In addition, although family, people, and landscapes are taken up as display titles to be added here, mountains, animals, friends, alumni, etc. may be used as long as they use image analysis results, and are not particularly limited.

Priorities are set for each display title as in the first embodiment, and a hierarchical structure of title images is realized by generating title images having different designs according to the priorities. The title image control table (T5) and the family structure table (T6) are set in advance by the user and stored in the storage unit 110.

FIG. 9 is a flowchart showing an imaging operation of the imaging device 100. S401 to S403 are processing of acquiring a photographed image and acquiring metadata, and are the same as S201 to S203 of the first embodiment, and thus the description thereof will be omitted. It should be noted that the metadata 1 is acquired in S403.

The image analysis execution unit 104d analyzes the captured image, determines whether the image is a family image, a person image, or a landscape image, and creates the determination result as metadata 2 (S404). The details of the image analysis process will be described later. The captured image and the metadata 1 and 2 are associated with each other and temporarily stored in the temporary storage area 104f of the memory unit 104 (S405).

Next, the variable PRI for internal processing is initially set to 1 (S406). The title image control table stored in the storage unit 110 is referred to, and the display title of the priority set in the variable PRI is searched for (S407). For example, in the case of the title image control table (T5) of FIG. 8A, the display title "family" is searched for the variable PRI = 1. Next, the title image generation process is executed for the searched display titles according to the settings in the title image control table (S408).

The specific content of the title image generation process will be described later, but it is not only the title image that is generated when the display title that was shot this time changes to the target display title, but also the display title that is the target of the previously shot image. When the captured image changes to a display title other than the target display title, a title image indicating that the target display title has "finished" is generated. The generated title image is temporarily stored in the temporary storage area 104f in the memory unit 104. After the title image generation process is completed, it is determined whether or not the processed display title has the lowest priority (S409).

In the determination of S409, when the priority of the processed display title is not the lowest, 1 is added to the variable PRI (S410) to branch to S407. Then, the same processing is performed for the display title of the next priority. In the case of the control table (T5) of FIG. 8A, the title image generation process is performed for the display title “landscape” which is the priority of the variable PRI = 2.

In the determination of S409, when the processed display title has the lowest priority, the title image generation process is terminated. Then, the temporarily saved title image is stored in the various information / data storage areas 110a of the storage unit 110 (S411). Next, the image data and the metadata 1 and 2 temporarily stored in the temporary storage area 104f of the memory unit 104 are stored in the various information / data storage areas 110a of the storage unit 110 (S412). This completes the imaging process.

FIG. 10A is a flowchart showing the details of the image analysis process (S404) of FIG.
The captured image is analyzed by the image analysis execution unit 104d (S501). It is determined whether or not a person is included in the captured image (S502). When it is determined that the captured image does not include a person, the captured image is determined to be a "landscape" image and metadata 2 is created (S507).

When it is determined that the captured image includes a person, it is determined whether the person includes a family member (S503). The family composition table (T6) shown in FIG. 8B is used for determining the family. The family composition table has a family face image link, and it is determined whether or not the person included in the photographed image matches the face image of the family face image link destination. This determination is sequentially performed for all families registered in the family composition table.

When it is determined that the captured image includes a family member, it is determined whether or not the number of family members included is a predetermined number or more (S504). That is, it is determined whether or not the number of people whose face images match is equal to or greater than the "matched number of people" in the family composition table. When it is determined that the number of family members is equal to or greater than the number of matching persons, the captured image is determined to be a "family" image and metadata 2 is created (S505).

When it is determined in S503 that the captured image does not include a family, or when it is determined in S504 that the number of family members is less than the matching number, the captured image is determined to be a "person" image and metadata 2 is created (). S506).
This completes the image analysis process, and proceeds to S405 in FIG.

FIG. 10B is a flowchart showing the details of the title image generation process (S408) of FIG. This is executed following S407 in FIG. Here, the case where the target display title is "family" will be described as an example. The same processing is performed when the display title is "person" or "landscape".

With reference to the title image control table, it is determined whether or not the title image generation flag of "family" is set to 1 (S511). When the generation flag is not set to 1, this process ends.

When the generation flag is set to 1, it is determined from the metadata 1 and 2 of the temporarily saved image taken this time whether the image taken this time is a "family" image (S512). When it is determined that the current image is a family image, the process proceeds to S513, and when it is determined that the current image is not a family image, the process proceeds to S515.

In S513, it is determined from the metadata 1 and 2 of the previously captured image stored in the storage unit 110 whether the previous image is a family image. When it is determined that the previously captured image is a family image, this process ends. When it is determined that the previously captured image is not a family image, a title image is generated. That is, a title image including characters indicating "family" is generated with a design (image size, etc.) according to the priority of the display title "family" and temporarily stored in the temporary storage area 104f of the memory unit 104. (S514). As the background image of the title image, a typical family image (an image of a representative of the family or an image of all members), a photographed image of a temporarily stored family, or the like may be used. Then proceed to S517.

In S515, as in S513, it is determined whether the previous image is a family image. When it is determined that the previously captured image is not a family image, this process ends. When it is determined that the previously captured image is a family image, a title image is generated. However, in this case, a title image having a design according to the priority of "family" and including a character indicating "end of family" is generated and temporarily stored in the temporary storage area 104f of the memory unit 104 (S516). An image or the like that emphasizes "end" may be combined as the background image of the title image. Then proceed to S517.

In S517, it is determined whether or not the title image composition flag of the title image control table is set to 1. When the synthesis flag is not set to 1, this process ends. When the composition flag is set to 1, it is determined whether or not there is a title image having a higher priority that is generated first and temporarily stored in the memory unit 104 (S518). If there is no title image temporarily saved earlier, this process ends. When there is a title image temporarily saved earlier, the title image generated earlier and the title image generated this time are combined and temporarily saved in the memory unit 104 (S519). In the composition of title images, the vertical arrangement and character size are different according to the priority. This completes the title image generation process, and proceeds to S409 in FIG.

11A and 11B are diagrams showing various generation examples of title images. FIG. 11A shows the cases of (T7) to (T9) as an example of the title image control table. Here, some items of the title image generation flag = 0 are omitted. FIG. 11B shows an example of a title image generated according to each of these control tables.

In the (initial state) of FIG. 11B, when two captured images P4 and P5 are stored in the storage unit 110 together with the metadata 1 and 2, a new imaging process is executed and the temporary storage area 104f of the memory unit 104 is executed. It is assumed that the captured image P6 and the metadata 1 and 2 are temporarily stored in. In this case, the image P5 is the previous image and the image P6 is the current image. Below each image, information on the shooting date and time, which is metadata 1, and the result of image analysis (family, person, landscape), which is metadata 2, is described. At this time, how the title image is generated by each title image control table of FIG. 11A will be described.

In the case of the control table (T7), the order of priority is "family", "person", and "year", the title image generation flag is 1, and the title image composition flag is 0. In this case, since the image analysis result of the previous captured image (P5) is "person", the image analysis result of the current captured image (P6) is "family", and the shooting year is different, " Three title images, a title image 31 of "family", a title image 32 of "end of person", and a title image 33 of "year", are generated in order of priority. Since the composite flags are all 0, the title image is not composited.

In the case of the control table (T8), since the title image composition flag of "person" is set to 1 with respect to the control table (T7), the title images 31 and 32 of "family" and "end of person" are combined. Will be done.

In the case of the control table (T9), the title image generation flag is 1 only for "person" with respect to the control table (T7). In this case, since the previously captured image is "person" and the current captured image (P6) is "family", only the title image 32 of "person end" corresponding to priority 2 is generated.

As described above, in this embodiment, the title image having a hierarchical structure having the priority set in the title image control table can be generated by referring to the metadata 2 created by the image analysis result of the captured image. Therefore, it is possible to increase the types of display titles and classify and organize a plurality of captured images by items that are easier to use.

In the third embodiment, a method of searching the captured image stored in the storage unit 110 will be described. In order to efficiently perform the search process of the captured image, in this embodiment, the image search execution unit 104e configured in the memory unit 104 executes the search as follows.

FIG. 12 is a diagram showing an example of a display screen at the time of image search. (A) shows the list display state of the photographed image, and the title image is inserted at the delimiter position of the photographed image 10. Here, according to the title control table (T1) of FIG. 7A, the respective title images 21, 22, and 23 are hierarchically generated according to the priority order of “year”, “season”, and “month”. The title image is not combined. The user selects a search target by tapping, for example, a desired title image on the operation unit 140. At that time, since the size of the title image and the like are changed according to the priority, the higher title image can be easily identified and selected.

(B) is the case where the title image 21 of "2016" is selected, and the photographed image (target range indicated by the broken line 21a) included in the title image of "2016" is highlighted. (C) is the case where the title image 22 of "spring" is selected (target range 22a), and (d) is the case where the title image 23 of "March" is selected (target range 23a). In this way, the captured image belonging to the selected title image is displayed with more emphasis than the other images, so that the user can quickly search for the desired image.

In the case of FIG. 12, since the captured images are arranged in chronological order, when searching by "year", the range of the target images is continuous, so that the search is easy. On the other hand, when searching by "spring" or "March", the range of the target image is dispersed depending on the difference in "year", so that it becomes difficult to search when searching over different "years". Therefore, it will be described that the priority of the title images is changed to rearrange the arrangement of the captured images.

FIG. 13 is a diagram showing an example in which the captured images are rearranged by changing the priority order. (A) is a list display when the title image 21 of "year" is given the highest priority, and has the same arrangement as in FIG. 12 (a). On the other hand, (b) is a list display when the title image 22 of "season" is changed so as to have the highest priority. At this time, the user operates, for example, by dragging the position of the title image 22 of "spring" on the screen (a) to replace the title image 21 of "2016".

When the user's drag operation is detected, the title generation execution unit 104c changes the current title image control table (T1) to a priority flag of "season" = 1 and a priority flag of "year" = 2, and newly changes. Generate title images according to the highest priority. Then, a process (sorting process) of rearranging the order of the captured images stored in the storage unit 110 according to a new priority is performed. In this sorting process, since the image group classified by the title image generated in (a) can be treated as a group, the processing can be performed in a short time without referring to the metadata of each image one by one. As a result (b), since the captured images over different "years" are continuously arranged in the new title image 22'of "spring" (indicated by the broken line 22a'), the user can easily search. can do.

Although some examples of the present invention have been described above, it goes without saying that the configuration for realizing the technique of the present invention is not limited to the above examples, and various modifications can be considered. For example, in the above embodiment, the title image is generated at the timing of shooting by the imaging device, but the title image is generated according to the title image control table by analyzing a plurality of shooting data after the shooting is completed, and is inserted as a delimited image. You may. In addition, the numerical values appearing in the text and figures are merely examples, and the effects of the present invention may not be impaired even if different values are used.

The above-mentioned functions of the present invention and the like may be realized by hardware, for example, by designing a part or all of them by an integrated circuit. Further, it may be realized by software by interpreting and executing a program in which a microprocessor unit or the like realizes each function or the like. Hardware and software may be used together.

10: Captured image 11-13, 21-25, 31-33: Title image, 100: Imaging device, 104: Memory unit, 104c: Title generation execution unit, 104d: Image analysis execution unit, 104e: Image search execution unit , 104f: Temporary storage area, 110: Storage unit, 110a: Various information / data storage areas, 121: Display unit, 123 to 125: Imaging unit (camera), 140: Operation unit.

Claims (10)

This is the display method of the image pickup device.
A shooting step in which the imaging device shoots a subject,
A title generation step of generating and inserting a title image including a display title that classifies the captured image into a plurality of captured images captured by the imaging device in the shooting step.
The display step to be displayed by the imaging device is provided.
In the title generation step, the title image is hierarchically generated according to the priority set for each type of display title that classifies the captured image.
A display method comprising displaying a list of the captured image and the title image in the display step. In the display method according to claim 1,
The image pickup apparatus has title image control information that stores the type of the display title and the priority order of the display title when the title image is generated.
A display method characterized in that, in the title generation step, the title image having a different design is generated based on the title image control information. In the display method according to claim 2,
A display method characterized in that at least one of the image size, the character size in the image, the background color of the image, and the resolution of the image is different from the different designs of the title images. In the display method according to claim 2,
The title image control information is whether or not to generate a title image by combining the title image generation information indicating whether or not to generate a title image for each display title with another type of title image generated earlier. A display method characterized by including title image composition information indicating. In the display method according to claim 2,
The title generation step compares the captured image captured last time with the captured image captured this time, and generates the title image when the shooting date and time or shooting location of each captured image is different as the type of the display title. A display method characterized by that. In the display method according to claim 5,
The imaging device includes an image analysis step for identifying the subject.
In the image analysis step, the type of the subject in the captured image is determined.
A display method characterized in that, in the title generation step, the title image is generated when the type of the subject determined in the image analysis step is different as the type of the display title. In the display method according to claim 6,
A display method characterized in that when a person is present in the captured image as the type of the subject, the display title is different depending on whether or not the person includes a family member. In the display method according to claim 1,
A display method characterized by searching for a desired captured image from a plurality of captured images and title images displayed in a list by the imaging device. In the display method according to claim 8,
A display method characterized in that when the title image displayed in a list by the imaging device is selected by the user, the captured image classified by the selected title image is emphasized and displayed. In the display method according to claim 9,
In the title generation step, when the user receives an operation to change the priority of the title images listed by the imaging device, a new title image is generated according to the new priority.
A display method characterized in that the captured images are rearranged according to a new title image and displayed in a list by the image pickup apparatus.

Images