JP6230386B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for supporting a photographing action.

  Various proposals have been made regarding techniques for presenting assist information for supporting a shooting action. For example, the image photographing system proposed in Patent Document 1 presents a sample image corresponding to an input situation keyword and image keyword together with the photographing condition and photographing technique to the user. In addition, the camera proposed in Patent Document 2 analyzes image data according to the type of the subject of interest, and presents guide information indicating an optimal composition according to the subject of interest from the analysis result to the user. ing.

JP 2007-235508 A JP 2011-223599 A

  Here, in the technique of Patent Document 1, information is not presented unless a user inputs a keyword himself. For this reason, with the technique of Patent Document 1, it is difficult to present information that the user himself / herself is not aware of. Further, in the technique of Patent Document 2, guide information is presented according to the result of image analysis, and thus information that the user is not aware of may be presented. However, in the technique of Patent Document 2, in the case where the same kind of subject is continuously photographed, the same kind of information is repeatedly presented, which may interfere with the user's photographing action.

  The present invention has been made in view of the above circumstances, and provides an image processing apparatus, an image processing method, and an image processing program capable of controlling the frequency of assist information presentation to the extent that it does not interfere with a user's shooting action. The task is to do.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention includes an image acquisition unit that acquires image data, a shooting information acquisition unit that acquires shooting information related to the image data, and the shooting information. A determination unit that determines information related to a shooting scene or subject in the image data as scene / subject related information, and an assist information search unit that searches for assist information according to the scene / subject related information determined by the determination unit A history management unit that stores history information of the scene / subject-related information, a scene / subject-related information determined by the determination unit, and history information of the scene / subject-related information stored in the history management unit When the similarity is lower than a predetermined value, display output of the searched assist information is permitted, and the scene / subject relation determined by the determination unit is permitted. Output determination that determines that display output of the searched assist information is not permitted when the similarity between the information and the history information of the scene / subject-related information stored in the history management unit is higher than the predetermined value And a display unit for displaying the assist information permitted for display output together with the corresponding display image data of the image data. The image data is associated with the scene / subject related information. The history management unit stores history information of a plurality of the scene / subject related information, and the output determination unit includes the scene / subject related information and the display image data of the display image data. According to the degree of similarity of the image data acquired by the image acquisition unit by shooting after the corresponding shooting with the scene / subject related information, the display image data is displayed. Whether the scene / subject-related information of the data is the end of the similar scene / subject-related information in succession, and whether to permit the display output of the searched assist information according to the determination result Determine whether .
In order to achieve the above object, an image processing apparatus according to a second aspect of the present invention includes an image acquisition unit that acquires image data, a shooting information acquisition unit that acquires shooting information related to the image data, and the shooting information. A determination unit that determines information related to a shooting scene or subject in the image data as scene / subject related information, and an assist information search unit that searches for assist information according to the scene / subject related information determined by the determination unit A history management unit that stores history information of the scene / subject-related information, a scene / subject-related information determined by the determination unit, and history information of the scene / subject-related information stored in the history management unit When the similarity is lower than a predetermined value, display output of the searched assist information is permitted, and the scene / subject relation determined by the determination unit is permitted. Output determination that determines that display output of the searched assist information is not permitted when the similarity between the information and the history information of the scene / subject-related information stored in the history management unit is higher than the predetermined value And a display unit that displays assist information for which display output is permitted together with corresponding display image data of the image data, and the imaging information acquisition unit relates to an attitude or a moving speed of the imaging device Shooting information is acquired, and the output determination unit performs shooting in the same state based on the shooting information acquired after shooting corresponding to the display image data is performed. It is determined whether or not it has been completed, and if it is determined that it has been completed, display output of the searched assist information is permitted .

In order to achieve the above object, an image processing method according to a third aspect of the present invention acquires image data, acquires shooting information related to the image data, and the image based on the shooting information. Determining information relating to a shooting scene or subject in the data as scene / subject related information, searching for assist information in accordance with the determined scene / subject related information, and the determined scene / subject related information; When the similarity with the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted, and the determined scene / subject related information And the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the search is performed. Comprising a determining not to allow the display output of the assist information, and by displaying assist information which the display output is permitted on the display unit together with the corresponding display image data among the image data, the image data Are associated with the scene / subject-related information, and the history management unit stores history information of the plurality of scene / subject-related information, and the scene / subject-related information of the display image data And the scene / subject-related information of the display image data in accordance with the similarity between the image data acquired by the image acquisition unit after shooting corresponding to the display image data and the scene / subject-related information. Is the end of the scene-subject-related information that is similar in succession, and the searched assist is determined according to the determination result. It comprises determining whether to permit the display output bets information.
In order to achieve the above object, an image processing method according to a fourth aspect of the present invention obtains image data, obtains photographing information related to the image data, and the image based on the photographing information. Determining information relating to a shooting scene or subject in the data as scene / subject related information, searching for assist information in accordance with the determined scene / subject related information, and the determined scene / subject related information; When the similarity with the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted, and the determined scene / subject related information And the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the search is performed. Determining that the display output of the assist information is not permitted, and displaying the assist information permitted to be displayed on the display unit together with the corresponding display image data of the image data. As a result, the shooting information related to the posture or moving speed of the imaging device is acquired, and shooting in the same state is performed based on the shooting information acquired after shooting corresponding to the display image data is performed. It is determined whether or not the shooting is completed corresponding to the data, and display of the searched assist information is permitted when it is determined that the shooting is completed.

In order to achieve the above object, an image processing program according to a fifth aspect of the present invention includes a function of acquiring image data, a function of acquiring shooting information related to the image data, and the image based on the shooting information. A function for determining information relating to a shooting scene or subject in the data as scene / subject related information, a function for searching for assist information according to the determined scene / subject related information, and the determined scene / subject related information When the similarity with the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted, and the determined scene / subject related information And the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the detection is performed. A function of determining not to allow the display output of the assist information, thereby realizing the function of displaying assist information which the display output is permitted on the display unit together with the corresponding display image data among the image data to the computer The image data is associated with the scene / subject-related information, and the history management unit stores history information of a plurality of the scene / subject-related information. , According to the degree of similarity between the scene / subject-related information of the display image data and the scene / subject-related information of the image data acquired by the image acquisition unit after shooting corresponding to the display image data. The scene / subject related information of the display image data is continuously similar at the end of the scene / subject related information. It determines whether Luke, to realize the function of determining whether to permit the display output of the retrieved assisting information according to the determination result.
In order to achieve the above object, an image processing program according to a sixth aspect of the present invention includes a function of acquiring image data, a function of acquiring shooting information related to the image data, and the image based on the shooting information. A function for determining information relating to a shooting scene or subject in the data as scene / subject related information, a function for searching for assist information according to the determined scene / subject related information, and the determined scene / subject related information When the similarity with the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted, and the determined scene / subject related information And the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the detection is performed. The computer realizes a function of determining that display output of the assist information is not permitted and a function of displaying the assist information permitted to be displayed on the display unit together with the corresponding display image data of the image data. An image processing program for acquiring imaging information related to an attitude or a moving speed of an imaging apparatus as the imaging information, and based on the imaging information acquired after imaging corresponding to the display image data is performed A function to determine whether or not shooting in the same state is completed by shooting corresponding to the display image data, and to permit display output of the searched assist information when it is determined that shooting is completed. Let

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program capable of controlling the frequency of presenting assist information to the extent that does not interfere with the user's shooting action.

1 is a block diagram illustrating an overall configuration of an imaging apparatus as an application example of an image processing apparatus according to each embodiment of the present invention. It is a figure which shows the flowchart which concerns on the assist process by the control part of the imaging device which concerns on 1st Embodiment. It is the figure which showed the assist possible display superimposed on the through image display. It is a figure which shows an example of the display form of assist information. It is a figure which shows the flowchart which concerns on the assist process by the control part of the imaging device which concerns on 2nd Embodiment. It is a figure which shows the flowchart which concerns on the assist process by the control part of the imaging device which concerns on 3rd Embodiment. It is a figure which shows the flowchart which concerns on the assist process by the control part of the imaging device which concerns on 4th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
First, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing an overall configuration of an imaging apparatus as an application example of an image processing apparatus according to each embodiment of the present invention. The imaging apparatus 1 illustrated in FIG. 1 can be applied to various portable terminal devices such as a digital camera, a camera-equipped mobile phone, and a smartphone. The imaging apparatus 1 includes a control unit 102, an image acquisition unit 104, a shooting information acquisition unit 106, a determination unit 108, an assist information search unit 110, an assist information database (DB) 112, and a history management unit 114. , An output determination unit 116, a recording unit 118, a ROM 120, a RAM 122, an operation unit 124, and a display unit 126. As shown in FIG. 1, the control unit 102, the image acquisition unit 104, the shooting information acquisition unit 106, the determination unit 108, the assist information search unit 110, the assist information DB 112, the history management unit 114, The output determination unit 116, the recording unit 118, the ROM 120, the RAM 122, the operation unit 124, and the display unit 126 are connected to each other via a bus 128 so that they can communicate with each other.

  The control unit 102 includes, for example, a CPU, and controls the overall operation of the imaging apparatus 1. For example, the control unit 102 controls the image acquisition operation by the image acquisition unit 104 or causes the display unit 126 to display an image based on the image data acquired by the image acquisition unit 104. Further, the control unit 102 performs image processing on the image data acquired by the image acquisition unit 104 to generate an image file, and records the generated image file in the recording unit 118.

  The image acquisition unit 104 acquires image data. The image acquisition unit 104 is an imaging unit having, for example, a lens, a diaphragm, a shutter, an imaging device, an A / D conversion circuit, and the like. When the image acquisition unit 104 is an imaging unit, the image acquisition unit 104 converts an optical image of a subject incident through the lens into an analog electrical signal in the imaging device, and the analog electrical signal is digitally converted in the A / D conversion circuit. Image data is acquired by converting it into a signal.

  The shooting information acquisition unit 106 acquires shooting information related to the image data acquired by the image acquisition unit 104. The shooting information here includes the subject distance at the time of shooting, the brightness of the image data, the color, the image feature amount, the amount of motion of the subject or the imaging device 1, and the attitude of the imaging device 1. As the image feature amount, for example, LBP (Local Binary Pattern), SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated Brief), HSV histogram, and the like are used.

  The determination unit 108 determines the shooting scene and main subject of the image data based on the shooting information acquired by the shooting information acquisition unit 106. For example, the determination unit 108 determines that the scene is a night scene shooting scene when the luminance of the background area is low, and determines that the scene is a sunset shooting scene if the background area includes a large amount of red component, and compares the background area with the luminance of the subject. When the brightness of the is high, it is determined that the scene is a backlight photographing scene. The determination unit 108 determines that the subject is a macro shooting scene when the subject distance corresponds to a macro area set for each lens. Further, the determination unit 108 determines the main subject by comparing the similarity between the image feature amount extracted as the shooting information from the image data and the image feature amount for each subject recorded in advance in the recording unit 118, for example. To do.

  The assist information search unit 110 searches the assist information DB 112 for assist information corresponding to the determination result (scene / subject related information) of the shooting scene or main subject in the determination unit 108. The assist information DB 112 stores assist information in association with the type of shooting scene and the type of subject. The assist information is information including images (sample images) showing examples of various photographs and information showing shooting conditions (for example, aperture value, shutter speed, image processing setting) used for shooting each sample image. In addition, the assist information may include guide information for explaining various points such as a composition for capturing an image equivalent to the sample image to the user. Moreover, it is desirable that the assist information stored in the assist information DB 112 is sequentially updated when the imaging apparatus 1 is connected to a network, for example.

  The history management unit 114 manages the shooting scene and main subject determination result history (scene / subject-related information history) in the determination unit 108. The history management unit 114 of the first embodiment stores the previous (latest) determination result by the determination unit 108.

  The output determination unit 116 determines whether to display and output the assist information searched by the assist information search unit 110 according to the determination result history managed by the history management unit 114. Then, when the output determination unit 116 determines to display and output the assist information, the output determination unit 116 notifies the control unit 102 to display the assist possible display on the display unit 126.

  The recording unit 118 is, for example, a flash memory, and records the image data obtained by shooting and the image feature amount for each subject used when the shooting scene / determination unit 108 determines the subject. Here, it is desirable that the image feature amount for each subject is sequentially updated by machine learning using, for example, SVM (Support Vector Machine). Furthermore, the recording unit 118 records various programs executed by the control unit 102 and various parameters used when the programs are executed.

  The ROM 120 records programs and various parameters necessary for the control unit 102 to control the imaging apparatus 1. The RAM 122 temporarily stores various data such as image data obtained by the image acquisition unit 104 and image data being processed by the control unit 102. The operation unit 124 is an operation unit for the user to perform various operations on the imaging apparatus 1. The operation unit 124 may be a mechanical operation unit such as a button, or may be an operation unit using a touch panel. Depending on the operation of the operation unit 124 by the user, an instruction to perform a shooting operation is given, and various settings such as a shooting scene mode are made. The display unit 126 is a display unit such as a liquid crystal display or an organic EL display, and displays various images according to the control of the control unit 102. The display unit 126 also displays assist information searched by the assist information search unit 110, or displays an assist permission display according to the determination result of the output determination unit 116.

  Hereinafter, the assist process by the control unit 102 of the imaging apparatus 1 according to the first embodiment will be described. FIG. 2 is a diagram illustrating a flowchart related to assist processing by the control unit 102 of the imaging apparatus 1 according to the first embodiment. Here, prior to the processing of this flowchart, the history management unit 114 displays the history information of the determination result such as the shooting scene and the type of main subject determined at the time of assist information search in the previous assist processing as scene / subject related information. Assume that it is stored as a history.

  In FIG. 2, the control unit 102 causes the image acquisition unit 104 to acquire image data (step S101). For example, when the image acquisition unit 104 is an imaging unit, the image acquisition unit 104 images a subject and acquires image data. After the image acquisition, the control unit 102 causes the shooting information acquisition unit 106 to acquire shooting information (step S102). At this time, the shooting information acquisition unit 106 acquires shooting information such as the subject distance, the brightness of the image data, the color, and the image feature amount.

  Subsequently, the control unit 102 causes the determination unit 108 and the assist information search unit 110 to search for assist information (step S103). At this time, the determination unit 108 determines the shooting scene and main subject of the currently acquired image data according to the shooting information acquired by the shooting information acquisition unit 106. Then, the assist information search unit 110 searches the assist information DB 112 for assist information according to the shooting scene and main subject determined by the determination unit 108.

  Next, the control unit 102 causes the output determination unit 116 to make a determination. At this time, the output determination unit 116 is similar to the scene / subject-related information history that is the determination result history information stored in the history management unit 114 and the scene / subject-related information indicating the determination result obtained in step S103. It is determined whether or not (step S104). Here, for example, it is determined whether the similarity of the shooting information (image feature amount or the like) used for determining the shooting scene or the main subject is higher than a predetermined value. In the present embodiment, the current scene / subject related information is compared with the previous scene / subject related information history stored in the history management unit 114, that is, the previous previous one.

  If it is determined in step S104 that the degree of similarity between the current scene / subject-related information and the scene / subject-related information history is high, there is a high possibility that assist information similar to the previous assist information has been searched. Therefore, the output determination unit 116 notifies the control unit 102 that display of assist information is not permitted. At this time, the control unit 102 ends the process of FIG. In this case, assist information is not displayed. Processing other than the display of assist information is performed as usual. For example, when the assist process is performed during the through image display, the assist information is not displayed, but the through image display is continued.

  If it is determined in step S104 that the similarity between the scene / subject related information and the scene / subject relation information history is low, the output determination unit 116 notifies the control unit 102 that the display of assist information is permitted. In response to this, the control unit 102 performs an assistable display on the display unit 126 (step S105). Then, the control part 102 complete | finishes the process of FIG.

  For example, when the assist process is performed during the through image display, the assist enable display is performed so as to overlap the through image display. When the assist process is not performed during the through image display, the assist possible display is performed via an indicator (not shown) provided in the imaging apparatus 1. FIG. 3 shows the assistable display superimposed on the live view display. In FIG. 3, the character “i” is displayed, but the display method of assistable display is not limited to FIG. Note that when a user operation (for example, an operation of touching the display of “i”) is performed in a state where the assistable display as illustrated in FIG. 3 is performed, the assist information searched in step S103 is displayed.

  FIG. 4 is a diagram illustrating an example of a display form of assist information. In the example illustrated in FIG. 4, assist information is displayed in a display area adjacent to the through image display area. The assist information in FIG. 4 is assist information when it is determined that the main subject is “sunflower”. In this case, for example, guide information (information for guiding the composition in the figure) for guiding points for photographing the sample image is displayed as assist information together with the sample image of the sunflower. Here, FIG. 4 shows an example in which two pieces of assist information are displayed on the display unit 126 at the same time, but the number of pieces of assist information displayed on the display unit 126 at the same time is not limited to two. In addition, when there is assist information that cannot be displayed at the same time because the area of the display area of the display unit 126 is limited, the assist information that is not displayed is displayed using scrolling of the display area or the like. You may make it make it.

  In addition, in any case where the user has taken a picture with reference to the assist information and the user has taken a picture without referring to the assist information, if the assist possible display is made, the assist process at that time The scene / subject related information obtained at this time is stored in the history management unit 114, and the information in the history management unit 114 is updated. Alternatively, the information in the history management unit 114 is updated every time it is determined whether or not the scene / subject related information and the scene / subject related information history are similar, regardless of whether or not the assist enable display is performed. You may make it do. In this case, when the assist information is further displayed, the information “Assist information is displayed” may be further added as a history. In the method in which the information in the history management unit 114 is updated when the assistable display is made, the information in the history management unit 114 is not updated when the assistable display is not made, and the calculation processing amount can be reduced. . On the other hand, in the method in which the information in the history management unit 114 is updated every time it is determined whether or not the scene / subject-related information history is similar to the scene / subject-related information history, the assistable display is not performed. In addition, since the information of the history management unit 114 is updated, the history management unit 114 is similar to the immediately preceding scene / subject related information history in comparison with the method in which the information of the history management unit 114 is not updated when the assistable display is not made. It can be determined whether or not. Thereby, the frequency with which the same type of information is repeatedly presented can be controlled in consideration of time in addition to the order in the history.

  As described above, according to the first embodiment, the current scene / subject related information is compared with the immediately previous scene / subject related information history stored in the history management unit 114. The display of the assist information is controlled so that the assist information can be displayed only when the similarity is low. As a result, while the user is continuously shooting the same type of subject (not limited to continuous shooting), similar assist information is not continuously displayed, and the user's shooting action is disturbed. There is no. In addition, when the display of assist information is permitted, the assist information is not displayed immediately, but the assist-enabled display is performed first, so that the assist information is unexpectedly displayed and disturbs the user's photographing action. Can also be prevented.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In order to avoid duplication of explanation, differences from the first embodiment will be described. The second embodiment is an example in which the display of assist information is controlled not only immediately before but also by comparison with a predetermined number of past scene / subject related information histories. Accordingly, the history management unit 114 in the second embodiment stores scene / subject-related information up to a plurality of previous times as history information.

  Hereinafter, an assist process by the control unit 102 of the imaging apparatus 1 according to the second embodiment will be described. FIG. 5 is a diagram illustrating a flowchart related to assist processing by the control unit 102 of the imaging apparatus 1 according to the second embodiment. In order to avoid duplication of explanation, differences from the first embodiment will be described. That is, the processing in steps S201 to S203 is the same as the processing in steps S101 to S103 in FIG.

  After searching the assist information, the control unit 102 causes the output determination unit 116 to make a determination. At this time, the output determination unit 116 includes information on the determination result (scene / subject related information) obtained in step S203 and history information (predetermined number of past subject related information stored in the history management unit 114). It is determined whether or not there is information similar to the scene / subject-related information history (step S204).

When it is determined in step S204 that there is similar information between the current scene / subject-related information and the predetermined number of past scene / subject-related information histories, the output determination unit 116 displays the assist information. The control unit 102 is notified that display is not permitted. At this time, the control unit 102 ends the process of FIG. In this case, assist information is not displayed. On the other hand, when it is determined in step S204 that there is no similar information between the scene / subject related information and the predetermined number of past scene / subject related information histories, the output determination unit 116 displays the assist information. The control unit 102 is notified that display is permitted. In response to this, the control unit 102 performs an assistable display on the display unit 126 (step S205). Thereafter, the control unit 102 ends the process of FIG.
The processing in steps S204 to S205 will be specifically described. For example, assume that the determination results of the past two times are used. Here, assuming that the determination result of the main subject in the six assist processes is the order of flower → dish → flower → dish → dish → flower, in step S204 in each assist process, the determination is as follows. Made.
Flowers (○) → Cooking (○) → Flowers (×) → Cooking (×) → Cooking (×) → Flowers (○)
○: Assist information display is permitted ×: Assist information display is not permitted That is, since the types of main subjects determined in the first and second assist processes are different, the first and second assist processes In, display of assist information is permitted. On the other hand, “flower”, which is the type of the main subject determined in the third assist process, is the same as the type of the main subject two times before, and thus display of assist information is not permitted in the third assist process. . Similarly, in the fourth and fifth assist processes, there is a main subject of the same type two times before, so display of assist information is not permitted in the fourth and fifth assist processes. Finally, “flower”, which is the type of the main subject determined in the sixth assist process, is different from “cooking”, which is the type of the main subject up to the second time, so in the sixth assist process, Display of assist information is allowed.

  As a tendency of shooting, it is generally possible to continuously shoot the same kind of subject even if continuous shooting is not performed. In this case, even if assist information for the same type of subject is selected from a plurality of types each time, there is a high possibility that the assist information is similar, and there is a high possibility that similar assist information is displayed. On the other hand, according to the second embodiment, similar assist information can be continuously displayed by controlling the display of assist information based on the continuity of similar scene / subject related information. And the user's shooting action is not disturbed.

  Here, in the above-described specific example, the past two determination results are used, but this number is not limited. This number of times may be set by the user.

[Modifications of First Embodiment and Second Embodiment]
Hereinafter, modifications of the first and second embodiments will be described. The output determination unit 116 in the first and second embodiments controls the display of assist information according to the similarity between the current scene / subject related information and the past scene / subject related information. However, it is not always necessary to control the display of assist information according to only the similarity of scene / subject related information, and the display of assist information may be controlled according to other information.

  For example, if the difference between the time when the previous scene / subject related information is determined and the time when the current scene / subject related information is determined is greater than or equal to a predetermined value, a predetermined number of past scenes / Even if the subject-related information includes information similar to the current scene / subject-related information, it can be evaluated that the similarity of the assist information is low. Therefore, in this case, display of assist information may be permitted.

  For example, even if the main subject type and the shooting scene type in the scene / subject related information are similar, if the assist information is sufficiently different, the display of the assist information may be permitted. In other words, flower assist information centered on macro photography was searched during the previous assist process, and the art filter was applied with a pulling composition even if the current main subject determination result was “flower” When the flower assist information is searched, it can be evaluated that the assist information is different even if the assist information is related to the subject type “flower”. Therefore, in this case, display of assist information may be permitted. Here, the determination of the difference in assist information is performed by, for example, setting a tag expressing the type of information in each assist information and evaluating the difference between the tags. For example, by treating a plurality of tags assigned to a plurality of assist information to be presented as BOW (Bag of words), a difference in each assist information can be quantitatively expressed as a difference in the number of tags.

  In the first and second embodiments, an example is shown in which the assist process is performed during live view display. For example, the assist process is performed when a captured image is displayed immediately after shooting (during rec view). Also good. That is, in this case, the image acquisition unit 104 does not acquire the image data by capturing the subject, but acquires the image data obtained as a result of the shooting. The processing after the acquisition of the image data is performed in the same manner as the processing shown in FIG. Even if the assist information is displayed during the REC view, it does not interfere with the photographing action, and at the same time, the assist information is presented immediately after photographing, so that an effect of prompting the user to photograph again can be expected. Here, in the case of the REC view, it is less likely that the user's shooting action will be hindered by displaying the assist information than during the live view display. Therefore, the determination threshold value may be set so that the assist information display frequency is higher in the assist process during the rec view than in the assist process during the live view display.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In order to avoid duplication of explanation, differences from the first embodiment will be described. The assist processing according to the third embodiment mainly assumes processing during reproduction of image data. The ability to reproduce image data means that the image data has already been captured. In this case, scene / subject-related information corresponding to the image data being reproduced is also obtained. In the present embodiment, when image data is captured, scene / subject related information about the image data is sequentially stored in the history management unit 114. The scene / subject-related information stored for each image data is used for assist processing during reproduction of the image data.

  Hereinafter, an assist process by the control unit 102 of the imaging apparatus 1 according to the third embodiment will be described. FIG. 6 is a diagram illustrating a flowchart relating to assist processing by the control unit 102 of the imaging apparatus 1 according to the third embodiment.

  In FIG. 6, the control unit 102 causes the image acquisition unit 104 to acquire image data (step S301). In the third embodiment, the image acquisition unit 104 acquires, from the recording unit 118, image data instructed to be reproduced by the user during the reproduction mode. After acquiring the image data, the control unit 102 causes the output determination unit 116 to acquire scene / subject-related information of the current image data acquired by the image acquisition unit 104 (step S302). At this time, the output determination unit 116 acquires scene / subject-related information associated with the current image data from the history management unit 114.

  Subsequently, the control unit 102 causes the output determination unit 116 to acquire scene / subject-related information of the image data that was captured after the current image data, that is, the imaging date and time is later (step S303). At this time, the output determination unit 116 acquires from the history management unit 114 the scene / subject-related information that is one after the image data corresponding to the previous scene / subject-related information. After acquiring the scene / subject related information, the output determining unit 116 determines whether or not the number of scene / subject related information acquired has reached a predetermined number (step S304). If it is determined in step S304 that the number of acquired scene / subject related information has not reached the specified number, the process returns to step S303. That is, the processes in steps S303 to S304 are repeated until the number of scene / subject-related information acquisition reaches a specified number.

  If it is determined in step S304 that the number of acquired scene / subject related information has reached the specified number, the output determination unit 116 captures the scene / subject related information of the current image data after the current image data. It is determined whether the image data is included in the scene / subject-related information (step S305). Whether the current scene / subject-related information is included in the subsequent scene / subject-related information is determined according to various determination criteria described in the second embodiment and its modifications. That is, when there is scene / subject-related information with high similarity, it is determined that the information is included.

  If it is determined in step S305 that the scene / subject-related information of the current image data is included in the scene / subject-related information of the image data captured after the current image data, the image data being reproduced This indicates that the scene / subject-related information of (current image data) is not information on the end of the same type of scene / subject-related information. At this time, the output determination unit 116 notifies the control unit 102 that display of assist information is not permitted. In response to this, the control unit 102 ends the process of FIG. In this case, assist information is not displayed.

  If it is determined in step S305 that the scene / subject-related information of the current image data is not included in the scene / subject-related information of the image data captured after the current image data, the image data being reproduced This indicates that the scene / subject-related information is the end information of the same type of scene / subject-related information. At this time, the output determination unit 116 notifies the control unit 102 that the display of assist information is permitted. In response to this, the control unit 102 performs an assistable display on the display unit 126 (step S306). Thereafter, the control unit 102 ends the process of FIG.

The processing in steps S304 to S305 will be specifically described. For example, it is assumed that the determination result of the second time is used for the current image data. Here, assuming that the scene / subject-related information is in order of flowers → flowers → flowers → cooking → flowers → clouds → clouds, the following determination is made in step S304.
Flowers (×) → Flowers (×) → Flowers (×) → Cooking (○) → Flowers (○) → Clouds (×) → Clouds (×)
○: Display of assist information is permitted ×: Display of assist information is not permitted In other words, if the currently reproduced image data is the first captured image data, the subject type of the image data is “flower”. On the other hand, the subject type of the image data two times behind this image data is also “flower”. In this case, the display of the assist information is not permitted because the current subject type is included in the subject types twice in the rear. Also, assuming that the current image data being reproduced is image data taken for the second time, the subject type of this image data is “flower”, whereas the subject type of the next image data of this image data is Is “flower”, and the subject type of the next image data is “cooking”. In this case, the display of the assist information is not permitted because the current subject type is included in the subject types twice in the rear. Also, assuming that the current image data being reproduced is image data taken for the third time, the subject type of this image data is “flower”, whereas the subject type of the next image data of this image data is Is “cooking”, and the subject type of the next image data is “flower”. In this case, the display of the assist information is not permitted because the current subject type is included in the subject types twice in the rear. Also, if the image data being reproduced is image data taken for the fourth time, the subject type of this image data is “dishes”, whereas the subject type of the next image data of this image data is Is “flower”, and the subject type of the next image data is “cloud”. In this case, the display of assist information is permitted because the current subject type is not included in the two rear subject types. Thereafter, the display of assist information is similarly controlled.

  When a user browses an image, if the assist information is displayed at the beginning of a series of subject types as in the second embodiment, the browsing action may be hindered. On the other hand, according to the present embodiment, the assist information is displayed at the end of a series of subject types, so that the user can view the assist information without being obstructed to view a series of images.

  Here, in the present embodiment, an example in which all scene / subject-related information is acquired from the history management unit 114 is shown. This is because scene / subject-related information is obtained at the time of shooting. On the other hand, determination of scene / subject-related information may be performed during the playback mode. When the scene / subject related information is determined during playback, the determination result is stored in the history management unit 114, and the determination result stored in the history management unit 114 is acquired and the assist process is performed from the next time onward. You may be made to be.

[Modification of Third Embodiment]
Hereinafter, modifications of the third embodiment will be described. Similar to the first and second embodiments, in the third embodiment, the display of assist information is controlled according to the similarity of scene / subject related information. However, it is not always necessary to control the display of assist information according to only the similarity of scene / subject related information, and the display of assist information may be controlled according to other information. For example, it is effective to use the reproduction time of the image data being reproduced. Specifically, even in a situation where the similarity is determined to be high based on scene / subject-related information, that is, in a situation where the display of assist information is not permitted, the playback time of the current image data is predetermined. If it is longer than the time, it is considered that the user is paying attention to the image being reproduced, and therefore control is performed so that display of assist information related to this image data is permitted. As a result, the user can easily view the assist information by gazing (displaying) the image of interest for a certain period of time.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. In order to avoid duplication of explanation, differences from the first embodiment will be described. In the fourth embodiment, the posture information of the imaging device 1 is acquired as shooting information. Then, assist information is controlled in accordance with the posture difference before and after photographing. Note that the posture information is acquired using, for example, an acceleration sensor.

  Hereinafter, an assist process by the control unit 102 of the imaging apparatus 1 according to the fourth embodiment will be described. FIG. 7 is a diagram illustrating a flowchart relating to assist processing by the control unit 102 of the imaging apparatus 1 according to the fourth embodiment.

  In FIG. 7, the control unit 102 causes the image acquisition unit 104 to acquire image data (step S401). Subsequently, the control unit 102 causes the shooting information acquisition unit 106 to acquire posture information as shooting information (step S402). Thereafter, the control unit 102 causes the display unit 126 to display an image based on the image data acquired by the image acquisition unit 104 (step S403). The display in step S403 is a through image display or a REC view. After displaying the image, the control unit 102 causes the shooting information acquisition unit 106 to acquire posture information as shooting information (step S404).

  Subsequently, the control unit 102 causes the output determination unit 116 to make a determination. At this time, the output determination unit 116 determines whether the difference between the posture information obtained in step S402 and the posture information obtained in step S404, that is, the posture difference before and after the acquisition of the image data is equal to or greater than a predetermined threshold value. Is determined (step S405).

  If the posture difference is less than the threshold value in step S405, it means that the user points the imaging device 1 in the direction of the same subject before and after the acquisition of the image data and continues the shooting action. In this case, it can be said that the degree of similarity with the assist information is high. Therefore, the output determination unit 116 notifies the control unit 102 that display of assist information is permitted. At this time, the control unit 102 ends the process of FIG. In this case, assist information is not displayed.

  If the posture difference is greater than or equal to the threshold value in step S405, it means that the user points the imaging device 1 in different object directions before and after the acquisition of the image data. At this time, the output determination unit 116 notifies the control unit 102 that the display of assist information is permitted. Receiving this, the control unit 102 performs an assistable display on the display unit 126 (step S406). Thereafter, the control unit 102 ends the process of FIG.

  For example, when the user is continuously shooting cherry blossoms, the shooting is continued in a posture slightly facing upward to capture the cherry blossoms. Therefore, even during rec view, the deviation from the shooting posture is small. On the other hand, unlike the shooting posture, the last one that has been shot is considered to be in a posture (viewing state) that is easy to watch the screen and that faces slightly downward. In this case, since it is greatly different from the posture at the time of shooting, it is determined that the shooting state is completed, and the display of assist information is permitted. In the case of the fourth embodiment, the assist information permitted to be displayed is assist information corresponding to the scene / subject-related information about the image data acquired last (acquired in step S401).

  As described above, according to the present embodiment, the user can concentrate a series of photographing actions without being hindered by the assist information. In addition, by permitting the display of the assist information after the situation (viewing state) is sufficient for browsing the assist information, the user can view the assist information with a margin.

[Modification of Fourth Embodiment]
Hereinafter, modifications of the fourth embodiment will be described. In the case of the fourth embodiment, the assist information permitted to be displayed is, for example, assist information according to scene / subject-related information for the image data acquired last. On the other hand, assist based on information (for example, information obtained by arranging a plurality of assist information) obtained by integrating scene / subject related information obtained in the process from the previous viewing state (after the posture has changed) to the current viewing state. It is also effective to allow the display of information. In this case, after continuously shooting various subjects, the user can easily obtain assist information regarding the various subjects.

  It is also effective to determine whether or not the shooting state is continued using not only the camera posture but also the moving speed before and after the acquisition of image data during live view display or REC view. The moving speed can be acquired using, for example, an acceleration sensor. In such a modification, the shooting information acquisition unit 106 acquires the moving speed as shooting information. If the moving speed is equal to or higher than a predetermined value, the output determination unit 116 determines that the user is not gazing at the live view or the REC view and is performing an operation for the next shooting. Do not allow display of assist information.

  Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are naturally possible within the scope of the gist of the present invention. Further, in the description of each operation flowchart described above, the operation is described using “first”, “next”, and the like for convenience, but this does not mean that it is essential to perform the operations in this order. Absent.

  Each process according to the above-described embodiment can be stored as a program that can be executed by the control unit 102. In addition, it can be stored and distributed in a storage medium of an external storage device such as a memory card (ROM card, RAM card, etc.), magnetic disk (hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc. . Then, the control unit 102 reads the program stored in the storage medium of the external storage device, and the operation described above can be executed by controlling the operation by the read program.

  Further, the above-described embodiments include various stages of the invention, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some configuration requirements are deleted from all the configuration requirements shown in the embodiment, the above-described problem can be solved, and this configuration requirement is deleted when the above-described effects can be obtained. The configuration can also be extracted as an invention.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device 102 ... Control part 104 ... Image acquisition part 106 ... Shooting information acquisition part 108 ... Determination part 110 ... Assist information search part 112 ... Assist information DB 114 ... History management part 116 ... Output Judgment unit 118 ... Recording unit 120 ... ROM 122 ... RAM 124 ... Operation unit 126 ... Display unit 128 ... Bus

Claims (12)

An image acquisition unit for acquiring image data;
A shooting information acquisition unit that acquires shooting information related to the image data;
A determination unit that determines information about a shooting scene or subject in the image data as scene / subject related information based on the shooting information;
An assist information search unit that searches for assist information according to the scene / subject-related information determined by the determination unit;
A history management unit for storing history information of the scene / subject-related information;
When the similarity between the scene / subject related information determined by the determination unit and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, the retrieved assist information When the display output is permitted and the similarity between the scene / subject related information determined by the determination unit and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, An output determination unit that determines not to permit display output of the searched assist information;
A display unit for displaying the assist information permitted for the display output together with the corresponding display image data of the image data ;
Comprising
The image data is associated with the scene / subject-related information,
The history management unit stores history information of the plurality of scene / subject related information,
The output determination unit includes the scene / subject-related information of the image data acquired by the image acquisition unit by shooting after the shooting corresponding to the scene / subject-related information of the display image data and the display image data. The scene / subject related information of the display image data is determined to be the end of the similar scene / subject related information in succession, and the searched assist is determined according to the determination result. Determine whether to allow display output of information,
Image processing device. The output determination unit, when the reproduction time of the display image data to determine long or not than a predetermined time, it is determined that longer than the time the reproduction time of the display image data is predetermined, the Permit display output of searched assist information,
The image processing apparatus according to claim 1 . The output determination unit includes the scene / subject related information of the display image data and the scene / subject of the plurality of pieces of image data acquired by the image acquisition unit by shooting after shooting corresponding to the display image data. According to the similarity to the subject related information, it is determined whether the scene / subject related information of the display image data is the end of the scene / subject related information that is continuously similar, and the search is performed according to the determination result. To determine whether to allow display output of the assist information
The image processing apparatus according to claim 1. An image acquisition unit for acquiring image data;
A shooting information acquisition unit that acquires shooting information related to the image data;
A determination unit that determines information about a shooting scene or subject in the image data as scene / subject related information based on the shooting information;
An assist information search unit that searches for assist information according to the scene / subject-related information determined by the determination unit;
A history management unit for storing history information of the scene / subject-related information;
When the similarity between the scene / subject related information determined by the determination unit and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, the retrieved assist information When the display output is permitted and the similarity between the scene / subject related information determined by the determination unit and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, An output determination unit that determines not to permit display output of the searched assist information;
A display unit for displaying the assist information permitted for the display output together with the corresponding display image data of the image data;
Comprising
The imaging information acquisition unit acquires imaging information related to the posture or moving speed of the imaging device,
The output determination unit determines whether shooting in the same state has been completed by shooting corresponding to the display image data based on the shooting information acquired after shooting corresponding to the display image data is performed. Permitting display output of the searched assist information when it is determined that the search is completed.
Image processing device. The shooting information acquisition unit acquires posture information of the imaging device as the shooting information,
The output determination unit is based on the posture information when the shooting corresponding to the display image data is performed and the posture information acquired after the display corresponding to the display image data is performed. The posture difference after the shooting corresponding to the display image data is calculated, and when the calculated posture difference is equal to or larger than a predetermined difference, the shooting in the same state is completed by the shooting corresponding to the display image data. It is determined that
The image processing apparatus according to claim 4 . The imaging information acquisition unit acquires the moving speed of the imaging device as the imaging information,
When the movement speed acquired after shooting corresponding to the display image data is less than a predetermined speed , the output determination unit is configured to perform shooting in the same state by shooting corresponding to the display image data. Determine that it is complete ,
The image processing apparatus according to claim 4 . The image acquisition unit acquires the image data a plurality of times,
The output determination unit permits display output of assist information obtained by integrating the assist information searched for each of the image data acquired a plurality of times when permitting display output of the searched assist information.
The image processing apparatus according to any one of claims 1 to 6. The assist information includes at least one of sample image data, information indicating a photographing condition of the sample image data, and guide information for photographing the sample image data.
The image processing apparatus according to claim 1 . Acquiring image data,
Obtaining shooting information relating to the image data;
Determining information relating to a shooting scene or subject in the image data based on the shooting information as scene / subject related information;
Searching for assist information according to the determined scene / subject-related information;
When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted. When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the retrieved assist information is displayed. Determining that output is not allowed,
Displaying the assist information permitted to be displayed on the display unit together with the corresponding display image data of the image data ;
Equipped with,
The image data is associated with the scene / subject-related information,
The history management unit stores history information of the plurality of scene / subject related information,
According to the similarity between the scene / subject-related information of the display image data and the scene / subject-related information of the image data acquired by shooting after the shooting corresponding to the display image data, It is determined whether or not the scene / subject-related information is the end of the similar scene / subject-related information in succession, and whether display output of the searched assist information is permitted according to the determination result. judge,
An image processing method. Acquiring image data,
Obtaining shooting information relating to the image data;
Determining information relating to a shooting scene or subject in the image data based on the shooting information as scene / subject related information;
Searching for assist information according to the determined scene / subject-related information;
When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted. When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the retrieved assist information is displayed. Determining that output is not allowed,
Displaying the assist information permitted to be displayed on the display unit together with the corresponding display image data of the image data;
Comprising
As the shooting information, acquiring shooting information related to the posture or moving speed of the imaging device,
Based on the shooting information acquired after shooting corresponding to the display image data, it is determined whether shooting in the same state has been completed by shooting corresponding to the display image data. Permitting the display output of the searched assist information when it is determined that
An image processing method. A function to acquire image data;
A function of acquiring shooting information related to the image data;
A function of determining information related to a shooting scene or subject in the image data based on the shooting information as scene / subject related information;
A function of searching for assist information according to the determined scene / subject-related information;
When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted. When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the retrieved assist information is displayed. A function that determines that output is not permitted;
A function of causing the display unit to display the assist information for which the display output is permitted together with the corresponding display image data of the image data ;
An image processing program for realizing the to the computer,
The image data is associated with the scene / subject-related information,
The history management unit stores history information of the plurality of scene / subject related information,
According to the similarity between the scene / subject-related information of the display image data and the scene / subject-related information of the image data acquired by shooting after the shooting corresponding to the display image data, It is determined whether or not the scene / subject-related information is the end of the similar scene / subject-related information in succession, and whether display output of the searched assist information is permitted according to the determination result. An image processing program for realizing a determination function . A function to acquire image data;
A function of acquiring shooting information related to the image data;
A function of determining information related to a shooting scene or subject in the image data based on the shooting information as scene / subject related information;
A function of searching for assist information according to the determined scene / subject-related information;
When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is lower than a predetermined value, display output of the searched assist information is permitted. When the similarity between the determined scene / subject related information and the history information of the scene / subject related information stored in the history management unit is higher than the predetermined value, the retrieved assist information is displayed. A function that determines that output is not permitted;
A function of causing the display unit to display the assist information for which the display output is permitted together with the corresponding display image data of the image data;
An image processing program for causing a computer to realize
As the shooting information, acquiring shooting information related to the posture or moving speed of the imaging device,
Based on the shooting information acquired after shooting corresponding to the display image data, it is determined whether shooting in the same state has been completed by shooting corresponding to the display image data. An image processing program for realizing a function of permitting display output of the searched assist information when it is determined that.