JP7175772B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an imaging device and an imaging method .

2. Description of the Related Art In recent years, devices have been proposed that are configured to be able to use device operation histories by users. For example, the medical device disclosed in Patent Literature 1 is configured to store an operation performed on a medical image so that the operation can be reproduced later.

JP 2014-176516 A

2. Description of the Related Art In an imaging device such as a digital camera, a user performs various operations in order to take a desired image of a specific object or scene, check an image including the object, and edit the image. do. This operation often reflects the user's preference for each object.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and an image pickup apparatus and image pickup method that assist a user by reflecting the user's preference for each object or scene by utilizing the history of past operations. The purpose is to provide law .

The imaging device according to the first aspect of the present invention acquires a user's operation on a first image reproduced on a display element, and moves a first object in the first image according to the acquired history of the operation. determining whether or not a target object at the time of capturing a second image is a second target object similar to the first target object, and determining whether the target object at the time of capturing the second image is at least one control circuit configured to perform control for assisting shooting of the second image according to the history of the operation when the object is the second object, and reproduced on the display element The captured first image is an image captured by the user, and the control circuit defines an object included in the region confirmed by the user in the first image as the first object. It is determined whether or not control corresponding to the editing operation performed on the first image reproduced on the display element can be performed when the second image is captured, and performed. When it is determined that the first image can be edited, control corresponding to the editing operation performed on the first image is performed .

According to a second aspect of the present invention, there is provided an imaging method comprising : obtaining an operation by the user on a first image photographed by a user and reproduced on a display device; identifying a first object in the image; determining whether the object at the time of capturing the second image is a second object similar to the first object; performing control for assisting the photographing of the second image according to the operation history when the object is the second object at the time of photographing the second image ; An object included in the area confirmed by the user in the image is extracted as the first object, and when the second image is captured, the first image reproduced on the display device is determining whether or not control corresponding to the editing operation performed on the first image can be performed; and controlling .

According to the present invention, it is possible to provide an imaging apparatus and an imaging method that assist a user by reflecting the user's preference for each object or scene by using the history of past operations.

FIG. 1 is a diagram showing the configuration of an example of an imaging device according to the first embodiment. FIG. 2A is a diagram illustrating an example of a user's action while capturing a still image. FIG. 2B is a diagram illustrating an example of assist control at the time of shooting using the operation history at the time of shooting. FIG. 2C is a diagram illustrating another example of assist control during shooting using the operation history during shooting. FIG. 3A is a diagram illustrating an example of a user's action that reproduces a still image. FIG. 3B is a diagram illustrating an example of assist control at the time of shooting using the operation history at the time of playback. FIG. 4A is a diagram illustrating an example of a user's motion when shooting a moving image. FIG. 4B is a diagram illustrating an example of a user's action during video playback. FIG. 5 is a flow chart showing an example of the operation of the imaging device according to the first embodiment. FIG. 6 is a diagram showing an example of an image file containing an operation history. FIG. 7 is a flowchart showing an example of analysis processing performed after still image or moving image shooting. FIG. 8 is a flow chart showing an example of analysis processing performed during playback of a still image. FIG. 9 is a flowchart illustrating an example of analysis processing that is performed when playing back a moving image. FIG. 10 is a flowchart showing assist control during shooting. FIG. 11 is a diagram showing an example of a situation during photographing. FIG. 12 is a diagram showing an example of assist control for enlarged display. FIG. 13 is a flowchart showing assist control during reproduction. FIG. 14 is a diagram showing an example of assist control for frame selection. FIG. 15 is a diagram illustrating an example configuration of an imaging device according to the second embodiment. FIG. 16A is a diagram illustrating deep learning concepts for creating an inference engine 116a. FIG. 16B is a diagram illustrating deep learning concepts for creating the inference engine 116b. FIG. 17 is a flowchart illustrating an example of processing for generating an inference engine.

Embodiments of the present invention will be described below with reference to the drawings.
[First Embodiment]
A first embodiment will be described. FIG. 1 is a diagram showing the configuration of an example of an imaging device according to the first embodiment. The imaging device 100 has a control circuit 102 , an imaging unit 104 , a display element 106 , a memory 108 , a recording medium 110 , an operation unit 112 and a communication circuit 114 . The imaging device 100 has a display element 106 and also operates as a reproducing device for reproducing images. This imaging device 100 can be applied to various devices having an imaging function or a playback function, such as digital cameras, smart phones, tablet terminals, and portable game machines. The imaging device 100 may have a configuration other than that shown in FIG. 1, or may not have a part of the configuration shown in FIG.

A control circuit 102 controls the operation of each unit of the imaging apparatus 100 . The control circuit 102 includes hardware such as a CPU, GPU, ASIC, FPGA, and main memory. The control circuit 102 may be configured by a single CPU or the like, or may be configured by a plurality of CPUs or the like. Also, part of the processing by the control circuit 102 may be realized by software.

The imaging unit 104 acquires an image of an object. The imaging unit 104 includes, for example, an optical system 104a and an imaging device 104b. The optical system 104a allows light from the object to enter the imaging device 104b. The optical system 104a may have a zoom optical system for varying the focal length, or may have a focus optical system for varying the focus position. The imaging device 104b is, for example, an imaging device such as a CMOS imager or a CCD imager. The imaging device 104b converts light from the object into an electrical signal.

The memory 108 is memory such as RAM and ROM. The memory 108 stores programs and parameters for the operation of the imaging device 100 . In addition, the memory 108 stores information on the operation history of the operation unit 112 of the user. The information on the operation history is information on the history of user operations during image shooting or user operation history during image reproduction. Memory 108 may be single or multiple.

The recording medium 110 includes memory such as flash memory. Images captured by the imaging device 100 are recorded on the recording medium 110 . The recording medium 110 does not have to be built in the imaging device 100 . For example, the recording medium 110 may be configured to be detachable from the imaging device 100 .

The display element 106 displays various images. The display element 106 is a display element such as a liquid crystal display or an organic EL display. Moreover, the display element 106 may have a speaker or the like for reproducing sound.

The operation unit 112 includes mechanical operation members such as buttons, switches, and dials. Moreover, the operation unit 112 may include a touch panel. When the imaging device 100 includes a display element, the touch panel may be provided integrally with the display element.

The communication circuit 114 includes a circuit for the imaging device 100 to communicate with a server or the like external to the imaging device 100 . The communication circuit 114 is, for example, a communication circuit for Wi-Fi (registered trademark). Communication circuitry 114 may include, for example, communication circuitry for Bluetooth®. In addition, communication circuit 114 may include communication circuits for various wired communications.

The operation of the imaging device 100 according to the first embodiment will be described below. FIG. 2A shows an example of a user's behavior while taking a still image. In FIG. 2A , squares arranged on the left and right schematically represent captured images at each point in time, and it is assumed that time elapses from left to right, for example.

For example, when the operation of the imaging device 100 is set to the shooting mode, the live view L is displayed on the display element 106 of the imaging device 100 . A live view L is an image that is sequentially acquired by the imaging unit 104 through continuous operation of the imaging unit 104 . By displaying the live view L in real time, the user can use the display element 106 to check the object O to be photographed.

While the live view L is being displayed, the user performs various operations such as mode setting, framing, zooming, and focusing/exposure adjustment while watching the live view L. In FIG. 2A, operations are performed in the order of framing, zooming, and focusing/exposure. However, this order is an example. Mode setting, framing, zooming, focusing/exposure can be done in any order. Further, operations other than mode setting, framing, zooming, and focusing/exposure adjustment may be performed during display of the live view L, or any of the operations may not be performed.

In mode setting, the user sets the mode of the imaging device 100 to a specific mode according to the image to be captured. This mode includes modes such as high dynamic range (HDR) mode, focus stacking mode, super resolution mode, and RAW mode. The HDR mode is a mode in which an image with an expanded dynamic range is obtained by synthesizing a plurality of images obtained by photographing the same object multiple times with different exposures. Focus stacking mode is a mode in which a plurality of images obtained by photographing the same scene multiple times with different focus states are combined to obtain images of objects at different distances that are in focus at the same time. . The super-resolution mode obtains an image with higher resolution than the original image by synthesizing the images obtained by photographing the same object multiple times while shifting the position of the optical system 104a or the image sensor 104b. mode. The RAW mode is a mode in which an image obtained by the imaging unit 104 is recorded as it is on the recording medium 110 without image processing. In the embodiment, the control circuit 102 stores the mode operation history in the memory 108, for example, each time the mode is set. The mode operation history includes, for example, the name of the set mode.

In framing, the user moves the housing of the imaging device 100, for example, while confirming the live view L, so that the desired object O is placed in the desired position in the live view L. In order to store such operations, the control circuit 102 causes the image of the live view L to be stored in the memory 108, for example. This live view L can be used as an operation history for framing. If the imaging device 100 has a motion sensor, the motion of the imaging device 100 detected by the motion sensor may be used as the framing operation history. Furthermore, in framing, when the object is too small to be seen or when the user wants to see the object in more detail, the user may perform an operation to enlarge and display the object of interest. The control circuit 102 may cause the memory 108 to store a history of enlarging operations in the shooting mode as an operation history.

In zooming, the user uses an optical zoom or an electronic zoom so that the size of the object O displayed in the live view L becomes a desired size. The control circuit 102 stores a zooming operation history in the memory 108, for example, each time an operation related to zooming is performed. The zooming operation history includes an optical zoom operation history and an electronic zoom operation history. The optical zoom operation history includes, for example, the focal length of the zoom optical system. The electronic zoom operation history includes, for example, the position of the area in the electronically zoomed image and the zoom magnification. When the object can be identified, the zooming operation history may include information on the object that has been zoomed. The object information includes, for example, the name of the object. The object information may also include information representing the structural features of the object in the captured image, such as edge shape, brightness, and color.

In focusing, the focus of the optical system 104a of the imaging unit 104 is adjusted with respect to a specific object. The control circuit 102 causes the memory 108, for example, to store the operation history of focusing each time the focusing is performed. The operation history of focusing includes, for example, information indicating whether or not focusing was performed automatically, determination information used when focusing was performed, the position of the focusing optical system, and the image to be focused. Contains the location of the region within the . The determination information includes information such as a two-image interval value and a defocus amount in the case of the phase difference method, and contrast difference in the case of the contrast method. When the target can be specified, the focusing operation history may include information on the target for focusing.

In exposure matching, the aperture and shutter speed are adjusted so that the brightness of a specific object is appropriate when photographing. The control circuit 102 causes the memory 108, for example, to store the operation history of the exposure adjustment each time the exposure adjustment is performed. The operation history of exposure adjustment includes, for example, information indicating whether or not exposure adjustment was performed automatically, and the position of the area within the image in which exposure adjustment was performed. When the target can be identified, the exposure adjustment operation history may include information on the target for exposure adjustment.

While checking the live view L, the user gives an instruction to shoot a still image at a desired timing. In response to this, the control circuit 102 performs necessary processing for still image shooting. As a result, a still image is recorded on the recording medium 110 . The control circuit 102 records, for example, the recording medium 110 in association with the still image obtained as a result of photographing each operation history up to photographing.

After recording the operation history, the control circuit 102 analyzes the operation history. For example, the control circuit 102 extracts the object from each of the still image and the live view obtained as the imaging result. Object extraction can be performed by techniques such as edge extraction and image matching. An object appearing in a still image obtained as a result of photographing can be determined to be an object in which the user is interested in the current photographing. On the other hand, an image may contain multiple objects. When an image contains multiple objects, for example, the object closest to the center of the image, the closest object, the focused or exposed object , an object that is in the live view for the longest time, that is, the object that is most captured in the angle of view by the user's framing can be determined to be the object that the user is interested in.

Also, for example, the control circuit 102 classifies the set modes for each object. That is, if a specific mode is set many times for a certain object, it can be determined that there is a high possibility that the mode will be set for the next similar object to be photographed. Based on such a determination, the control circuit 102 can, for example, suggest using that particular mode the next time a similar object is photographed.

Also, for example, the control circuit 102 classifies the areas that have been enlarged, zoomed, focused, and exposed for each object. For an object, the number of times a specific position of the object is magnified during live view, the number of times zooming is applied to the object, or the specific position is not in focus. If there is a large number of times that the camera is in a specific position or the number of times that the exposure is adjusted to a specific position, the next time a similar object is photographed, the specific position will be magnified, zoomed, focused, or exposed. can be determined to be likely to occur. Based on such a determination, the control circuit 102 preliminarily enlarges the specific position, zooms the target, adjusts the focus or exposes the target, and adjusts the focus or exposure for the target when photographing a similar target next time. Control such as alignment can be performed.

The analysis of the operation history is not limited to that performed by classifying the operation history for each object. For example, when the operation history is obtained as a numerical value, various statistical processing such as calculating the average value of this numerical value may be used. These statistical processes may be used alone or in combination.

FIG. 2B is an example of assist control at the time of shooting using the operation history at the time of shooting. For example, the control circuit 102 determines whether the object currently being photographed is similar to an object or scene that has been photographed in the past. This determination is made, for example, by comparing the live-view image with the feature information of the object stored in the memory 108 in advance. Then, when it is determined that the object currently photographed is similar to the object or scene photographed in the past, the control circuit 102, for example, as shown in FIG. An arrow mark A is displayed to indicate to the user that it has entered the . It is expected that the user who sees the arrow mark A recognizes that an object of interest to him/herself is within the angle of view, and performs a photographing operation. Alternatively, users may have different preferences for color reproduction of scenes such as sunsets, which are colored in various shades. Since such a scene also consists of objects such as the sky and clouds, it can be called an object.

Further, when the control circuit 102 determines that the current object to be photographed is an object that has been photographed in the HDR mode in the past, the control circuit 102 instructs the user to photograph in the HDR mode, for example, as shown in FIG. 2B. display a message M1 proposing to At this time, the control circuit 102 may capture an image of the object in the HDR mode without waiting for the user's instruction to start capturing, and store the captured image in the HDR mode in the memory 108, for example. In this case, when the user actually instructs the start of photographing of the object, the control circuit 102 displays the photographed image obtained in advance in HDR mode photographing together with the photographed image obtained in response to the instruction to start photographing. may be recorded on the recording medium 110 .

Furthermore, the control circuit 102 may display a mark for guiding the direction of framing so that the object O can be aimed. Further, when a specific portion of the object was displayed in an enlarged manner when the previous image was captured, the control circuit 102 may enlarge and display the specific portion in the live view during the current capturing. Further, when the control circuit 102 determines that a specific portion of an object or scene has been focused or exposed during the past image capturing, the control circuit 102 adjusts the focus or exposure to the specific portion during the current capturing. And exposure adjustment may be performed automatically.

FIG. 2C is an example of another assist control at the time of shooting that utilizes the operation history at the time of shooting. The imaging device 100 may be used together with equipment other than the imaging device 100, such as lighting equipment. For example, if the imaging device 100 and the lighting equipment are configured to be able to communicate with each other, and the imaging device 100 can know the positional relationship between the imaging device 100 and the lighting equipment through the communication between the imaging device 100 and the lighting equipment. For example, when photographing an object similar to an object photographed using illumination in the past, the control circuit 102, for example, as shown in FIG. It may be displayed instead of the view. Further, as shown in FIG. 2C, the control circuit 102 may display the positional relationship of the object O, the imaging device 100, and the lighting equipment I for capturing an image equivalent to the past image instead of the live view. good. FIG. 2C is, for example, a schematic diagram viewed from the sky. In this display, the control circuit 102 may display not only the position of the lighting device I but also the direction of the lighting device I and the like. Furthermore, the display of the positional relationship may be superimposed on the live view. In this case, an image indicating the position of the lighting device I is displayed at a specific position on the live view viewed from the imaging device 100 .

Also, the imaging device 100 may be used together with an accessory such as an external flash. If the number of times a particular accessory is used in photographing a particular object is high, the control circuit 102 may display a message in live view suggesting the use of that accessory in the next photographing.

Also, the assist control utilizing the operation history at the time of shooting as described above may be performed at the time of playback. For example, the area displayed in an enlarged manner during shooting, and the focused area may be displayed in an enlarged manner during playback.

FIG. 3A shows an example of a user's action that reproduces a still image. A user may check whether the captured still image is a desired one, or may edit the captured still image so as to obtain a desired image. In general, the user often has more time to check after shooting than during shooting. Therefore, the user can check the image in more detail than during shooting. Therefore, an object of interest to the user can also be extracted by analyzing the operation history of confirmation work after photographing.

The operations of FIG. 3A can be performed at any timing desired by the user. For example, the operation shown in FIG. 3A may be performed immediately after shooting, or may be performed several hours or days after shooting. Note that the operation of the confirmation work after photographing shown in FIG. 3A is an example. Some of the operations shown in FIG. 3A may be omitted, and operations other than those shown in FIG. 3A may be performed. In particular, immediately after photographing, the user confirms whether the target object or the combination of objects is as intended, and decides whether or not to perform the next photographing based on the confirmation result, or decides whether or not to perform the next photographing based on the confirmation result. I often decide whether to reflect it in the way I shoot. Such determinations require prompt confirmation. In other words, such confirmation is a minimum necessary action, but it reflects the user's intention and contains a lot of important information for objectively estimating the intention. Therefore, the user's operation at the time of such confirmation can be effective reference information when the user infers or identifies an object or scene from which the expression is particular. Assuming that the image at the time of such confirmation is the first image, if the object at the time of photographing the second image, which is a similar image, is similar to the first object or scene in the first image, , is useful for assisting shooting that reflects the user's preferences. The meaning of the similar images is assumed to be that the objects to be photographed are the same, or that the scenes or situations are the same when the images are consecutively photographed. If the control for assisting shooting is performed according to the history of previous operations, it is possible to quickly and accurately perform shooting that is particular to the user. At this time, whether or not the user is the same may be determined by manual switch operation or the like. In addition, when photographing and checking are continuously performed with the same photographing device, it is natural that the action is determined to be the action of the same user. Therefore, when photographing and confirmation are continuously performed with the same photographing device, it may be determined that the actions are performed by the same user. In other words, in order for the user to check the captured image immediately after shooting, the user acquires an operation (including trimming, enlarging, and confirmation display) for the first image reproduced on the display device, and According to the history of confirmation operations, specific elements in the first image (what the object is, its shape, posture, facial expression, color, focus and exposure) are specified, and when the subsequent second image is captured (confirmation As a result, when you find an improvement point and try to take an even better image, or when you simply shoot in various modes and take additional shots for later comparison) is similar to the first photographing situation, and control is performed to assist photographing of the second image according to the operation history when photographing the second image. The circuit should work. As will be described later with reference to FIG. 17, such confirmation operations are useful when inferring the user's preferences. In other words, when such inference is performed by machine learning, the relationship between each image and confirmation operation becomes training data. For each specific object or scene, it becomes possible to infer the relationship of operations when it is confirmed.

For example, the user sets the operation mode of the imaging device 100 to the reproduction mode to reproduce the desired image i. Then, the user confirms the interesting object O1 in the image. For example, when the object is too small to be seen or when the user wants to see the object in more detail, the user may perform an operation to enlarge and display the object of interest. The control circuit 102 causes the memory 108 to store the information of the enlarging operation in the reproduction mode as an operation history. The enlargement operation information includes the position and magnification of the enlarged area. Further, for example, when confirming an image of a bird, the user confirms the bird's eye region as the target object O1 and the tail region as the target object O2. It may be expanded. In this case, the control circuit 102 may cause the memory 108 to store the order of enlargement operations as a history of enlargement operations or reduction operations. This is because when an object is magnified in two or more parts, the order of confirmation by the user may also include the user's intention, for example, the order in which the object is of interest.

The user may also align the images so that the desired object is at the desired location. Also, as an editing task, the user may crop a desired portion of the image. In addition, as editing work, the user may change the brightness of the image, change the color tone, or add special effects such as blurring. When these editing operations are performed, the control circuit 102 stores information indicating the details of the editing operations and information on the edited area in the memory 108, for example, as an operation history.

The user may also publish the image to a particular site, such as in playback mode. When such a disclosure operation is performed, the control circuit 102 stores information indicating that the disclosure operation has been performed in, for example, the memory 108 as an operation history.

After confirming the image, the control circuit 102 records the operation history at the time of confirming a series of images in, for example, the recording medium 110 in association with the reproduced still image.

After recording the operation history, the control circuit 102 analyzes the operation history. For example, the control circuit 102 extracts an object included in an area in which an operation such as enlargement has been performed within the image. Then, for example, the control circuit 102 classifies the regions subjected to operations such as enlargement and trimming for each object. If the number of times a specific position of a certain object is enlarged during live view is large, the user will be interested in the specific position even when the image of a similar object is reproduced next time. It can be determined that there is a high possibility that Based on such determination, the control circuit 102 can perform control such as enlarging the specific position in advance when capturing or reproducing an image of a similar object next time.

FIG. 3B is an example of assist control during shooting using the operation history during playback. For example, assume that the user has performed an operation to enlarge and display two parts in the image when reproducing an image of an object in the past. At this time, as described above, the memory 108 stores, as an operation history, information indicating that an enlarged display operation has been performed and information on the enlarged target part.

The control circuit 102 determines whether or not the live view image includes an object similar to the past object. When it is determined that the live view image includes an object similar to the past object, the control circuit 102 displays the past enlarged object in the live view as shown in FIG. 3B. The guides G1 and G2 representing the parts of are displayed. The display positions of the guides G1 and G2 follow the object. That is, the control circuit 102 changes the display positions of the guides G1 and G2 according to the movement of the object. Here, when the order of enlarging operations is also stored, the guide G1 and the guide G2 may have different shapes of frames, different colors, or change the display order according to the order of the enlarging operations. . By looking at the guides G1 and G2, the user can remember that he/she was concerned about the guides G1 and G2 in the past. As a result, the user is expected to operate the guides G1 and G2 so as to obtain desired focus, exposure, and the like. Note that the guide does not necessarily have to be displayed on the display element 106 of the imaging device 1 . The guide may be displayed on a device other than the imaging device 1 having a display element, such as a smartphone.

The assist control shown in FIG. 3B is an example of displaying, at the time of shooting, the part that the user was concerned about at the time of playback. A display such as that of FIG. 3B can also be applied to other assist controls. For example, the display as shown in FIG. 3B can also be applied when a focus stacking operation is performed during shooting or editing. For example, the control circuit 102 may display a guide G1 for a location focused by focus stacking, and a guide G2 for a location out of focus by focus stacking. With such a display, the user is expected to perform an operation to focus on a specific portion or operate the aperture to blur a specific portion during photographing.

Also, the assist control shown in FIG. 3B may be performed during playback of an image including a similar object instead of when the similar object is photographed. In this case, the user can perform confirmation work while remembering the intention at the time of past confirmation. At this time, the control circuit 102 may not only display a guide frame, but also actually enlarge the relevant area or present the editing result of the relevant area to the user.

FIG. 4A shows an example of a user's action during video shooting. In the case of moving image shooting as well as still image shooting, the control circuit 102 stores the operation history in the memory 108, for example, every time the user performs an operation such as framing or zooming in the moving image mode. In the case of moving image shooting, unlike still image shooting, the user can perform an operation even during shooting. The control circuit 102 also causes the memory 108 to store an operation history of operations performed during shooting. After the video shooting is completed, the control circuit 102 records the operation history during the video shooting in, for example, the recording medium 110 in association with the video obtained as a result of shooting.

After recording the operation history, the control circuit 102 analyzes the operation history. The analysis of the operation history of moving images may be similar to the analysis of the operation history of still images. As described above, in the case of moving images, the operation history after the instruction to start shooting is also recorded. The control circuit 102 can also analyze the operation history after an instruction to start shooting is given.

FIG. 4B shows an example of a user's action during video playback. In the case of moving images, as in the case of still images, the user can confirm whether the captured moving image is what he or she desires, and edit the captured moving image so as to obtain the image desired by the user. be. For example, the user sets the operation mode of the imaging device 100 to the reproduction mode to reproduce a desired image. Then, the user confirms each frame of the moving image. In this confirmation, the user may magnify the object of interest in each frame. Also, in this confirmation, the user may select a piece of interest. The control circuit 102 stores these operations as an operation history in the memory 108, for example.

The user may also align the images so that the desired object is at the desired location. Also, as an editing task, the user may crop a desired portion of the image. When these editing operations are performed, the control circuit 102 stores information indicating the details of the operations in the memory 108, for example, as an operation history. The operation history includes selection operations and information for identifying the selected frame, such as the frame number. In FIG. 4B, for example, frame 2 and frame 3 are selected. Among them, the frame first selected by the user can be considered to be the frame that the user places the most importance on. Therefore, the control circuit 102 may set the frame selected first by the user as the priority frame, and store information for specifying the priority frame, such as the frame number, in the memory 108, for example. For example, in FIG. 4B, frame 3 is the priority frame.

Also, the user may edit the selected frame. When an editing operation is performed, the control circuit 102 stores information indicating the content of the operation and the result of editing in the memory 108, for example, as an operation history.

The user may also publish the image to a particular site, such as in playback mode. When a disclosure operation is performed, the control circuit 102 stores information indicating that the disclosure operation has been performed and information for specifying the disclosed image, for example, the file name of the image, in a memory 108, for example, as an operation history.

After confirming the images, the control circuit 102 records the operation history at the time of confirming a series of images in, for example, the recording medium 110 in association with the reproduced moving image.

After recording the operation history, the control circuit 102 analyzes the operation history. A user can be assisted by using the analysis result when shooting or editing a similar object. The assist control may be control to display an arrow mark to the user to indicate that an object has entered the angle of view, similar to the time of still image shooting, or to suggest to the user to shoot in a specific mode. The control may be such that a message to be displayed is displayed. Also, as assist control during video editing, control may be performed to present to the user a frame having characteristics similar to those previously selected by the user.

The operation of the imaging device 100 according to the first embodiment will be described in more detail below. FIG. 5 is a flow chart showing an example of the operation of the imaging device 100 according to the first embodiment. The processing in FIG. 5 is controlled by the control circuit 102, for example.

In step S1, the control circuit 102 determines whether or not the operation mode of the imaging device 100 is set to the shooting mode. The imaging device 100 has, for example, a shooting mode and a playback mode as operation modes. The shooting mode is a mode for shooting an image for recording. The playback mode is a mode for playing back recorded images. Of course, the imaging device 100 may have operation modes other than the shooting mode and the playback mode. The operation mode is set, for example, by the user's operation of the operation unit 112 . When it is determined in step S1 that the operation mode of the imaging device 100 is set to the shooting mode, the process proceeds to step S2. When it is determined in step S1 that the operation mode of the imaging device 100 is not set to the shooting mode, the process proceeds to step S17.

In step S<b>2 , the control circuit 102 performs processing for displaying the live view on the display element 106 . At this time, the control circuit 102 causes the memory 108 to store an image for live view acquired by imaging for live view by the imaging unit 104 . This live view image can be used as a framing operation history.

In step S3, the control circuit 102 determines whether or not to perform assist control. Whether or not to perform the assist control is determined, for example, based on the object shown in the live view. For example, when an object similar to the object associated with the past analysis result is included in the image for live view, it is determined to perform the assist control. Object similarity may be determined by image matching or the like. When it is determined in step S3 that the assist control should be performed, the process proceeds to step S4. When it is determined in step S3 that the assist control should not be performed, the process proceeds to step S5.

In step S4, the control circuit 102 performs assist control during photographing. Details of the assist control will be described later. After assist control, the process proceeds to step S5.

In step S5, the control circuit 102 determines whether or not the user has performed an operation before photographing. The operations before shooting are the above-described mode setting, framing, zooming, focusing, and exposure adjustment. When it is determined in step S5 that the user has performed an operation, the process proceeds to step S6. When it is determined in step S5 that no operation has been performed by the user, the process proceeds to step S8.

In step S6, the control circuit 102 executes processing according to the user's operation. For example, when a zooming instruction is issued, the control circuit 102 drives the zoom optical system. When a focusing instruction is issued, the control circuit 102 drives the focusing optical system.

In step S7, the control circuit 102 causes the memory 108, for example, to store an operation history corresponding to the user's operation.

In step S8, the control circuit 102 determines whether or not an operation to start shooting has been performed. The shooting start operation includes, for example, operation of the release button of the operation unit 112 and touch operation on the touch panel. When it is determined in step S8 that an operation to start photographing has been performed, the process proceeds to step S9. When it is determined in step S8 that the user has not operated to start shooting, the process proceeds to step S16.

In step S9, the control circuit 102 starts the imaging process. In the case of still image shooting, the control circuit 102 performs processing necessary for still image shooting. In the case of moving image shooting, the control circuit 102 performs processing necessary for moving image shooting.

In step S10, it is determined whether or not the user has performed an operation during shooting. This operation during shooting is mainly an operation during moving image shooting. In other words, in moving image shooting, the user can perform operations such as mode setting, framing, zooming, focusing, and exposure adjustment during shooting. In step S10, it is determined whether or not there is such an operation during shooting. When it is determined in step S10 that the user has performed an operation, the process proceeds to step S11. When it is determined in step S10 that the user has not performed an operation, the process proceeds to step S13.

In step S11, the control circuit 102 executes processing according to the user's operation.

In step S<b>12 , the control circuit 102 causes the memory 108 to store an operation history corresponding to the user's operation. After that, the process moves to step S13.

In step S13, the control circuit 102 determines whether or not to end the shooting. In the still image mode, it is determined that the photographing is finished when the processing of one still image photographing is completed. On the other hand, in the moving image mode, it is determined to end the shooting when the user performs an operation to end shooting. The shooting end operation includes, for example, an operation such as a second operation of the release button of the operation unit 112 or a touch operation on the touch panel. When it is determined in step S13 that the photographing is finished, the process proceeds to step S14. When it is determined in step S13 that the photographing is not finished, the process returns to step S9.

In step S14, the control circuit 102 records the captured still image or moving image in the recording medium 110 as an image file. Further, the control circuit 102 associates the operation history stored in the memory 108 in step S7 or S12 with the image file. The operation history may be associated by being recorded in the image file, or may be associated as a file separate from the image file.

In step S15, the control circuit 102 analyzes the recorded operation history. Details of the operation history analysis will be described later. After analysis, the process proceeds to step S16.

In step S<b>16 , the control circuit 102 determines whether or not to end the operation of the imaging device 100 . For example, when the imaging device 100 is turned off by the user and there is no user operation for a certain period of time, it is determined that the operation of the imaging device 100 is finished. When it is determined in step S16 that the operation of the imaging device 100 is not finished, the process returns to step S1. When it is determined in step S16 that the operation of the imaging device 100 is finished, the process of FIG. 5 is finished.

In step S17, the control circuit 102 determines whether or not the operation mode of the imaging device 100 is set to the reproduction mode. When it is determined in step S17 that the operation mode of the imaging device 100 is set to the reproduction mode, the process proceeds to step S18. When it is determined in step S17 that the operation mode of the imaging device 100 is not set to the reproduction mode, the process proceeds to step S16.

In step S<b>18 , the control circuit 102 causes the display element 106 to display a list of image files recorded on the recording medium 110 . The display of the list is performed, for example, by displaying thumbnails representing the contents of the image files. At step S19, the control circuit 102 determines whether or not the user has selected an image file. When it is determined in step S19 that an image file has been selected, the process proceeds to step S20. When it is determined in step S19 that no image file has been selected, the process proceeds to step S29.

In step S<b>20 , the control circuit 102 reads the image file selected by the user from the recording medium 110 and reproduces the read image file on the display element 106 .

In step S21, the control circuit 102 determines whether or not to perform assist control. Whether or not to perform assist control is determined, for example, based on the object in the image selected by the user. For example, when an object in the selected image is similar to an object extracted as a result of past analysis, it is determined that assist control should be performed. When it is determined in step S21 that the assist control should be performed, the process proceeds to step S22. When it is determined in step S21 that the assist control should not be performed, the process proceeds to step S23.

In step S22, the control circuit 102 performs assist control during reproduction. Details of the assist control will be described later. After assist control, the process proceeds to step S23.

In step S23, the control circuit 102 determines whether or not the user has performed an operation for reproduction. The operations during this playback are operations such as editing such as enlargement and trimming of the object described above. When it is determined in step S23 that the user has performed an operation, the process proceeds to step S24. When it is determined in step S23 that no user operation has been performed, the process proceeds to step S26.

At step S24, the control circuit 102 executes a process according to the user's operation. For example, when an enlarged display is instructed, the control circuit 102 causes the display element 106 to display an enlarged area specified by the user.

In step S25, the control circuit 102 causes the memory 108, for example, to store an operation history corresponding to the user's operation. After that, the process moves to step S26.

At step S26, the control circuit 102 determines whether or not to end the reproduction. For example, it is determined to end the reproduction when the user performs an operation to end the reproduction. The operation for ending reproduction includes, for example, an operation of a specific button on the operation unit 112 and a touch operation on the touch panel. When it is determined in step S26 that the reproduction should be terminated, the process proceeds to step S27. When it is determined in step S26 not to end the reproduction, the process returns to step S20.

In step S27, control circuit 102 associates the operation history stored in memory 108 in step S25 with the image file. After that, the process moves to step S28. The operation history may be associated by being recorded in the image file, or may be associated as a file separate from the image file.

In step S28, control circuit 102 analyzes the recorded operation history. Details of the operation history analysis will be described later. After analysis, the process moves to step S29.

In step S29, the control circuit 102 determines whether or not to end the reproduction mode. For example, it is determined that the reproduction mode should be ended when the user performs an operation to change the operation mode. When it is determined in step S29 that the reproduction mode should be terminated, the process proceeds to step S16. When it is determined in step S29 not to end the reproduction mode, the process returns to step S18.

FIG. 6 is a diagram showing an example of an image file containing an operation history. As shown in FIG. 6, an example image file includes an image, shooting information, an operation history during shooting, and an operation history during playback. As described above, the operation history during shooting and the operation history during playback may be recorded in a file separate from the image file. In this case, the operation history file includes information for identifying the corresponding image file. Alternatively, the image file includes information for identifying the corresponding operation history file. Further, the operation history at the time of shooting and the operation history at the time of playback do not necessarily have to be recorded in the same file.

An image is an image obtained by photographing. This image is recorded, for example, in a compressed state. An image during still image shooting can be compressed using a still image compression method such as the JPEG method. Images during moving image shooting can be compressed using a moving image compression method such as the MPEG method.

The shooting information is information when the image file was shot. The shooting information includes information such as an image file name, shooting model name, shooting date and time, and exposure conditions. Exposure conditions include aperture value and shutter speed at the time of shooting.

The operation history during shooting includes, for example, the object being photographed, the operation history of the live view display mode, the operation history of the enlarged display, the operation history of zooming, the operation history of focusing, the operation history of exposure adjustment, and external devices/accessories. including the usage history of During movie shooting, the operation history of each frame of the movie, that is, the operation history of the movie frame, the operation history of the enlarged display, the operation history of zooming, the operation history of focusing, the operation history of exposure adjustment, and the use history of external devices/accessories may contain

The photographed object includes information for specifying the object appearing in the photographed image, such as the name of the object. In addition, the object to be photographed may include information representing structural features on the image of the object in the photographed image, such as edge shape, brightness, and color.

The mode operation history includes information on mode names set by the user during shooting. The mode operation history is not limited to the information of the name of the mode set at the start of shooting. The mode operation history may include information on the name of the mode set during live view display before shooting.

The enlarged display operation history includes the position and enlargement ratio of the area in the image that was enlarged by the user during live view and video shooting. The magnified display operation history may include information representing the structural features of the object in the magnified display area.

The zooming operation history includes an optical zoom operation history and an electronic zoom operation history. The optical zoom operation history includes, for example, the focal length of the zoom optical system associated with the user's optical zoom operation. The electronic zoom operation history includes, for example, the position of the area within the electronically zoomed image and the magnification of the zoom, which accompanies the user's electronic zoom operation. When the object can be identified, the zooming operation history may include information on the object that has been zoomed.

The focusing operation history includes information indicating whether or not focusing was performed automatically, the position of the focusing optical system, and the position of the area in the image that was the focus. When the target can be specified, the focusing operation history may include information on the target for focusing.

The operation history of exposure adjustment includes information indicating whether or not exposure adjustment was performed automatically, and the position of the area within the image that was the target of exposure adjustment. When the target can be specified, the exposure adjustment operation history may include information on the target for exposure adjustment.

The history of use of external devices/accessories includes information indicating whether or not an external device or accessory was used during shooting, and information for identifying the external device or accessory if used. The information for specifying the external device or accessory may be the name, ID, or the like of the external device or accessory. In addition, the usage history of the external device/accessory may include settings of the external device or accessory at the time of use. For example, if the external device is a flash, the settings may include the orientation and strength of the flash, and so on.

The operation history during playback includes, for example, an enlarged display operation history, an editing operation history, and a public operation history.

The enlarged display operation history includes the positions of areas within the image that were enlarged by the user during image playback. The magnified display operation history may include information representing the structural features of the object in the magnified display area.

The editing operation history includes information indicating the content of the editing operation. Editing includes trimming, brightness adjustment, color adjustment, gamma adjustment, and the like. For editing such as trimming, which is performed on a specific region within an image, the editing operation history includes the position of the region within the image that is the target of editing. Furthermore, in the case of moving images, the editing operation history may include information for specifying frames selected by the user.

The disclosure operation history includes information on whether or not there has been a disclosure operation.

Further, in the case of a moving image file, the operation history at the time of reproduction may include information for specifying important frames. The information for specifying the important frame is, for example, the frame number.

Next, analysis processing will be described. FIG. 7 is a flowchart showing an example of analysis processing that is performed after still image or moving image shooting in step S15 of FIG. In step S101, the control circuit 102 reads an image file recorded in the recording medium 110 by photographing, and extracts an object to be photographed from the image recorded in the image file. As described above, object extraction can be performed by techniques such as edge extraction and image matching. When a plurality of objects are included in the photographed image, the control circuit 102 selects, for example, the object that exists closest to the center of the image, the object that exists at the shortest distance, and the live image. An object that has been in the view for a long time, that is, an object that has been captured in the angle of view for the longest time by framing by the user is extracted.

In step S102, the control circuit 102 refers to a plurality of image files and classifies the mode set at the time of shooting for each object. For example, the control circuit 102 calculates the number of mode settings for each object.

In step S103, the control circuit 102 refers to a plurality of image files and classifies the enlarged display, focused, and exposed areas for each object. For example, the control circuit 102 calculates the number of times the eye region of each object has been enlarged, focused, and exposed. Similarly, the control circuit 102 calculates the number of times an area having other characteristics of the object has been enlarged, focused, and exposed.

In step S104, the control circuit 102 refers to a plurality of image files and classifies how to use lighting for each object. For example, the control circuit 102 calculates the number of times illumination is used for each object. Also, the control circuit 102 calculates the position where the illumination is used when the illumination is used.

In step S<b>105 , the control circuit 102 associates the object with the classification result and records them in the recording medium 110 , for example. After that, the process of FIG. 7 ends. The classification result is recorded in a file different from the image file, for example.

FIG. 8 is a flowchart showing an example of analysis processing performed after the still image is reproduced in step S22 of FIG. In step S201, the control circuit 102 reads the reproduced image file and extracts the object from the enlarged display or edited area in the image recorded in the image file. The extraction of the target object can be performed by techniques such as edge extraction and image matching in the same manner as analysis processing at the time of photographing.

In step S202, the control circuit 102 refers to a plurality of image files, and classifies the enlarged display and edited area for each object. For example, the control circuit 102 calculates the number of times the eye region of each object has been enlarged and displayed. Similarly, the control circuit 102 displays the number of times that other feature areas of the object have been magnified. Also, for example, the control circuit 102 calculates areas of the object that have been trimmed a large number of times.

In step S<b>203 , the control circuit 102 associates the object with the classification result and records them in the recording medium 110 , for example. After that, the process of FIG. 8 ends. The classification result is recorded in a file different from the image file, for example.

FIG. 9 is a flowchart showing an example of analysis processing that is performed after the moving image is reproduced in step S22 of FIG. In step S301, the control circuit 102 reads the reproduced image file and determines whether or not any frame in the reproduced moving image file has been selected by the user. When it is determined in step S301 that a frame has been selected, the process proceeds to step S302. When it is determined in step S301 that no frame has been selected, the process proceeds to step S305.

In step S302, the control circuit 102 extracts an object from the enlarged display or edited area in the selected frame image.

In step S303, the control circuit 102 refers to a plurality of frames similar to the selected frame, and classifies the enlarged display and edited area for each object.

In step S304, the control circuit 102 associates the information of the selected frame, the object, and the classification result, and records them in the recording medium 110, for example. After that, the process of FIG. 9 ends. The classification result is recorded in a file different from the image file, for example.

In step S305, the control circuit 102 extracts the object from the enlarged display or edited area in the image of each frame.

In step S306, the control circuit 102 refers to a plurality of image files, and classifies the enlarged display and edited area for each object.

In step S307, the control circuit 102 associates the information of the selected frame, the object, and the classification result, and records them in the recording medium 110, for example. After that, the process of FIG. 9 ends. The classification result is recorded in a file different from the image file, for example.

FIG. 10 is a flow chart showing assist control at the time of photographing in step S4 of FIG. For example, as shown in FIG. 11, it is assumed that the user intends to photograph a bird, which is an object O, using the imaging device 100. FIG. At this time, it is assumed that the imaging device 100 stores an operation history and an analysis result when the user has photographed birds in the past.

In step S401, the control circuit 102 highlights an object within the live view. The highlighting is performed by displaying an arrow mark A near the object, as shown in FIG. 2B, for example. Of course, the highlighting is not limited to displaying arrow marks. For example, highlighting may be performed by displaying a frame surrounding the object. After highlighting, the process moves to step S402.

In step S402, the control circuit 102 determines whether or not the specific area of the object to be photographed has been enlarged and displayed many times. In step S402, the control circuit 102 compares the number of times of enlarged display with a predetermined value (for example, 10 times) for each region such as the region of the eyes and the region of the tail. As shown in FIG. 6, the operation history of enlarged display is recorded both during shooting and playback. Therefore, the enlarged display and analysis results can be separately obtained at the time of shooting and at the time of playback. The control circuit 102 individually compares the number of times the image has been displayed enlarged during shooting and the number of times the image has been displayed enlarged during reproduction with a predetermined value. Then, when the number of times of enlarged display during either shooting or playback is equal to or greater than a predetermined value, it is determined that the number of times of enlarged display is large. When it is determined in step S402 that the number of times the specific area is enlarged and displayed is large, the process proceeds to step S403. When it is determined in step S402 that the number of times the specific area is enlarged and displayed is not many, the process proceeds to step S404.

In step S403, the control circuit 102 performs assist control for enlarged display. After that, the process moves to step S404. FIG. 12 is a diagram showing an example of assist control for enlarged display. For example, assume that the eye area and the tail area of the bird O have been enlarged and displayed many times in the past when the bird O was photographed or confirmed. At this time, as shown in FIG. 12, the control circuit 102 generates a guide G1 indicating that the eye area of the bird O has been enlarged and a guide G2 indicating that the tail area of the bird O has been enlarged. display. The display positions of the guides G1 and G2 follow the movement of the bird O. That is, when the position or orientation of the bird O within the live view L changes, the control circuit 102 changes the display positions of the guides G1 and G2 according to the change in the position or orientation. The shape of the guide G1 and the shape of the guide G2 may be the same shape or different shapes. As described above, if there is a clear order in the enlargement display operations by the user in the past, the shape of the guide G1 and the shape of the guide G2 may be changed according to the order.

Assist control for enlarged display is not limited to guide display. For example, the assist control for enlarged display may include processing for displaying a message prompting the user to operate enlarged display of a specific area of the object displayed in the current live view. In addition, the assist control for enlarged display may include a process of enlarging and displaying a specific area without user's operation.

In step S404, the control circuit 102 determines whether or not the number of times the object to be photographed has been photographed in a specific mode is high. When it is determined in step S404 that the number of shots taken in the specific mode is large, the process proceeds to step S405. When it is determined in step S404 that the number of shots taken in the specific mode is not many, the process proceeds to step S406.

In step S405, the control circuit 102 performs assist control for mode setting. After that, the process moves to step S406. For example, the control circuit 102 displays a message prompting the user to shoot in a specific mode as shown in FIG. 2B as assist control for mode setting. Also, the control circuit 102 may set the mode without waiting for the user's operation. Alternatively, the control circuit 102 may set the mode without waiting for the user's operation, and automatically photograph the object. The image obtained by this automatic shooting may be presented to the user after the user's operation to start shooting.

In step S406, it is determined whether or not a specific area of the object to be photographed is frequently targeted for focusing or exposure. When it is determined in step S406 that the number of times the specific area has been targeted for focusing or exposure adjustment is high, the process proceeds to step S407. When it is determined in step S406 that the number of times the specific area is the target of focusing or exposure is not large, the process proceeds to step S408.

In step S407, the control circuit 102 performs assist control for focusing and exposure. After that, the process moves to step S408. For example, the control circuit 102 displays a message prompting the user to focus or expose a specific region of the object, for example, as assist control for focusing and exposure. At this time, the control circuit 102 may also display a guide such as an arrow so that a specific area of the object can be visually recognized. Also, the control circuit 102 may adjust the focus or the exposure without waiting for the user's operation. Further, the control circuit 102 may automatically photograph an object by performing focusing and exposure without waiting for user's operation. The image obtained by this automatic shooting may be presented to the user after the user's operation to start shooting.

In step S408, the control circuit 102 determines whether or not the number of post-shooting edits of an image containing an object similar to the object to be shot is large. When it is determined in step S408 that the number of edits is large, the process proceeds to step S409. When it is determined in step S408 that the number of times of editing is not large, the processing of FIG. 10 ends.

In step S409, the control circuit determines whether or not an effect similar to editing can be reproduced by shooting. For example, if the content of editing is brightness adjustment, color adjustment, or addition of special effects, the same effect as editing can be reproduced by exposure adjustment and image processing at the time of shooting. can be done. In addition, if the content of editing is adjustment of the focus position, the same effect as editing can be reproduced by adjusting the focus at the time of shooting. For example, the control circuit 102 can make the determination in step S409 by storing in advance in the memory 108 the content of editing and information indicating whether or not an effect equivalent to the editing can be reproduced. When it is determined in step S409 that the same effect as editing can be reproduced by photographing, the process proceeds to step S410. When it is determined in step S409 that the same effect as editing cannot be reproduced by photographing, the processing in FIG. 10 ends.

In step S410, the control circuit 102 is set to perform control necessary to reproduce the same effect as editing by photographing. After that, the process of FIG. 10 ends. For example, if the exposure is often corrected during editing, the control circuit 102 adjusts the exposure by adding the amount of exposure correction made by the user. Further, when the desired exposure cannot be obtained only by adjusting the exposure, the control circuit 102 may display a message for prompting the user to use the lighting. Also, for example, if similar color adjustments are made during editing, the control circuit 102 adjusts the white balance and the like in accordance with the amount of color adjustment made by the user.

FIG. 13 is a flow chart showing assist control during reproduction in step S22 of FIG. For example, it is assumed that the user is trying to check a moving image of a bird captured using the imaging device 100 . At this time, the imaging device 100 stores the operation history and the analysis results of when the user has photographed birds in the past.

In step S<b>501 , the control circuit 102 determines whether or not a frame with a specific composition is selected frequently in a moving image including an object similar to the reproduced moving image. When it is determined in step S501 that the number of times that a frame with a specific composition is selected is large, the process proceeds to step S502. If it is determined in step S501 that the reproduced image is a still image, or if it is determined that the frame with the specific composition has not been selected many times, the process proceeds to step S503.

In step S502, the control circuit 102 performs assist control for frame selection. After that, the process moves to step S503. FIG. 14 is a diagram showing an example of assist control for frame selection. As assist control for frame selection, the control circuit 102 displays a frame S1 for emphasizing frames similar to the priority frame, as shown in FIG. 14, for example. In addition, as assist control for frame selection, the control circuit 102 displays a frame S2 for emphasizing frames that have been selected many times other than the important frames.

In step S503, the control circuit 102 determines whether or not the specific area of the object included in the reproduced image is enlarged and displayed many times. When it is determined in step S503 that the number of times the specific area is enlarged and displayed is large, the process proceeds to step S504. When it is determined in step S503 that the number of times the specific area is enlarged and displayed is not many, the process proceeds to step S505.

In step S504, the control circuit 102 performs assist control for enlarged display. After that, the process moves to step S505. The assist control for enlarged display during playback may be the same as the assist control during shooting.

In step S<b>505 , the control circuit 102 determines whether or not the number of times a specific edit has been performed on images of similar objects is high. For example, when it is determined in step S505 that the number of times that a similar region is trimmed is large in an image of a similar object, and it is determined that the number of times that a specific edit is made is large, the process proceeds to step S506. do. When it is determined in step S505 that the number of times that the specific editing has been performed is not many, the processing of FIG. 13 ends.

In step S506, the control circuit 102 performs assist control for editing. After that, the process of FIG. 13 ends. For example, the control circuit 102 displays a message prompting the user to perform trimming or the like in assist control for editing. At this time, the control circuit 102 may also display a guide such as a frame so that the region to be trimmed can be visually recognized. Also, the control circuit 102 may perform the editing process without waiting for the user's operation.

As described above, according to the present embodiment, the user's operation history at the time of shooting or playback is associated with the image that is the target of the operation, and the operation is classified for each target object of the image. At the time of capturing or playing back an image including the same object from the next time onward, the assist control of the user's operation is performed.

For example, during shooting or playback, the user often performs various operations to check the target object. An object that the user is interested in can be estimated from such confirmation operations. In addition, it is considered that similar operations are often performed on similar objects. Therefore, by classifying the operations for each object, assist control can be performed for the user's favorite operation for each object.

Also, in the assist control, for example, a guide that emphasizes the target object is displayed. Also, in the assist control, for example, a message is displayed to prompt the user to perform an operation. With these assist controls, the user can operate shooting or playback while remembering his/her own past operations or objects that he/she was concerned about in the past. In addition, if the processing according to the user's intention is automatically performed, the user's operation can be saved.

Further, in the assist control, when an effect similar to editing performed by the user in the past can be reproduced by shooting, the same effect as editing is reproduced by image processing or the like at the time of shooting. In this case as well, the user's trouble is saved. Conversely, when the same effect as editing performed by the user in the past can be reproduced by photographing, control can be performed to shorten the time until image recording by omitting image processing at the time of photographing.

Here, in the first embodiment, the imaging device 100 is described as recording an operation history, analyzing the operation history, and performing assist control. However, all of these processes do not necessarily have to be performed in the imaging device 100 . For example, the analysis of the operation history and the process of specifying the object for assist control may be performed by a server outside the imaging device 100, or may be performed using cloud computing. In other words, the processing of the first embodiment may be performed by a system having an imaging device, a server, and the like.

[Second embodiment]
Next, a second embodiment will be described. FIG. 15 is a diagram illustrating an example configuration of an imaging device according to the second embodiment. In FIG. 15, the same reference numerals as in FIG. 1 are assigned to the configurations corresponding to those in FIG. 1, and the description thereof is omitted.

An imaging device 100 according to the second embodiment has a recording medium 116 . The recording medium 116 has two inference engines, namely an inference engine 116a for still images and an inference engine 116b for moving images. In FIG. 15, a recording medium 116 for recording the inference engine is provided separately from the recording medium 110 for recording image files and the like. The recording medium 110 and the recording medium 116 may be composed of one recording medium.

The inference engine 116a is a trained model configured to infer an object from an input still image and determine assist control according to the inferred object. The inference engine 116a has, for example, a neural network. A neural network has an input layer, an intermediate layer, and an output layer. Depending on the configuration of the hidden layer, the configuration of the neural network can be changed.

FIG. 16A is a diagram illustrating deep learning concepts for creating an inference engine 116a. For example, teacher data T1 and T2 are input to a learning program. The teacher data T1 is, for example, data of a set of a still image T11 and an assist control T12 related to the still image T11. Also, the teacher data T2 is data of a set of a still image T21 and an assist control T22 related to the still image T21. This assist control is, for example, enlarged display of a specific area in a still image. Here, only two examples of teacher data are shown. The training data is selected from a certain population and is not limited to two.

The inference engine 116a, which is a learned model, outputs an assist control T12 when the input still image is similar to the still image T11, and outputs an assist control T12 when the input still image is similar to the still image T21. It is generated by learning in the learning program so as to output control T22. When a still image similar to the still image T11 is input as the input IN to the inference engine 116a generated in this way, the assist control T12 is output as the output OUT. Similarly, when a still image similar to still image T21 is input as input IN, assist control T22 is output as output OUT.

The inference engine 116b is a trained model configured to infer an object in each frame of the input moving image and determine assist control according to the inferred object. The inference engine 116b has, for example, a neural network.

FIG. 16B is a diagram illustrating deep learning concepts for creating the inference engine 116b. For example, teacher data T3 and T4 are input to a learning program. The teacher data T3 is data of a set of a moving image T31 and an assist control T32 related to the moving image T31. Also, the teacher data T4 is data of a set of a moving image T41 and an assist control T42 related to the moving image T41. For example, this assist control is, for example, control for selecting an important frame from moving image frames. Although only two pieces of teacher data are illustrated here, the number of teacher data is not limited to two.

The inference engine 116b, which is a trained model, outputs assist control T32 when the input moving image is similar to the moving image T31, and outputs assist control T42 when the input moving image is similar to the moving image T41. It is generated by learning in the learning program so as to do. When a still image similar to the moving image T31 is input as the input IN to the inference engine 116b generated in this way, the assist control T32 is output as the output OUT. Similarly, when a moving image similar to the moving image T41 is input as the input IN, the assist control T42 is output as the output OUT.

FIG. 17 is a flow chart showing an example of processing for creating an inference engine. The processing in FIG. 17 may be performed in the imaging device 100 . Also, the processing in FIG. 17 may be performed by a server or the like that is separate from the imaging device 100 . In the following description, it is assumed that the control circuit 102 of the imaging apparatus 100 performs the processing in FIG. 17 .

In step S<b>601 , the control circuit 102 processes an image captured or played back by the user and assist control associated with them into teacher data. This processing is processing for processing the image and the assist control into a form suitable for use in the teacher's program. Alternatively, this processing is processing using the relationship between each image and the confirmation operation as teacher data. Through such processing, an inference model is obtained that enables inference of the relationship of operations when an image including specific objects or scenes is confirmed for each specific object or scene.

In step S602, control circuit 102 generates an inference model from the learning program and teacher data.

In step S603, the control circuit 102 determines whether or not the reliability of the image and the assist control is equal to or higher than a predetermined value. In other words, when the relationship between the image that is input data and the assist control that is output is substantially equal to the relationship between the image that is input as teacher data and the assist control, it is determined that the reliability is equal to or greater than the predetermined value. When it is determined in step S603 that the reliability is equal to or higher than the predetermined value, the process proceeds to step S604. When it is determined in step S603 that the reliability is not equal to or greater than the predetermined value, the process proceeds to step S605. This is a process for confirming whether or not the relationship of the particular confirmation operation that the user is likely to perform has been correctly inferred for each image. Such processing allows the correct inference of the operational relationship for each particular object or scene when an image containing them is viewed.

In step S604, control circuit 102 records the learned model in recording medium 116 as an inference engine. After that, the process of FIG. 17 ends.

In step S605, the control circuit 102 resets the population for learning. After that, the process moves to step S601. In other words, if the reliability of the learning result is low, it means that the relationship between the image and the obsessive confirmation operation that the user who took the image is likely to perform is not correctly inferred. selection of training data may not be appropriate. Therefore, the setting of the population or the selection of teacher data must be redone. As a result, in each object or in each scene, a particular part that the user wants to confirm can be determined without user's operation, and the result of this determination can be used during shooting control. In a simple example, a longer confirmation playback time is always set for a sunset scene, and assistance can be provided so that the bird's posture and facial expression can always be enlarged and confirmed when photographing a wild bird. Since the portion to be enlarged and confirmed is an important portion, it may be reflected in parameter control and image processing control, such as focus, exposure, color reproduction, and the like, being heavily controlled.

As described above, according to the present embodiment, by classifying the relationship between the image and the assist control using artificial intelligence, the assist control can be performed more in line with the user's preference.

Here, it is not necessary for the imaging apparatus 100 to create the learned model. A trained model may be created, for example, by a specific server or by using cloud computing. A server or the like can handle a large amount of data. Therefore, when a learned model is created by a server or the like, the images and assist controls used as teacher data may not necessarily be the images captured or played back by the user of the imaging device 100 and the assist controls associated therewith. . For example, an image taken by a cameraman or the like and assist control may be used as training data. Images shot by an unspecified number of people and assist control may also be used. If an image taken by a cameraman or the like and assist control are used as training data, for example, assist control can be performed so that the user can take an image equivalent to that of the cameraman. In other words, it is possible to issue guides for confirming the same parts as the cameraman and control various shooting settings. In addition, if images shot by an unspecified number of people and assist control are used, assist control can be performed so that, for example, images in fashion at the moment can be shot.

Although the present invention has been described above based on the embodiments, the present invention is not limited to the above-described embodiments, and various modifications and applications are of course possible within the scope of the present invention.

In addition, in the explanation of each of the operation flowcharts described above, the operations are described using "first", "next", etc. for the sake of convenience, but it does not mean that it is essential to perform the operations in this order. do not have.

Further, each process according to the above-described embodiment can also be stored as a program that can be executed by the control circuit 102, which is a computer. In addition, it can be distributed by being stored in a storage medium of an external storage device such as a magnetic disk, an optical disk, or a semiconductor memory. The control circuit 102 reads the program stored in the storage medium of the external storage device, and the operation is controlled by the read program, thereby executing the above-described processing.

In the embodiments, parts described as "parts" (sections or units) may be configured by combining a dedicated circuit or a plurality of general-purpose circuits, and if necessary, according to pre-programmed software It may be configured by combining a microcomputer that operates, a processor such as a CPU, or a sequencer such as an FPGA. Also, it is possible to design such that part or all of the control is taken over by an external device, in which case a wired or wireless communication circuit intervenes. Communication may be performed by Bluetooth communication, Wi-Fi communication, telephone line, or the like, and may be performed by USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrated into an ASIC.

100 imaging device, 102 control circuit, 104 imaging unit, 104a optical system, 104b imaging device, 106 display device, 108 memory, 110 recording medium, 112 operation unit, 114 communication circuit, 116 recording medium.

Claims (9)

Acquiring a user's operation on the first image reproduced on the display element;
Identifying a first object in the first image according to the acquired history of the operation;
Determining whether the target at the time of capturing the second image is a second target similar to the first target,
performing control for assisting shooting of the second image according to the history of the operation when the object is the second object at the time of shooting the second image;
at least one control circuit configured to
and
the first image reproduced on the display element is an image captured by the user;
The control circuit extracts, as the first object, an object included in the region confirmed by the user in the first image, and displays the It is determined whether or not control corresponding to the editing operation performed on the reproduced first image can be performed, and when it is determined that it can be performed, the control is performed on the first image. An imaging device that performs the editing operations performed and the corresponding controls . 2. The imaging apparatus according to claim 1 , wherein the area confirmed by the user includes at least one of an area for which the user has performed an operation for enlarged display and an area for which the user has performed an editing operation. 3. The imaging apparatus according to claim 1, wherein the control circuit controls to display a guide for emphasizing the second object on the display element when the second image is captured. 4. The imaging apparatus according to claim 3, wherein said control circuit controls to display said guide for each part of said second object. 3. The imaging apparatus according to claim 1, wherein the control circuit performs control to enlarge and display the second object when the second image is captured. 3. The imaging apparatus according to claim 1, wherein the control circuit performs control so as to focus or expose the second object when the second image is captured. 7. The imaging apparatus according to any one of claims 1 to 6 , wherein the control circuit determines control for assisting shooting of the second image from the second image by artificial intelligence. 8. The imaging apparatus according to claim 7 , wherein said control circuit associates said first image with control for assisting photographing of said second image, and uses said control as teacher data for learning of said artificial intelligence. Acquiring a user's operation on a first image captured by a user and reproduced on a display device;
identifying a first object in the first image according to the acquired history of the operation;
Determining whether an object at the time of capturing the second image is a second object similar to the first object;
performing control for assisting the photographing of the second image according to the operation history when the object is the second object at the time of photographing the second image;
An object included in the region confirmed by the user in the first image is extracted as the first object, and the first object reproduced on the display element when the second image is captured. It is determined whether or not control corresponding to the editing operation performed on the image can be performed, and when it is determined that it can be performed, the editing performed on the first image performing an operation and corresponding control;
An imaging method comprising:

Images