JP2022175451A - Imaging apparatus, control method for the same, and program - Google Patents

Abstract

To provide an imaging apparatus capable of automatically capturing an image at a suitable timing predicted from the current image.SOLUTION: An imaging apparatus includes prediction means for generating a prediction image, in which an image at a second point in time future than a first point in time is predicted using an image at the first point in time output from imaging means, determination means for determining, with respect to the prediction image, a degree of suitability indicating the degree to which the prediction image is suitable as an image to be recorded, and control means for controlling the imaging means so as to capture an image when the time reaches the second point in time when the determined suitability is higher than a predetermined threshold.SELECTED DRAWING: Figure 4

Description

The present invention relates to an imaging apparatus, its control method, and program.

In recent years, the development of image prediction techniques using machine learning has been actively carried out. A future image of the subject can be predicted by inputting the image captured by the camera to an inference model that has learned a large amount of video data in which the action scene of the subject is captured as training data. In Patent Document 1, an image captured by an in-vehicle camera is input to a trained convolutional neural network, and when a vehicle equipped with a camera continues to run, a moving object may appear and collide with a dangerous area and its characteristics. A technique for estimating is disclosed.

JP 2017-162438 A

By the way, by applying technology for predicting the future image of a subject to the shooting assist function of cameras, smartphones, etc., it is expected that the shooting assist function with improved usability will be provided. For example, it is conceivable that user-friendliness will be improved if a photographing assist function of a camera or a smartphone automatically executes photographing at suitable photographing timing according to changes in the subject. Japanese Patent Application Laid-Open No. 2002-200000 uses a technique for predicting future images, but does not consider using the technique for the shooting assist function.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object thereof is to realize a technique capable of automatically capturing an image at a suitable timing predicted from the current image.

In order to solve this problem, for example, the imaging device of the present invention has the following configuration. i.e.
Prediction means for generating a predicted image by predicting an image at a second point in the future from the first point in time using an image at a first point in time output from an imaging means;
determination means for determining a degree of suitability of the predicted image as an image to be recorded;
and control means for controlling the imaging means to capture an image when the time reaches the second point in time when the determined suitability is higher than a predetermined threshold.

According to the present invention, it is possible to automatically shoot an image at a suitable timing predicted from the current image.

1 is a block diagram showing a configuration example of a camera as an example of an imaging device according to the present invention; FIG. FIG. 4 is a diagram showing a configuration example of an image prediction unit according to the present embodiment; A diagram showing a configuration example of a suitable determination unit according to the present embodiment. A diagram showing a data flow from LV image generation to automatic photographing in the present embodiment. A diagram showing the relationship between predicted images generated in the present embodiment and time. A diagram showing an example of a shooting reservation management table according to the present embodiment. Flowchart showing a series of operations of automatic photographing processing in the present embodiment A diagram showing an example of a display screen of notification information according to the present embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

In the following, as an example of an imaging device, an example using a camera capable of automatically capturing an image at a suitable timing using a machine learning model will be described. However, the present embodiment is applicable not only to cameras but also to any other device capable of automatically capturing images at suitable timings using machine learning models.

<Camera configuration>
An example of the functional configuration of a camera 100 as an example of an imaging device according to this embodiment will be described with reference to FIG. One or more of the functional blocks shown in FIG. 1 may be implemented by hardware such as an ASIC or programmable logic array (PLA), or implemented by a programmable processor such as a CPU or GPU executing software. may It may also be implemented by a combination of software and hardware. Therefore, in the following description, even when different functional blocks are described as main entities, the same hardware can be implemented as the main entities.

101 is a control unit. The control unit 101 includes processors such as one or more CPUs, for example, and controls the overall operation of the camera 100 by reading and executing programs stored in the ROM 102 or storage medium 108 . The control unit 101 may further include a GPU. The control unit 101 also includes a hardware timer (also referred to as a timer function) for making a shooting reservation. As will be described later, the control unit 101 determines whether the predicted image output from the image prediction unit 109 satisfies a predetermined condition based on the output from the suitable determination unit 110, and determines whether the predicted image will be captured at a future time. set up a reservation.

Reference numeral 102 denotes a ROM (Read Only Memory), which is composed of, for example, a non-volatile semiconductor memory. ROM 102 stores a control program executed by control unit 101 .

103 is a RAM (Random Access Memory), and the RAM 103 is composed of, for example, a volatile semiconductor memory. The RAM 103 is used as a work memory for the control unit 101 to execute programs, or as a temporary storage area for various data.

Reference numeral 104 denotes an imaging unit, and the imaging unit 104 includes, for example, an imaging lens, an aperture, a shutter, an imaging element such as a CMOS sensor, an A/D converter, and the like. The imaging unit 104 acquires image data by converting an analog signal input via the imaging lens into digital data. Image data is stored in the RAM 103 .

The imaging unit 104 generates a live view image (also referred to as an LV image) captured by the camera and still image data for recording. The LV image is YUV format image data that is sequentially output from the imaging unit 104 at a predetermined frame rate during a shooting operation and displayed on the display unit 106 in real time. The LV image is also input to the image prediction unit 109 . The still image data for recording is high-resolution image data generated by the user's shooting instruction, and is recorded in the storage medium 108 via the recording unit 107 . Since still image data for recording has high resolution, it is compressed into JPEG format, for example.

105 is an operation unit. The operation unit 105 receives user operations and notifies the control unit 101 of the input information. The operation unit 105 may be configured by at least one of a touch panel, buttons, switches, and a cross key, for example.

106 is a display unit. The display unit 106 is a processing unit that displays information regarding the state of the camera 100, live view images generated by the imaging unit 104, image data recorded in the storage medium 108, and the like to the user. The display unit 106 is configured by, for example, a liquid crystal panel or an LED.

107 is a recording unit. The recording unit 107 controls reading and writing of data with respect to the storage medium 108 . The recording unit 107 controls initialization of the storage medium 108 and data transfer between the storage medium 108 and the RAM 103 .

108 is a storage medium. The storage medium 108 is a device having a large-capacity storage area, and is composed of, for example, a memory card or flash memory. Storage medium 108 may store a control program executed by control unit 101 .

109 is an image prediction unit. The image prediction unit 109 configures an image prediction model. The image prediction unit 109 may be realized by a circuit or processor for executing the processing of the image prediction model, or may be realized by software for processing the image prediction model executed by the control unit 101. good. The image prediction unit 109 receives an image and generates a predicted image of the image at future time.

A specific configuration example of the image prediction unit 109 will be described with reference to FIG. As shown in FIG. 2A, the image prediction unit 109 includes a configuration in which a plurality of prediction units 202 are connected. When the image 201 is input, the image prediction unit 109 performs image prediction model processing and outputs a plurality of predicted images 203 . The prediction unit 202 generates and outputs a prediction image corresponding to the time next to the input image captured at a certain time. A predicted image is generated by using an image captured at a certain time (first time point) to predict how the subject and background will be captured at the next time (second time point). This is an image that will be Also, the prediction unit 202 to which the predicted image is input further generates a predicted image for the next time. In this way, the image prediction unit 109 generates a plurality of predicted images as many as the number of connected prediction units 202 .

FIG. 2B shows a specific configuration example of each prediction unit 202 . The prediction unit 202 is configured by, for example, a neural network, and includes a convolutional network 204 that extracts feature information of an input image, and a recurrent neural network 205 that stores past states and infers the next state. The convolutional network 204 may be similar to a network forming a CNN (Convolutional Neural Network) known as deep learning technology for image recognition. The convolutional network 204 is a neural network that performs convolutional processing for condensing image feature points using a filter and pooling processing for reducing the information amount of image data while maintaining important feature point information. The recursive neural network 205 is, as shown in FIG. 2B, a network having recursive connections for state storage in hidden layers (also called intermediate layers) within the recursive neural network 205 . The recursive neural network 205 can add features that accompany time-series changes by recursive connection, and generates future predicted images based on past images. For the recurrent neural network 205, for example, an LSTM (Long Short-Term Memory) known as a recurrent neural network suitable for long-term memory can be used. It should be noted that any other configuration may be used as long as it is capable of generating a predicted image at the next time based on an image captured at a certain time. In addition, the neural network of the prediction unit 202 uses learning data such that, for example, the image at the first time point of the moving image is the input image, and the image at the second time point, which is the timing of the next time point in the moving image, is the correct data. can be learned.

110 is a suitable determination unit. When an image is input, the suitability determination unit 110 determines the suitability of photographing and recording for the image. Here, the suitability information is information indicating the degree to which an input image is suitable as an image to be recorded (also referred to as a recorded image), that is, the degree to which the user likes the recorded image. The suitability determination unit 110 may be realized by a circuit or processor for executing processing for determining the suitability of shooting recording, or may be implemented by software for the suitability determination processing executed by the control unit 101. may be implemented. A configuration example of the suitable determination unit 110 will be described with reference to FIG. The suitable determining unit 110 may be configured using the CNN described above. The suitability determination unit 110 includes a convolution network 302 that extracts feature information of an input image 301, and a suitability determination network 303 that has undergone learning processing in advance using a large number of images with high suitability as learning data. include. The configuration of convolutional network 302 is similar to convolutional network 205 described above. The suitability determination network 303 is a network that configures a forward propagation type fully connected layer in the hidden layer, and outputs suitability information 304 in the output layer as a result of suitability classification based on feature information. For example, when photographing figure skating, data in which a high degree of suitability is assigned to an image in which the athlete's face is facing the front after reaching the top of the jump, data in which the athlete's face is facing the front at the beginning of the jump, and data in which the athlete's face is not facing the front. The image may contain data that gave it a lower preference rating. The learning data is not limited to the example of figure skating, and may include images given high or low suitability for various sports and various shooting scenes.

For example, based on the suitability information output from the suitability determining unit 110, the control unit 101 determines whether or not the predicted image is suitable as the recording image. Further, the control unit 101 sets a shooting reservation at the future time of the predicted image determined to be suitable as the recording image.

Reference numeral 111 denotes an internal bus that connects control signals exchanged between the processing units described above.

<Overview of automatic shooting processing>
Next, an outline of the processing (simply referred to as automatic imaging processing) from the generation of the LV image to the execution of automatic imaging according to the present embodiment will be described. As an example, it is assumed that the automatic photographing process according to the present embodiment is used as a photographing assist function. For example, consider a case where automatic shooting processing is applied to figure skating shooting. If a player about to jump is included in the LV video, it is possible to automatically shoot at the timing when the player's face is facing the front after reaching the top of the jump, that is, at a suitable timing for shooting. become.

An outline of the automatic photographing process will be described below with reference to FIG. The imaging unit 104 generates LV images at regular time intervals while the imaging unit 104 is operating, and transmits the generated LV images to the display unit 106 and the image prediction unit 109 . As an example in this embodiment, the LV image is generated as YUV data with a resolution of 800×600 at a speed of 20 frames per second (50 millisecond intervals). The display unit 106 displays the transmitted LV image in real time.

The image prediction unit 109 generates predicted images at a plurality of future times based on the input LV images, and stores the generated predicted images in the predicted image buffer 401 . A predicted image buffer 401 is a memory area that temporarily stores a predicted image configured as part of the RAM 103 .

Note that the image prediction unit 109 in this embodiment is configured with 20 prediction units 202 as an example. The image prediction unit 109 uses LV images input at intervals of 50 milliseconds to generate predicted images up to 1 second later (50 milliseconds×20). For example, the relationship between the predicted image generated by the image prediction unit 109 and time is as shown in FIG.

The image prediction unit 109 performs image prediction using the LV image f _t0 at time t0, and generates predicted images f _t1 to f _t20 corresponding to future times t1 to t20. Subsequently, at the next time t1 after 50 milliseconds, the image prediction unit 109 executes image prediction using the LV image f _t1 to generate predicted images f _t2 to f _t21 corresponding to future times t2 to t21. . Similarly, 20 predicted images are generated at subsequent times t2 and t3.

Next, the control unit 101 inputs the predicted image stored in the predicted image buffer 401 to the suitability determination unit 110 and acquires suitability information from the suitability determination unit 110 . Based on the suitability information acquired from the suitability determination unit 110, the control unit 101 determines whether or not the corresponding predicted image should be photographed and recorded. Set the shooting reservation according to the future time. Specifically, the control unit 101 executes a timer function so that the image capturing unit 104 performs shooting recording in accordance with the corresponding future time, and stores the predicted image of the shooting reservation target and the suitability information in the shooting reservation management table. to record. The photography reservation management table is stored in the storage medium 108, for example, and configured like the table shown in FIG. The ID shown in FIG. 6 is a number for identifying each photography reservation, and may correspond to the execution order of photography. The predicted image ID shown in FIG. 6 is information for identifying a predicted image to be scheduled for shooting, and is expressed, for example, so that the corresponding time information can be known. The time information may be relative time from the LV image or absolute time. The degree of suitability shown in FIG. 6 is information indicating the degree of suitability of the predicted image that is the target of the shooting reservation, and is indicated by percentage (%) in 100 steps, for example. The degree of suitability may be another numerical value such as a decimal value from 0 to 1 as long as it indicates the degree of suitability as a recording image. The example shown in FIG. 6 shows a state in which three shooting reservations are set.

Note that the series of processes from image prediction to shooting reservation described above are executed while ensuring real-time performance so as to be completed within the LV image generation cycle (for example, 50 milliseconds).

<A series of operations related to automatic shooting processing>
Next, with reference to FIG. 7, a series of operations related to automatic photographing processing will be described. Note that this process is realized by the control unit 101 expanding a program stored in the ROM 102 or the storage medium 108 into a working area of the RAM 103 and executing the program, unless otherwise specified. Further, this processing is executed in a state where the machine learning model (neural network) in the image prediction unit 109 and the suitable determination unit 110 has been trained. Further, this process is started when the operation mode of the camera 100 is changed to the shooting mode by the user's operation on the operation unit 105, for example. Here, the shooting mode is one of operation modes of the camera 100, and is an operation mode in which the user can shoot a still image at any time by pressing a shooting record button. For example, this does not apply to an operation mode in which a still image recorded in a memory card is reproduced and displayed on a liquid crystal monitor without operating the imaging function.

In S<b>701 , the imaging unit 104 generates one frame of LV image and outputs it to the image prediction unit 109 . In S<b>702 , the image prediction unit 109 generates predicted images corresponding to a plurality of future times based on the LV image output from the imaging unit 104 and stores the generated predicted images in the predicted image buffer 401 . When the image prediction unit 109 stores the generated image in the image buffer 401 , it notifies the control unit 101 . In S<b>703 , the control unit 101 extracts one predicted image from the predicted image buffer 401 , inputs it to the suitable determination unit 110 , and acquires the corresponding suitability information from the suitable determination unit 110 .

In step S<b>704 , the control unit 101 uses the suitability information acquired from the suitability determination unit 110 to determine whether to execute a shooting reservation. For example, when the degree of suitability is greater than a predetermined threshold, the control unit 101 determines that the shooting reservation for the corresponding predicted image should be executed. The predetermined threshold may be, for example, 70%. If the degree of suitability is greater than the predetermined threshold, the control unit 101 determines that the photographing reservation should be executed, and advances to step S705. to proceed to S711.

In step S<b>705 , the control unit 101 determines whether there is another imaging reservation at a time close to the future time determined to be reserved. For example, the control unit 101 refers to the shooting reservation management table in the storage medium 108 and determines whether another shooting reservation has already been set at a time close to the future time determined to be reserved in S704. . If the control unit 101 determines that a shooting reservation has been set, the process advances to step S709, and if it determines that a shooting reservation has not been set, the process advances to step S706. Here, the state in which another shooting reservation is already set at a close time means that the time interval between them is narrower than a predetermined time range, for example, shorter than the time required for shooting and recording processing per image. This means that both shooting reservations cannot be validated. In the present embodiment, as an example, it is assumed that the time required for shooting and recording processing for one image is 200 milliseconds. In this case, the judgment as to whether or not another photographing reservation is set at a near time is a judgment as to whether another photographing reservation is included within 200 milliseconds before or after. In this way, by judging duplication of photographing reservations, it is possible to avoid setting a photographing reservation that is actually difficult to execute.

In S706, the control unit 101 sets the imaging reservation determined to be reserved in S704. Specifically, the timer function is executed so that the imaging unit 104 performs shooting recording in accordance with the corresponding future time, and the predicted image for which the shooting reservation is made and information on the degree of suitability are stored in the storage medium 108 for shooting reservation management. Add to table. In addition, in this step, the control unit 101 notifies the user that the imaging unit 104 continues to capture the subject, so that the camera 100 is notified that a shooting reservation has been set (that automatic shooting is being performed). The information to be notified is displayed on the display unit 106 . For example, the screen indicating that automatic shooting is in progress may be the screen 801 shown in FIG. 801 is the entire screen, and 802 is notification information displayed by this step. The notification information 802 includes, for example, information indicating that a shooting reservation has been set (that automatic shooting is being performed), and information indicating the period of automatic shooting and the number of images that are automatically shot. The notification information 802 may be configured using information in the shooting reservation management table. The notification information 802 shown in FIG. 8 shows an example in which information is displayed in text, but not limited to text, it may be indicated by an icon, or the notification information may be notified by voice.

In S707, the control unit 101 determines whether all the predicted images stored in the predicted image buffer 401 in S702 have been extracted in S703. If the control unit 101 determines that all the predicted images stored in the predicted image buffer 401 have been taken out, the process proceeds to S708, and if it is determined that all the predicted images have not been taken out, the process returns to S703 again.

In S708, the control unit 101 determines whether the user has changed the operation mode of the camera 100 from the shooting mode to another mode. If the mode is changed to another mode, this flowchart ends. If the shooting mode continues, the process returns to S701.

In step S<b>709 , the control unit 101 performs processing when there is a shooting reservation at a near time. Specifically, the control unit 101 refers to the photographing reservation management table, and detects in S705 that the degree of suitability of the predicted image determined to be reserved in S704 indicates that the photographing has already been reserved in the near future. It is determined whether the degree of suitability is higher than the degree of suitability of the predicted image. If the control unit 101 determines that the new predicted image to be reserved has a higher degree of suitability than the set predicted image, the control unit 101 proceeds to S710. The processing of the photographing reservation determined to be necessary is interrupted, and the process advances to step S707.

In S710, the control unit 101 cancels the already set imaging reservation detected in S705, and newly sets the imaging reservation determined to be reserved in S704. Here, in order to cancel the photography reservation, the control unit 101 stops the corresponding timer function and deletes the corresponding reservation information from the photography reservation management table. The content of the shooting reservation setting processing is the same as the content of the shooting reservation setting processing described in step S706. In this way, when there are predictive images close to each other, by prioritizing the shooting reservation of the predictive image with a higher degree of preference, it is possible to obtain an image that is more beneficial to the user by automatic shooting.

In S711, the control unit 101 determines whether or not a shooting reservation has already been set at the same time as the future time when it was determined in S704 that the shooting reservation should not be executed. The control unit 101 refers to the shooting reservation management table of the storage medium 108, and if it determines that a shooting reservation has already been set at the same time as the future time determined in S704, proceeds to S712; to S707.

In S712, the control unit 101 cancels the already set shooting reservation detected in S711. The content of the photography reservation cancellation processing is the same as the content of the photography reservation cancellation processing described in S710. At this time, in this step, the control unit 101 updates the display of the notification information started in S706 as the shooting reservation is canceled. Alternatively, the display unit 106 may display information notifying that the shooting reservation has been cancelled. When there are no more shooting reservations (when all shooting reservations are canceled), the display of the notification information started in S706 may be erased. Through S711 and S712, the future time was determined to be suitable for shooting recording in the image prediction of the previous LV image, but if it is determined not to be suitable in the image prediction of the subsequent LV image, the existing You can cancel the shooting reservation of After that, the control unit 101 executes S707 and S708 as described above, and ends the series of operations of the automatic photographing process.

In the above-described embodiment, for example, the case where the camera 100 includes one suitable determining unit 110 has been described as an example. However, the number of suitable determining units 110 is not limited to this, and the camera 100 may be configured to be able to control a plurality of suitable determining units 110 . Although the scale of the system configuration is increased by constructing a plurality of suitability determination units 110, it is possible to execute suitability determination processing for a plurality of predicted images in parallel, thereby shortening the overall processing time. becomes. By using this method, the number of predicted images output from the image prediction unit 109 can be increased. This makes it possible to perform image prediction for a farther future time.

Further, in the above-described embodiment, the case where the image prediction unit 109 and the suitable determination unit 110 are configured by a neural network has been described as an example. When using the image prediction unit 109 or the suitable determination unit 110 having a neural network with a large amount of calculation, a mechanism for reducing the processing load may be further implemented. For example, after one shooting reservation is set by the control unit 101, the control unit 101 continues to set a new shooting reservation until shooting recording by the shooting reservation is executed or until the shooting reservation is cancelled. You may choose not to set it. Then, the control unit 101 may adjust only the time of the photography reservation that has already been set. For example, the control unit 101 does not execute the suitability determination processing by the suitability determination unit 110 on all the predicted images generated by the image prediction unit 109, but on the predicted image at a time close to the shooting reservation time. You only have to do it for Here, the time close to the shooting reservation time may be, for example, a time within 300 milliseconds before and after the shooting reservation time. In this way, by limiting the times for which photography reservations are made, it is possible to reduce the processing load of the suitable determination unit 110 and the power consumption of the camera 100 as a whole.

As described above, in the present embodiment, an image at a first time point output from the imaging unit 104 is used to generate a predicted image by predicting an image at a second time point in the future from the first time point. Then, the degree of suitability (the degree of suitability of the predicted image as an image to be recorded) is determined for the generated predicted image. was taken. This makes it possible to automatically shoot at a suitable timing predicted from the current image. In other words, it is possible to realize a system that predicts a future image suitable for photographing from a current LV image using an image prediction model and automatically photographs the image. Further, in the present embodiment, after setting a photographing reservation at a future time, if it is determined that photographing at that time is not suitable based on an LV image generated thereafter, the photographing reservation is canceled. By doing so, it is possible to adjust the future time once the shooting reservation is set to the correct shooting timing each time the LV image is generated, that is, as the set future time approaches the current time. .

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

DESCRIPTION OF SYMBOLS 101... Control part 104... Imaging part 105... Operation part 106... Display part 108... Storage medium 109... Image prediction part 110... Suitable determination part

Claims (16)

Prediction means for generating a predicted image by predicting an image at a second point in the future from the first point in time using an image at a first point in time output from an imaging means;
determination means for determining a degree of suitability of the predicted image as an image to be recorded;
and control means for controlling the imaging means so as to capture an image when the time reaches the second point in time when the determined suitability is higher than a predetermined threshold. Device. 2. The method according to claim 1, wherein said prediction means is configured by a machine learning model that generates said predicted image corresponding to a future point in time from said first point in time based on said image at said first point in time. Imaging device. 3. The imaging apparatus according to claim 1, wherein said determining means is configured by a machine learning model for determining said suitability of an input image. 4. The imaging apparatus according to any one of claims 1 to 3, wherein the image at the first time point is a live view image sequentially output from the imaging means. The control means is characterized in that, when the determined degree of suitability is higher than the predetermined threshold, the control means sets a photography reservation for capturing an image when the time reaches the second time point. Item 5. The imaging device according to any one of Items 1 to 4. 6. The imaging apparatus according to claim 5, wherein, in response to the setting of the photographing reservation, the control unit causes the display unit to display information notifying that the photographing reservation has been set. After setting a first photography reservation for capturing the image at the second time based on the image at the first time, the control means uses the image at the third time, which is later than the first time. 7. The method according to claim 5, wherein it is determined whether the degree of suitability of the generated predicted image is higher than the degree of suitability of the predicted image generated using the image at the first time point. The imaging device described. When the suitability of the predicted image generated using the image at the third time point is higher than the suitability of the predicted image generated using the image at the first time point, 8. The imaging apparatus according to claim 7, wherein the first reservation for photographing is cancelled, and a second reservation for photographing based on the image at the third time is set. After setting a first photography reservation for capturing the image at the second time based on the image at the first time, the control means uses the image at the third time, which is later than the first time. If the suitability of the generated predicted image at the second time point or within a predetermined time range from the second time point is equal to or less than the predetermined threshold value, the first photography reservation is canceled. The imaging device according to claim 5 or 6. In response to the cancellation of the first photography reservation, the control means causes the display means to display information notifying that the photography reservation has been cancelled, or notifies that the photography reservation has been set. 10. The imaging device according to claim 8, wherein the information is changed. 10. The imaging apparatus according to claim 9, wherein said predetermined time range is the time required for an image to be output from said imaging means and recorded on a storage medium. The prediction means outputs predicted images at a plurality of points in the future after the first point in time,
The control means captures an image at the time of the predicted image in which the degree of suitability among the degrees of suitability determined for the predicted images at the plurality of times is higher than the predetermined threshold. 12. The image pickup apparatus according to any one of claims 5 to 11, wherein the image pickup apparatus controls the In response to the setting of the shooting reservation, the determining means determines the suitability only for the predicted images at points within a predetermined time range from the point at which the shooting reservation is set, among the predicted images at the plurality of points in time. 13. The imaging device according to claim 12, wherein: 14. The imaging apparatus according to any one of claims 1 to 13, further comprising recording means for recording an image captured at the second time point in a storage medium. a prediction step of generating a predicted image obtained by predicting an image at a second point in the future from the first point in time using an image at a first point in time output from an imaging means;
a determining step of determining a degree of suitability of the predicted image as an image to be recorded;
and a control step of controlling the image capturing means to capture an image when the time reaches the second point in time when the determined suitability is higher than a predetermined threshold. How to control the device. A program for causing a computer to function as each means of the imaging apparatus according to any one of claims 1 to 14.

Images