JP7166837B2 - Teacher data creation device, teacher data creation method, teacher data creation program, learning device, and imaging device - Google Patents

Description

The present invention relates to a teacher data creation device, a teacher data creation method, a teacher data creation program, a learning device, and an imaging device for machine learning.

2. Description of the Related Art In recent years, portable devices (photographing devices) with photographing functions such as digital cameras have become widespread. Some of this type of photographing equipment have automated various settings at the time of photographing. For example, some digital cameras are equipped with an AF function that automates focusing and an automatic exposure (AE) function that automates exposure. In addition, photographing equipment having a so-called continuous shooting function, in which photographs are taken continuously, has become widespread.

By the way, a technique has been developed for obtaining a desired inference result by machine learning for a captured image acquired by such a photographing device. Machine learning obtains inference results about unknown matters by learning the characteristics, time-series information, spatial information, etc. of known input information and inferring based on the learning results. That is, in machine learning, first, a trained model is obtained from specific input information so that a determinable output result can be inferred.

A large amount of information with known relationships between inputs and outputs is used as learning data when generating a trained model so that inference results can be obtained with high reliability. For example, in deep learning, a network is designed using a large amount of training data so that an expected output can be obtained for a known input. A trained model obtained by such a process (hereinafter also referred to as an inference model) can be used independently of the network that performed the training.

For example, in Patent Literature 1, for the purpose of preventing deterioration of learning accuracy even when the number of learning data is small, a relationship between a set of first content and second content whose type is different from that of the first content. a generation unit that generates a new second learner by using a part of the first learner that has undergone deep learning of the sex, and the second learner generated by the generation unit includes a first content and the second A technology is disclosed that includes a learning unit that deep-learns a relationship between a set of content and a third content of a type different from the content.

Japanese Patent No. 6151404

Conventionally, no device has been developed that performs machine learning for predicting a desired timing based on an image of an object that moves in an unknown manner.

The present invention provides a teacher data creation device, a teacher data creation method, a teacher data creation program, a learning device, and an imaging device that can predict a desired timing from an image of a subject by machine learning. With the goal.

A training data creation device according to an aspect of the present invention includes a target object image determination unit that detects a specific state image, which is an image of a specific target in a specific state, from a series of images having time information based on shooting time; a time determination unit for determining a time difference between the photographing time of each image in the series of images and the photographing time of an image including the specific state image in the series of images; and a control unit that combines the data of the time difference obtained with respect to and sets it as teacher data.

A learning device according to an aspect of the present invention includes an inference model that infers the time at which a given object is in the specific state from an input image by machine learning using the teacher data created by the teacher data creation device. and an inference model generation unit that generates

An imaging device according to an aspect of the present invention includes an inference engine that realizes an inference model generated by the learning device, an imaging unit, and an image captured by the imaging unit to the inference engine. and a setting control unit for obtaining an inference result of the time required for the predetermined object to reach the specific state.

A teacher data creation method according to an aspect of the present invention includes a detection step of detecting a specific state image, which is an image of a specific object in a specific state, from a series of images having time information based on shooting time; determining the time difference between the photographing time of each image and the photographing time of the image including the specific state image among the series of images; and a generation step of generating training data by pairing the data with the training data.

A training data creation program according to one aspect of the present invention comprises a computer, a detection step of detecting a specific state image, which is an image of a specific object in a specific state, from a series of images having time information based on shooting time; determining a time difference between the photographing time of each image of the series of images and the photographing time of an image including the specific state image among the series of images; and a generation step of generating teacher data by pairing the data with the time difference data obtained.

Advantageous Effects of Invention According to the present invention, it is possible to predict desired timing from an image of a subject by machine learning.

1 is a block diagram showing a learning device and an imaging device according to a first embodiment of the present invention; FIG. FIG. 2 is an explanatory diagram for explaining a network 12a of inference engines 12; FIG. 4 is an explanatory diagram showing an example of capturing each image of the image group 34a; FIG. 4 is an explanatory diagram showing the relationship between each image in the image group 34a and the shooting time; 4 is a flowchart for explaining a method of creating teacher data by a population creating unit 31a; Explanatory diagram for explaining the operation of the first embodiment. Explanatory diagram for explaining the operation of the first embodiment. 4 is a flowchart showing the operation of the imaging device 20; 4 is a flowchart showing the operation of the external device 30; Explanatory diagram for explaining the operation of the first embodiment. 6 is a flow chart showing an operation flow adopted in the second embodiment of the present invention; FIG. 4 is an explanatory diagram showing an example of a continuous image group taken in from an external image DB 32 to a mother set creating unit 31a; Explanatory drawing for demonstrating the method of producing|generating the network 12a. FIG. 4 is an explanatory diagram showing a display example of an image displayed on the display screen of the display unit 15; 9 is a flow chart showing an operation flow employed in the third embodiment of the invention; 4 is a flowchart showing control of the control unit 11 of the imaging device 20; FIG. 4 is an explanatory diagram showing a display example of an image displayed on the display screen of the display unit 15; Explanatory drawing for demonstrating the 4th Embodiment of this invention. Explanatory drawing for demonstrating the 4th Embodiment of this invention. Explanatory drawing for demonstrating the 4th Embodiment of this invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a learning device and an imaging device according to the first embodiment of the present invention. In the present embodiment, machine learning that predicts the time until reaching a predetermined moment (hereinafter also referred to as a decisive moment) is realized using an image having time information as learning data. As a specific example, it is possible to build an inference model that predicts the moment when a bird takes off by machine learning, and use the inference model to display the prediction result of the moment when a bird takes off from a live view image.

The imaging device 20 in FIG. 1 records an image obtained by imaging a subject. As the imaging device 20, not only a digital camera or a video camera but also a camera built in a smart phone or a tablet terminal may be adopted. As will be described later, the imaging device 20 can use an inference model at the time of live view display. 30 to obtain the inference model.

The imaging device 20 includes a control section 11 and an imaging section 22 . The control unit 11 may be configured by a processor using a CPU (Central Processing Unit) or the like, and may operate according to a program stored in a memory (not shown) to control each unit, or may be a hardware electronic circuit. may implement some or all of the functions.

The imaging unit 22 has an imaging element 22a and an optical system 22b. The optical system 22b includes a lens and a diaphragm (not shown) for zooming and focusing. The optical system 22b includes a zoom (variable magnification) mechanism, focus and diaphragm mechanism (not shown) for driving these lenses.

The imaging device 22a is composed of a CCD, CMOS sensor, or the like, and an optical image of a subject is guided to an imaging surface of the imaging device 22a by an optical system 22b. The imaging device 22a photoelectrically converts the subject optical image to obtain a captured image (image capturing signal) of the subject.

The imaging control unit 11a of the control unit 11 can drive and control the zoom mechanism, focus mechanism, and aperture mechanism of the optical system 22b to adjust zoom, aperture, and focus. The imaging unit 22 performs imaging under the control of the imaging control unit 11 a and outputs an imaging signal of the captured image (moving image and still image) to the control unit 11 .

An operation unit 13 is provided in the imaging device 20 . The operation unit 13 includes a release button (not shown), function buttons, various switches such as shooting mode setting and parameter operation, dials, ring members, and the like, and outputs operation signals to the control unit 11 based on user operations. The control section 11 controls each section based on an operation signal from the operation section 13 .

The control unit 11 captures captured images (moving images and still images) from the imaging unit 22 . The image processing unit 11b of the control unit 11 performs predetermined signal processing such as color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing on the captured image.

The imaging device 20 is provided with a display section 15, and the control section 11 is provided with a display control section 11f. The display unit 15 has, for example, a display screen such as an LCD (liquid crystal display device), and this display screen is provided on the rear surface of the housing of the imaging device 20, for example. The display control unit 11f causes the display unit 15 to display the picked-up image signal-processed by the image processing unit 11b. In addition, the display control unit 11f can also cause the display unit 15 to display various menu displays, warning displays, and the like of the imaging device 20. FIG.

The imaging device 20 is provided with a communication section 14, and the control section 11 is provided with a communication control section 11e. The communication unit 14 is controlled by the communication control unit 11e so that it can transmit and receive information to and from the external device 30 . The communication unit 14 is capable of short-range wireless communication such as Bluetooth (registered trademark) and wireless LAN communication such as Wi-Fi (registered trademark). It should be noted that the communication unit 14 is not limited to Bluetooth and Wi-Fi, and can employ various communication methods. The communication control unit 11 e can receive information on the inference model from the external device 30 via the communication unit 14 .

The control unit 11 is provided with a recording control unit 11c. The recording control unit 11c can compress the picked-up image after the signal processing, and supply the compressed image to the recording unit 16 for recording. The recording unit 16 is composed of a predetermined recording medium, and can record information given from the control unit 11 and output the recorded information to the control unit 11 . For example, a card interface may be used as the recording unit 16. In this case, the recording unit 16 can record image data on a recording medium such as a memory card.

The recording section 16 has an image data recording area 16a, and the recording control section 11c records image data in the image data recording area 16a. The recording control unit 11c can also read and reproduce information recorded in the recording unit 16. FIG.

Note that the recording unit 16 has a setting data recording area 16b. Setting information of the inference model is recorded in the setting data recording area 16b.

In this embodiment, the imaging device 20 is provided with an inference engine 12 as an inference unit. The inference engine 12 has a network 12a. The network 12a is constructed using setting values recorded in the recording unit 16, and constitutes a network obtained by completing learning in machine learning, that is, an inference model.

The recording control unit 11c receives information for configuring an inference model from the learning unit 31, which is the external device 30, via the communication unit 14, and records the setting information in the setting data recording area 16b of the recording unit 16. You may be able to do so.

2 to 4 are explanatory diagrams for explaining the network 12a of the inference engine 12. FIG. In FIG. 2, a predetermined network N1 is provided with a large amount of data set 31G corresponding to inputs and outputs as teacher data. As a result, the network design of the network N1 is determined so that the output corresponding to the input can be obtained. In this embodiment, an image is used as an input and an estimated time to a decisive moment is obtained as an output together with reliability information (reliability). Information on the determined network design of the network N1 is recorded as setting information in the setting data recording area 16b.

“Deep learning” is a multi-layered process of “machine learning” using neural networks. A typical example is a "forward propagation neural network" that sends information from front to back and makes decisions. In the simplest case, it consists of an input layer consisting of N1 neurons, an intermediate layer consisting of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be discriminated. It suffices if there are three output layers. The neurons of the input layer and the intermediate layer, and the neurons of the intermediate layer and the output layer are respectively connected by a connection weight, and the intermediate layer and the output layer are added with a bias value, thereby facilitating the formation of logic gates. Three layers may be sufficient for simple discrimination, but if a large number of intermediate layers are used, it becomes possible to learn how to combine a plurality of feature quantities in the process of machine learning. In recent years, those with 9 to 152 layers have become practical due to the relationship between the time required for learning, judgment accuracy, and energy consumption.
Various known networks may be employed as the network N1 employed for machine learning. For example, R-CNN (Regions with CNN features) using CNN (Convolution Neural Network) or FCN (Fully Convolutional Networks) may be used. It involves a process called "convolution" that compresses image features, works with minimal processing, and is robust to pattern recognition. In addition, a "recurrent neural network" (fully-connected recurrent neural network), which can handle more complicated information and can handle information analysis whose meaning changes depending on the order and order, may be used in which information flows in both directions.
In order to realize these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays) may be used, but much of the neural network processing is matrix multiplication. For this reason, GPUs (Graphic Processing Units) and Tensor Processing Units (TPUs) that specialize in matrix calculations are sometimes used. In recent years, artificial intelligence (AI) dedicated hardware "neural network processing unit (NPU)" is designed to be integrated and embedded with other circuits such as CPU, and it may be part of the processing circuit. be.
In addition, the inference model may be acquired by employing not only deep learning but also various known machine learning techniques. For example, there are methods such as support vector machines and support vector regression. The learning here is to calculate the classifier weights, filter coefficients, and offsets, and there is also a method using logistic regression processing. When a machine judges something, a human needs to teach the machine how to judge something. A rule-based technique that applies rules acquired by humans through empirical rules and heuristics may be applied and used.

The external device 30 has a learning unit 31 that determines such network design and an external image database (DB) 32 that records a large amount of data for learning. The learning unit 31 has a communication unit 31 b and the external image DB 32 has a communication unit 33 . The communication units 31b and 33 can communicate with each other. Note that the communication unit 31 c of the learning unit 31 can also communicate with the communication unit 14 .

The learning unit 31 has a control unit 31g. The control unit 31g is configured by a processor using a CPU or the like, and operates according to a program stored in a memory (not shown) to control each unit. Alternatively, a part or all of the functions may be realized by an electronic circuit of hardware. Note that the entire learning unit 31 may be configured by a processor using a CPU, a GPU (Graphics Processing Unit), an FPGA, or the like, and may operate according to a program stored in a memory (not shown) to control learning. However, a part or all of the functions may be realized by an electronic circuit of hardware.

The external image DB 32 has an image classification recording section 34 . The image classification recording unit 34 is composed of a recording medium (not shown) such as a hard disk or a memory medium, and classifies and records a plurality of images according to the types of objects included in the images. In the example of FIG. 1, the image classification recording unit 34 records only the image group 34a of the object type A, but the number of types to be classified can be set appropriately.

In this embodiment, for example, an image group of birds is recorded as the image group 34a. 3 and 4 are for explaining each image of the image group 34a. FIG. 3 is an explanatory diagram showing an example of capturing each image of the image group 34a, and FIG. 4 is an explanatory diagram showing the relationship between each image of the image group 34a and the shooting time.

FIG. 3 shows how a bird 46 perched on a branch 45 of a tree is photographed by the camera 40 . A display screen 42 composed of an LCD or the like is provided on the back of the camera 40, and an image of a bird 46 is displayed as a live view. A bird 46 can be photographed by operating the shutter button 41 . In this embodiment, a series of images are captured until the bird 46 takes off from the branch 45 . For example, if the camera 40 has a continuous shooting function, the continuous shooting function is used to capture a series of images of the bird 46 from resting on the branch 45 to taking off from the branch 45 at predetermined time intervals. may be acquired as a series of images. Further, if the camera 40 has a moving image capturing function, a moving image obtained by capturing a series of moving images from the bird 46 resting on the branch 45 to flying away from the branch 45 may be obtained. . Further, by operating the shutter button 41 of the camera 40 at predetermined time intervals, a series of images of the bird 46 from resting on the branch 45 to flying off the branch 45 are captured at predetermined time intervals to obtain a discrete image. A still image may be acquired at

FIG. 4 shows six images P1 to P6 acquired by the camera 40 arranged in chronological order. Each image P1 to P6 is an image obtained by photographing the bird 46 in FIG. Each of the images P1 to P6 contains time information, and the numbers under each image in FIG. 4 indicate the times acquired with reference to the image P5. In the example of FIG. 4, images P1 to P6 were taken at −20 seconds (2 seconds before), −15 seconds, −10 seconds, −5 seconds, 0 seconds, and +5 seconds with image P5 as the time reference. be. Generally, the camera 40 has a built-in timer, and time information is added to each of the images P1 to P6 by the timer. Relative time information may be added to each of the images P1 to P6 based on the time. For example, in the example of FIG. 4, the image P1 was captured at 15:30:27, and relative acquisition times of the images P2 to P6 may be stored as time information based on this time.

For example, it is assumed that the image that the user most wants to shoot is the image P5 including the image of the moment when the bird 46 is about to take off from the branch 45 (hereinafter also referred to as the specific state image). This moment is defined as a decisive moment, and the image P5 is called a decisive moment image. In this embodiment, a series of images having time information as shown in FIG. 4 are recorded in the image classification recording unit 34 as an image group 34a. The image classification recording unit 34 may record images of birds of the same type, or images of birds classified as having approximately the same size. In addition to these images, images of different types and image groups of birds classified into different sizes may be recorded.

As shown in FIG. 4, a bird may make a preparatory motion to straighten its bent legs and spread its wings at a predetermined time before taking off. For example, in the case of birds of different types, in the case of birds of different sizes, or depending on whether they are aiming at prey or not, the method of preparatory movement is considered to be somewhat different. By learning data, it is possible to predict the moment of flight from the state before flight.

In the image group 34a, for example, a huge data group of a series of images having time information as shown in FIG. 4 is recorded by photographing as shown in FIG. A mother set creation unit 31a of the learning unit 31 reads images from the external image DB 32 and creates a mother set that serves as a basis for learning.

It is also possible to acquire the learning data to be given to the learning unit 31 from the imaging device 20 . In this case, the imaging device 20 adds time information from a timer (not shown) incorporated in the control unit 11 to the captured image acquired by the imaging unit 22 and transmits the image to the learning unit 31 via the communication unit 14 . do.

In the present embodiment, the mother set creation unit 31a has a time determination unit 31a1 and an object image determination unit 31a2. The population generating unit 31a is controlled by the control unit 31g to generate teacher data used for generating an inference model. The target object image determination unit 31a2 determines an image portion of the target object (hereinafter referred to as target image) to be determined as the decisive moment, and determines the decisive moment when the target object image is in a specific state of the target object. It is determined whether or not the image (specific state image) when reaching . Further, the time determination unit 31a1 determines the time until each image reaches a decisive moment (specific state) based on the time information added to each image. That is, for each image in the series of images, the time determination unit 31a1 determines the time difference between the shooting time of each image and the shooting time of the image including the specific state image at the decisive moment in the series of images. The control unit 31g pairs the information of each image and the determined time difference as teacher data.
Here, the term "specific state" refers to a state in which the shape of an object being captured and tracked changes over time and assumes a specific posture or orientation, as well as a state in which the color or size of the object changes. It also includes changes, etc., and expresses changes in shape, position, orientation, etc. within the imaging screen. In addition, the state may include a temporal change in the sound emitted by the object. In addition, by comprehensively judging images and sounds, the user can set and designate performances such as performances and dances, some kind of work, the start and end of events, their climaxes and highlights. Character input, voice input, item selection, similar information input, etc. can be considered as the designation method. In addition, even if it is not specified one by one, a moment that many people perceive as a decisive moment may be determined automatically, and it may occur not only at one specific timing but also at a plurality of timings. The time lag may have a plurality of numerical values.

FIG. 5 is a flow chart for explaining a method of creating teacher data by the population creating unit 31a. At step S1 in FIG. 5, the mother set creation unit 31a acquires a series of images to which time information is added. For example, images P1-P6 in FIG. 4 are acquired. The object image determination unit 31a2 of the mother set creation unit 31a determines the object image from each image of the acquired series of images.

First, the object image determination unit 31a2 determines whether or not manual selection is instructed. Manual selection is performed by a user's operation of designating an object image from among the images. The learning unit 31 is provided with a display unit 31f configured by an LCD or the like, and the display unit 31f is provided with a touch panel (not shown) for accepting user operations. The object image determination unit 31a2 displays a series of images on the display unit 31f (step S9), and determines the object specified by the user's touch operation as the manually selected object image. If there is manual selection by the user, the series of images is used as a candidate for teacher data (step S10).

When the manual selection is not specified, the object image determining section 31a2 determines the object image in step S3. For example, the target object image determination unit 31a2 may determine an image portion of a subject located in the center of the image with a size equal to or larger than a predetermined size as the target object image. Further, an object to be a target object may be specified in advance by a user's operation. For example, when a bird is specified as the target object, the target image determining section 31a2 may determine the bird as the target object by a known recognition process for the captured image.

In the next step S4, the target object image determination unit 31a2 excludes images that do not include the target object image from the series of images, and determines whether or not the number of remaining images is equal to or greater than a predetermined number (step S5). . If the number of images including the object image in the series of images is less than a predetermined number, it may be difficult to determine the decisive moment and the time until the decisive moment. The image group is excluded from candidates for the training data group in step S11.

When the number of images including the target object image is equal to or greater than the predetermined number, the target object image determination unit 31a2 shifts the process to step S6, selects an image group including the target object image, and in step S7, Candidate images for selecting decisive moment images. The example of FIG. 4 shows an example in which the moment when a bird takes off is regarded as the decisive moment. An image is detected as a specific state image, and an image including the specific state image is set as a candidate for the decisive moment image. In the example of FIG. 4, the image P5 is a candidate for the decisive moment image.

In the next step S8, the object image determination unit 31a2 determines whether or not there are a predetermined number or more of images before the candidate for the decisive moment image. If the number of images acquired before the decisive moment image in the group of images including the object image is less than a predetermined number, the time to the decisive moment is too short to be used. Such image groups are excluded from candidates for the training data group in step S11.

When there are a predetermined number or more of images including an object image before the decisive moment image, the object image determination unit 31a2 shifts the process to step S12, and determines the candidate for the decisive moment image as the decisive moment image. A momentary image is determined, and the acquisition time of the decisive momentary image is standardized.

In the next step S13, the time determination unit 31a1 records the images including the target object image among the series of images with a relative time based on the acquisition time of the decisive moment image. In this way, as shown in FIG. 4, a series of images to which information on the relative time difference with respect to acquisition times of other images is attached, based on the image P5, which is the decisive moment image, is used as teacher data in the teacher data recording unit. 31e.

The input/output modeling unit 31d as an inference model generating unit reads out the teacher data created by the mother set creating unit 31a from the teacher data recording unit 31e, for example, by the method shown in FIG. A learning model (inference model) that has learned the relationship with the time until it is completed, that is, the network 12a and its setting information are obtained.

The learning unit 31 transmits the generated inference model to the imaging device 20 via the communication units 31 c and 14 when requested by the control unit 11 of the imaging device 20 . The control unit 11 records the setting information acquired via the communication unit 14 in the setting data recording area 16b and uses it for setting the network 12a of the inference engine 12. FIG. Thus, the inference model generated by the learning unit 31 can be used by the imaging device 20 .

A setting control unit 11d is provided in the control unit 11, and the setting control unit 11d can control the inference engine 12 to perform inference using the inference engine 12. FIG. That is, when the imaging unit 22 acquires a live view image, the setting control unit 11d provides the live view image to the inference engine 12 to obtain the time until the decisive moment (hereinafter referred to as image time inference). let it run. As a result, when information on the time until the decisive moment is obtained from the inference engine 12, the setting control unit 11d controls the display control unit 11f to display the result of inference by the inference engine 12 on the display unit 15. It can be displayed on the display screen. That is, in this case, the time until the decisive moment is displayed superimposed on the live view image.

Note that the setting control unit 11d may be capable of presenting the result of inference by the inference engine 12 to the user in various ways, not limited to display. For example, the setting control unit 11d may present the inference result by voice, or may present the inference result by mechanical control of the driving unit.

Next, the operation of the embodiment configured in this manner will be described with reference to FIGS. 6 to 10. FIG. 6, 7 and 10 are explanatory diagrams for explaining the operation of the first embodiment. 8 and 9 are flowcharts for explaining the operation of the first embodiment, FIG. 8 shows the operation of the imaging device 20, and FIG. 9 shows the operation of the external device 30. FIG.

FIG. 6 shows how an object is imaged by the imaging device 20 of FIG. Each part of the imaging device 20 in FIG. 1 is housed in a housing 20a in FIG. A display screen 15a constituting the display unit 15 is arranged on the rear surface of the housing 20a. A lens (not shown) forming an optical system 22b is arranged on the front surface of the housing 20a, and a shutter button 13a forming the operation unit 13 is arranged on the upper surface of the housing 20a.

FIG. 6 shows an example of photographing a bird 46 perched on a branch 45 of a tree as an object. While the bird 46 is caught in the field of view, the user presses the shutter button 13a with the finger of the right hand 48 to take a picture.

In this embodiment, an inference model is used to determine a decisive moment, which is a photo opportunity. That is, the inference engine 12 constructs an inference model for inferring the predicted time until the decisive moment arrives for an image (live-view image). Such an inference model can be generated by the external device 30 .

FIG. 9 shows the operation of the external device 30. As shown in FIG. In step S41 of FIG. 9, the external device 30 determines whether or not a learning request has been made, and is in a standby state until a learning request is made. When a learning request occurs, the external device 30 reads data for learning from, for example, the external image DB 32 and creates teacher data in step S42. Note that the teacher data creation step of step S42 may be performed according to the flow of FIG.

When the teacher data is created and recorded in the teacher data recording unit 31e, the input/output modeling unit 31d reads the teacher data from the teacher data recording unit 31e and performs learning to create an inference model in step S43. In the next step S44, the input/output modeling unit 31d sets exercises and verifies the created inference model. In step S45, the input/output modeling unit 31d determines whether or not the reliability of the inference is equal to or higher than a predetermined value as a result of verification using practice problems. If it is equal to or greater than the predetermined value, the input/output modeling unit 31d determines that the inference model is correctly generated, and transmits the inference model to the imaging device 20 via the communication unit 31c (step S49). .

If the reliability is not equal to or higher than the predetermined value, the input/output modeling unit 31d shifts the process from step S45 to step S46, resets the teacher data, etc., and then resets the reliability a predetermined number of times or more in step S47. It is determined whether or not the If resetting has not been performed the predetermined number of times or more, the input/output modeling unit 31d returns the process to step S43. When the resetting is performed more than the predetermined number of times, the input/output modeling unit 31d shifts the process from step S47 to step S48, and determines that the object image is a difficult image unsuitable for inference. After transmitting the weak image information to the imaging device 20, the process proceeds to step S49.

On the other hand, the control unit 11 of the imaging device 20 determines whether or not the imaging mode is designated in step S21 of FIG. If the photographing mode is specified, the control section 11 performs image input and display in step S22. That is, the imaging unit 22 captures an image of a subject, and the control unit 11 captures the captured image from the imaging unit 22 and provides the captured image as a live view image to the display unit 15 for display as shown in FIG.

Next, in step S23, the setting control unit 11d causes the inference engine 12 to execute image time inference for displaying the time until the decisive moment. The inference engine 12 uses an inference model realized by the network 12a to infer the time required for each live view image being captured to become a decisive moment image. The inference engine 12 outputs the result of inference to the control unit 11 . Note that the inference result includes information on the time and reliability until the live view image being displayed changes to the decisive moment image.

In step S24, the setting control unit 11d determines whether the inference engine 12 has an inference model related to the live view image. For example, if the reliability (reliability) of the inference result from the inference engine 12 is lower than a predetermined first threshold, the setting control unit 11d determines whether the inference engine 12 has an inference model related to the live view image. It may be determined that it is not. Also, the setting control unit 11d may recognize a subject in the live view image by a known recognition process, and determine whether or not there is an inference model related to the recognized subject.

If the setting control unit 11d does not have a related inference model, the process proceeds to step S29. If the setting control unit 11d determines that there is a related inference model, then in the next step S25, the setting control unit 11d displays a display indicating that there is an inference model related to the current live view image.

Next, the setting control unit 11d determines whether the reliability (reliability) of the inference result from the inference engine 12 is sufficiently high, for example, whether it is higher than a predetermined second threshold. If the reliability is equal to or higher than the second threshold, the setting control unit 11d shifts the process to step S27 to display a highly reliable time difference display, and if the reliability is smaller than the second threshold, , the process proceeds to step S28 to display the display of the time difference range with relatively high reliability.

FIG. 7 is an explanatory diagram showing a captured image displayed on the display screen 15a of the display section 15. As shown in FIG. As mentioned above, user 47 is attempting to photograph bird 46 on branch 45 . In particular, it is assumed that the user 47 wishes to photograph the moment when the bird 46 takes off from the branch 45 as the decisive moment. Images P11 to P14 in FIG. 7 show live view images at a predetermined time, and time passes in the order of images P11 to P14.

An image 46a in the image P11 shows a bird 46 perched on a branch. This image P11 is displayed as a live view image on the display screen 15a. An image P12, which is a live view image acquired after a predetermined time from the image P11, is displayed with a circle indication 51 indicating that there is an inference model related to the subject in the image P12. An image 46b of the bird 46 in the image P12 shows the bird 46 about to take off. Further, in the image P12, as a result of the image time inference by the inference engine 12, a time difference width display 52b is displayed indicating that the time to the decisive moment is from 5 seconds to 2 seconds. The time difference width display 52b displays the inference result with a predetermined width because the reliability of the inference result of the image time inference is not sufficiently high. , 65-84%).

On the other hand, an image 46c of a bird 46 immediately before taking off is displayed in an image P13, which is a live view image acquired after a predetermined time from the image P12. Further, in the image P13, as a result of the image time inference by the inference engine 12, a time difference display 52c indicating that the time to the decisive moment is one second is displayed. The time difference display 52c indicates one inference result with the highest reliability because the reliability of the inference result of the image time inference is sufficiently high (for example, 85% or more). According to the time difference display 52c in the image P13, it is highly likely that the bird 46, which is the subject, will take off one second after the start of the display of the time difference display 52c.

By pressing the shutter button 13a one second after the time difference display 52c is displayed, there is a high possibility that the decisive moment when the bird 46 takes off can be photographed. In step S29, the control unit 11 determines whether or not a moving image or still image shooting operation has been performed. If the photographing operation is not performed, the control unit 11 returns the process to step S21, and if the photographing operation is performed, the photographing and recording process is performed in step S30, and the process returns to step S21. That is, the control unit 11 causes the recording control unit 11c to record the captured image acquired by the imaging unit 22 in the image data recording area 16a of the recording unit 16. FIG. Note that, during moving image recording, a moving image file is recorded in the image data recording area 16a at the time of an operation to end shooting.

FIG. 10 is an explanatory diagram for explaining the captured image thus captured, and shows an example of the rec view image displayed on the display screen 15a. The left side of FIG. 10 shows a rec view image 55 displayed on the display screen 15a during continuous shooting. For example, the rec view image 55 can be obtained by starting continuous shooting after the user confirms the time difference width display 52b and the time difference display 52c. A thick frame indicates that one image taken continuously is the decisive moment image 55a.

The right side of FIG. 10 shows a rec view image 57 displayed on the display screen 15a during single shooting. For example, when the user confirms the time difference display 52c and presses the shutter button 13a after the displayed time has passed, the decisive moment image shown by the rec view image 57 is obtained.

If the control unit 11 determines in step S21 that the shooting mode has not been designated, the process proceeds to step S31 and determines whether acquisition of an inference model has been designated. If acquisition of an inference model is not specified, the control unit 11 returns the process to step S21, and if specified, sets the target object and the relearning object in step S32.

For example, the control unit 11 causes the display control unit 11f to display a menu for dictionary setting on the display screen 15a, and further displays a target object setting screen and a relearning object setting screen according to the user's operation. It is also possible to allow the user to specify an object and specify a relearning object. In step S<b>33 , the control unit 11 requests the external device 30 to learn or re-learn the target object or re-learning object designated by the user.

In step S<b>34 , the control unit 11 receives from the learning unit 31 the inference model whose reliability is equal to or higher than a predetermined value or the inference model corresponding to the weak image information via the communication unit 14 . The control unit 11 sets the received inference model in the inference engine 12 and records the weak image information in the recording unit 16 .

In the explanation of FIG. 8, an example was explained in which the time difference display or the time difference width display is switched depending on whether or not the reliability of the inference result of the image time inference is sufficiently high. can be considered in various ways. For example, the time difference of the inference result may be displayed by a numerical value indicating reliability or color coding, and the higher the reliability of the inference result, the greater the degree of gradation of the display. Also, the time difference display or the time difference width display may be always performed regardless of the reliability.

Also, in the above description, the shooting operation is described as being manually performed by the user, but it is also possible for the imaging control unit to control so that shooting is automatically performed at a decisive moment. Further, when continuous shooting is performed, it is possible to control the timing of continuous shooting to be synchronized with the decisive moment so that the shooting is surely performed at the decisive moment.

As described above, in the present embodiment, machine learning that performs image temporal inference for predicting the time until reaching a predetermined moment (decisive moment) is realized using an image having time information as learning data. By applying an inference model obtained by this machine learning to, for example, an imaging device, image time inference can be performed on live view images that change from moment to moment, and the arrival time to a decisive moment, for example, when a bird takes off, can be predicted. can be presented. The user can easily capture the decisive moment when the bird takes off by, for example, operating the shutter button in consideration of the presented arrival time. In addition, images with time information used as learning data can be obtained very easily, and teacher data can be obtained from this learning data by relatively simple processing, making image temporal inference possible. You can easily create an inference model that

(Second embodiment)
FIG. 11 is a flow chart showing the operation flow employed in the second embodiment of the invention. The hardware configuration of this embodiment is the same as that of the first embodiment. In FIG. 11, the same steps as in FIG. 9 are denoted by the same reference numerals, and descriptions thereof are omitted.

In the first embodiment, an image having time information is used as learning data, and training data is created using this learning data. An example of generating and using an inference model for inference was explained. On the other hand, in the present embodiment, an image having time information is used as learning data, and a set of a plurality of images having a predetermined time interval is created as teacher data using this learning data. This is an example of generating and using an inference model that performs image inference for predicting an image and an image after a predetermined time.

The flow in FIG. 11 differs from the flow in FIG. 5 in the method of creating teacher data. That is, the mother set creation unit 31a of the external device 30 acquires a continuous image group of similar objects from the external image DB 32 or the like in step S51 of FIG. In step S52, the mother set creation unit 31a gives the two images with a specific time difference as teacher data to the network for learning.

FIG. 12 is an explanatory diagram showing an example of a continuous image group taken in from the external image DB 32 to the mother set creation unit 31a. Images P21 to P29 are, as in FIG. 4, arranged in chronological order images obtained by photographing a series of situations before and after the bird takes off. Note that these images P21 to P29 may be acquired using a continuous shooting function or moving image function, or may be acquired by single shooting at predetermined time intervals.

The mother set creation unit 31a selects a set of two images before and after a predetermined time as teacher data by using the time determination unit 31a1 and the object image determination unit 31a2. For example, arrows in FIG. 12 indicate images after N seconds, and indicate that images P21 and P24, images P22 and P25, images P23 and P26, and images P24 and P27 form a set. The population generating unit 31a records the generated training data in the training data recording unit 31e.

FIG. 13 is an explanatory diagram for explaining a technique for generating the network 12a using the same description method as in FIG. In FIG. 13, a large amount of learning data set input to the predetermined network N1 is a set of images read out from the teacher data recording unit 31e. In this embodiment, the network design is determined so that when an image is given to the network N1 as an input, an image after N seconds is obtained as an output. Information on the network design thus determined is transmitted from the communication section 31c to the imaging device 20, and recorded as setting information in the setting data recording area 16b by the setting control section 11d.

Furthermore, in the present embodiment, when the learning unit 31 determines in step S45 that the reliability is equal to or higher than the predetermined value, in the next step S53, the output image estimated from the input image, that is, It is determined which image is to be output from among the images acquired after the lapse of a predetermined time from the acquisition time of the input image. This image is recorded on a recording medium (not shown) so as to be used as a representative image for composite display, as will be described later. The learning unit 31 also transmits the representative image to the imaging device 20 when transmitting the network design information. The recording control section 11c of the imaging device 20 records the representative image in the image data recording area 16a.

Next, the operation of the imaging device 20 of the embodiment configured as described above will be described with reference to FIG. FIG. 14 is an explanatory diagram showing a display example of an image displayed on the display screen of the display unit 15. As shown in FIG.

In the present embodiment, the inference engine 12 configures an inference model that performs the above-described image inference for outputting a predicted image after a predetermined time from a predetermined image input. Also, instead of steps S25 to S28 in FIG. 8, the control unit 11 of the imaging device 20 causes the inference engine 12 to perform image image inference, and displays based on the inference result.

Now, as in FIG. 6, it is assumed that the user 47 takes an image of a bird 46 perched on a branch 45 . Images P31a to P31d in FIG. 14 all show live view images immediately before the bird 46 takes off. The setting control unit 11d gives the live view image from the imaging unit 22 to the inference engine 12 to execute image inference. As a result of the image inference, the inference engine 12 predicts an image that will be captured after a predetermined time from the shooting time of the input live view image, and outputs the prediction result to the setting control unit 11d.

Based on the prediction result, setting control section 11d reads out the representative image stored in image data recording area 16a and provides it to display control section 11f. Thus, the display control unit 11f superimposes and displays a representative image that will be captured after a predetermined time on the current live view image. An image P31a in FIG. 14 shows an example of display in this case. In the image P31a, in addition to the image portion 61 immediately before the flight of the bird 46 included in the live view image, an image after two seconds is displayed. A predicted representative image 62a and a predicted representative image 62b as an image after 3 seconds are displayed. In addition, the display control unit 11f displays, in the vicinity of these images 62a and 63a, that the time that these images will be acquired is two seconds or three seconds later based on the acquisition time of the preview image. time displays 62b and 63b indicating .

As described above, the representative image is an image determined and recorded by the external device 30 and an image transferred from the external device 30 to the imaging device 20 . Therefore, there is a possibility that the representative image does not necessarily exist. Therefore, in this case, it is conceivable to copy and display the image portion 61 at the display position of the representative image. An image P31b in FIG. 14 shows a display example in this case, and in the image P31b, an image 2 seconds later is predicted in addition to the image portion 61 immediately before the bird 46 takes off, which is included in the live view image. An image 64a generated by copying the image portion 61 is displayed instead of the representative image. The display control unit 11f also displays a time display 64b near the image 64a indicating that the representative image is expected to be acquired in two seconds from the acquisition time of the preview image. ing.

An image P31c in FIG. 14 shows an example in which the display control unit 11f displays time displays 65b and 66b instead of the time displays 62b and 63b of the image P31a. Time displays 65b and 66b imitating the shape of hands of a clock and display of arrows indicate that the expected acquisition time of the representative images 62a and 63a is two seconds or three seconds after the shooting time of the preview image, respectively. there is

An image P31d in FIG. 14 shows an example of simultaneously displaying a plurality of representative images that are likely to be captured at the same time. In the above description of step S53 in FIG. 11, an example of selecting only one representative image has been described, but a plurality of images may be selected and recorded as the representative image. In this case, the setting control section 11d causes the plurality of representative images transferred from the external device 30 to be recorded in the image data recording area 16a.

Based on the prediction result of the inference engine 12, the setting control section 11d reads the representative image recorded in the image data recording area 16a and supplies it to the display control section 11f. When there are multiple representative images, the display control unit 11f displays the multiple representative images in an overlapping manner. An image P31d shows a display example in this case, and in the image P31d, in addition to the image portion 61 immediately before the flight of the bird 46 included in the live view image, an image after two seconds is predicted. Representative images 67a to 67c are displayed. In addition, the display control unit 11f displays a time display near these images 67a to 67c indicating that these images will be acquired two seconds after the acquisition time of the preview image. 67d is displayed.

As described above, in the present embodiment, machine learning that performs image inference for predicting an image after a predetermined time is realized using an image having time information as learning data. By applying the inference model obtained by this machine learning to, for example, an imaging device, image image inference is performed on live view images that change from moment to moment. Predict and present. The user can easily capture a bird in flight by performing, for example, a shooting operation in consideration of the presented image. In addition, images with time information used as learning data can be obtained very easily, and teacher data can be obtained from this learning data by relatively simple processing, enabling image image inference. You can easily create an inference model that

(Third Embodiment)
FIG. 15 is a flow chart showing the operation flow employed in the third embodiment of the invention. The hardware configuration of this embodiment is the same as that of the first embodiment. In FIG. 15, the same steps as in FIG. 11 are denoted by the same reference numerals, and descriptions thereof are omitted.

In the second embodiment, an example of generating an inference model for image image inference has been described. This is an example of generating and using an inference model that performs image position inference to predict the position of each image and the image after a predetermined time by creating pairs of images with intervals and their positional differences as training data. .

The flow of FIG. 15 differs from that of FIG. 11 in the method of creating teacher data, adopting step S61 instead of step S52 and omitting the processing of step S53. It should be noted that, as in the second embodiment, the external image DB 32 stores, for example, a continuous image group shown in FIG. 12 and the like.

The mother set creation unit 31a uses a set of two images before and after a predetermined time as training data by a time determination unit 31a1 and an object image determination unit 31a2, as in the second embodiment. For example, images P21 and P24, images P22 and P25, images P23 and P26, and images P24 and P27 in FIG. 12 are shown as pairs. The mother set creation unit 31a obtains positional information in the image for the target object image. For example, when the object image is the image of the bird in FIG. 12, the position of the bird's face and the tip of the tip of the toe may be obtained. Then, the mother set creation unit 31a obtains the difference in the positions of the object images in the set of images, and uses the position of the object in the image acquired earlier in time (hereinafter referred to as the previous image) as a reference. Second, the positional difference of the object in the image acquired later (hereinafter referred to as the post-image) is obtained. The mother set creating unit 31a records the relationship between the previous image and the position difference as teaching data in the teaching data recording unit 31e.

The method of generating the network 12a in this case is the same as in the second embodiment, and instead of the output image in FIG. , the network design is determined. That is, in the present embodiment, an inference model is obtained by inputting a previous image and performing image position inference for predicting the position of a subsequent image. Information on the network design thus determined is transmitted from the communication section 31c to the imaging device 20, and recorded as setting information in the setting data recording area 16b by the setting control section 11d.

Next, the operation of the imaging device 20 of the embodiment configured as described above will be described with reference to FIGS. 16 and 17. FIG. FIG. 16 is a flowchart showing the control of the control unit 11 of the imaging device 20, and FIG. 17 is an explanatory diagram showing a display example of an image displayed on the display screen of the display unit 15. FIG. The control flow of the control unit 11 shown in FIG. 16 is substantially the same as that in FIG. 8, except that steps S65 and S66 are employed instead of steps S27 and S28 in FIG.

In this embodiment, the inference engine 12 constructs an inference model that performs the above-described image position inference that outputs a predicted position after a predetermined time from a given image input.

Now, as in FIG. 6, it is assumed that the user 47 takes an image of a bird 46 perched on a branch 45 . Images P32a to P32d in FIG. 17 all show live view images immediately before the bird 46 takes off. The setting control unit 11d gives the live view image from the imaging unit 22 to the inference engine 12 to execute image position inference. As a result of the image position inference, the inference engine 12 uses the position difference from the input image, that is, the position of the object image in the input live view image as a reference, and the image is captured after a predetermined time from the time when the live view image was captured. It predicts the position of the object image in the image that is likely to be there, and outputs the prediction result to the setting control section 11d.

If the control unit 11 of the imaging device 20 determines in step S26 in FIG. 16 that the reliability of the prediction result by the inference engine 12 is sufficiently high, then in the next step S64, based on the inference result, a predetermined time A display is provided indicating the position of the object image in the image that will be captured later.

An image P32a in FIG. 17 shows a display example in this case, and the display control unit 11f displays an image (object image) portion 61 of the image P32a immediately before the bird 46 takes off, which is included in the live view image. In addition to , there is displayed a position display 64a indicating the position predicted as the image position of the object image two seconds later. The position display 64a is an image obtained by copying the image portion 61. FIG. The display control unit 11f also displays a time display 64b near this image 64a indicating that this image will be acquired two seconds after the acquisition time of the preview image. ing.

An image P32b in FIG. 17 shows an example in which the display control unit 11f displays a time display 71a instead of the time display 64a of the image P32a. In addition, the display control unit 11f displays a time display 71b near the image 71a indicating that the image will be acquired two seconds after the acquisition time of the preview image. ing. The time display 71a shows, by the shape of a curve, the range of positions in the image that the bird 46 will reach two seconds after the time the preview image was captured.

If the reliability of the inference is not sufficiently high in step S45, the control unit 11 displays a range of positions with relatively high reliability in step S66. An image P32c in FIG. 17 shows a display example in this case. The display control unit 11f displays a position range display 72a with two curved lines in the image P32c, and displays the range in which the subsequent image exists two seconds later. showing. In addition, the display control unit 11f displays a time display 72b near the position range display 72a indicating that the preview image will be acquired two seconds after the acquisition time of the preview image. is doing. It should be noted that the position range display 72a displays the closest inference results among a plurality of inference results with relatively high reliability (for example, 65 to 84%) when the reliability of the inference result of the image position inference is not sufficiently high. It indicates the range between the position and the furthest position.

An image P32d shows another display example in step S66. The display control unit 11f displays a circular position range display 73a in the image P32d to indicate the range in which the subsequent image will exist two seconds later. In addition, the display control unit 11f displays a time display 73b near the position range display 73a indicating that the preview image will be acquired in two seconds after the acquisition time of the preview image. is doing.

As described above, in the present embodiment, machine learning that performs image position inference for predicting the position of an image after a predetermined time is realized using an image having time information as learning data. By applying the inference model obtained by this machine learning to, for example, an imaging device, image position inference is performed on live view images that change from moment to moment. It can predict and present its position. The user can take a picture of the bird in flight by taking the presented position into consideration and, for example, performing a picture-taking operation. In addition, images with time information used as learning data can be acquired very easily, and teacher data can be acquired from this learning data by relatively simple processing, enabling image position inference. You can easily create an inference model that

(Fourth embodiment)
18 to 20 are explanatory diagrams for explaining the fourth embodiment of the present invention. The hardware configuration of this embodiment is the same as that of the first embodiment. FIG. 18 is an explanatory diagram showing a shooting scene in this embodiment.

FIG. 18 shows the user 47 photographing a bird 46 perched on a branch 45 of a tree. A river 81 flows between the user 47 and the tree, and fish 49 are swimming in the river 81 . A bird 46 perched on a branch 45 may take off into the sky, jump to a branch of an adjacent tree, or glide toward a fish 49 in a river 81 . In FIG. 18, these states are indicated by reference numerals 46h, 46i and 46j, respectively. In this case, if it is possible to predict in advance in which direction the bird 46 will fly away, it is possible to effectively photograph the bird 46 . The example of FIG. 18 indicates that the bird 46 has a 15% probability of being in the state 46h or 46i and a 70% probability of being in the state 46j. The present embodiment enables such prediction.

In the second embodiment, an image having time information is used as learning data, and a set of a plurality of images having a predetermined time interval is created as training data by using this learning data. This is an example of generating and using an inference model that performs image inference for predicting a (previous image) and an image after a predetermined time (post-image). The present embodiment is an example of generating and using an inference model that performs image image direction inference for predicting a later image and a moving direction after a plurality of times with respect to a previous image.

In this embodiment, for example, a continuous image group shown in FIG. 12 can be used as learning data. The mother set creating unit 31a generates teacher data in which a previous image and a later image after a plurality of predetermined times from the acquisition time of the previous image are paired, and stores the teacher data in the teacher data recording unit 31e. The input/output modeling unit 31d obtains the post-image corresponding to the pre-image and the moving direction of the post-image and selects a representative image to be used as the post-image every predetermined time. In this case, a plurality of representative images are selected according to the movement direction. For example, a representative image may be selected every 15 degrees in the moving direction. The network design information and the representative image thus generated are transmitted to the imaging device 20, the recording control unit 11c of the imaging device 20 records the network design information in the setting data recording area 16b, and the representative image is transferred to the image data recording area. 16a.

Next, an embodiment configured in this way will be described with reference to FIGS. 18 to 20. FIG. 19 and 20 are explanatory diagrams showing display examples of images displayed on the display screen of the display unit 15. FIG.

In the present embodiment, the inference engine 12 constructs an inference model that performs the above-described image image direction inference for outputting a plurality of predicted images after a predetermined time and their moving directions for a predetermined image input. User 47 is attempting to photograph bird 46 after it has taken off from branch 45 . Images P41 to P44 in FIG. 19 show live view images at a predetermined time, and time passes in the order of images P41 to P44.

An image 46a in the image P41 shows a bird 46 perched on a branch. This image P41 is displayed as a live view image on the display screen 15a. An image P42, which is a live view image acquired after a predetermined time from the image P41, is displayed with a circle indication 51 indicating that there is an inference model related to the subject in the image P42. An image 46b of the bird 46 in the image P42 shows the bird 46 about to take off. For example, assume that image orientation inference is not reliable enough at this point. In this case, in the image P42, as a result of the image image direction inference by the inference engine 12, the display control unit 11f displays a time display 88b indicating that the image is predicted after 5 seconds to 2 seconds, and the movement of the bird 46. A display showing the predicted direction is displayed.

In the example of FIG. 19, the image P42 contains a probability display 85hp indicating that there is a 15% chance that the bird 46 will fly upwards, a representative image 85h in that case, and a 15% possibility that the bird 46 will fly to the next branch. % and a representative image 85i in that case, and a probability display 85jp showing that the possibility of glide horizontally or downward is 70% and a representative image 85j in that case. The probability indicated by each probability display is obtained based on the reliability value of each direction obtained as a result of inference.

Further, an image P43, which is a live view image acquired after a predetermined time from the image P42, is displayed with a circle indication 51 indicating that there is an inference model related to the subject in the image P43. An image 46c of the bird 46 in the image P43 shows the state just before the bird 46 takes off. For example, assume that the reliability of image orientation inference at this point is sufficiently high. In this case, in the image P43, as a result of the image image direction inference by the inference engine 12, a display indicating the moving direction of the bird 46 one second later is displayed by the display control unit 11f.

In the example of FIG. 19, the image P43 includes a probability display 86ip indicating that the bird 46 has a 5% chance of jumping to the next branch, a representative image 86i of that case, and a possibility of glide horizontally or downward. A probability display 86jp indicating that the sex is 95% and a representative image 86j in that case are included. Also displayed is a time display 88c indicating that the prediction is for one second from the current time.

By confirming the image P43 displayed on the display screen 15a of the display unit 15, the user 47 can predict that the bird 46 will glide substantially horizontally after taking off from the branch 45. FIG.

For example, the user 47 can relatively easily keep the bird 46 in the shooting range by checking the image P43 and estimating the moving direction of the bird 46 . As a result, the user 47 can capture the desired decisive moment, such as the moment when the bird 46 catches the fish 49 .

FIG. 20 shows the display on the display screen 15a in this case, in which an image 49p of a fish and an image 46p of a bird holding a fish are displayed.

Note that the image P44 in FIG. 19 shows a live view image one second after the image P43 was captured. A probability display 87jp indicating that is displayed.

As described above, in the present embodiment, an image having time information is used as learning data, and machine learning is realized that performs image image direction inference for predicting an image after a predetermined plurality of times and its movement direction. By applying the inference model obtained by this machine learning to, for example, an imaging device, image image orientation inference is performed on live view images that change from moment to moment. can be predicted and presented. The user can easily capture a bird in flight by performing, for example, a shooting operation in consideration of the presented image. In addition, images with time information used as learning data can be obtained very easily, and teacher data can be obtained from this learning data by relatively simple processing, and image direction inference can be performed. You can easily create inference models that enable it.

In each of the above-described embodiments, only an example in which a bird is assumed as an object has been described, but any object can be used, and the decisive moment can be predicted by learning from images before and after. Anything is possible as long as it is something. For example, it may be possible to predict the moment a bird enters the water surface, the moment a fish jumps out of the water surface, or the moment a cat or dog turns around.

In addition, it is not limited to animals, and it is possible to predict the moment when a milk crown will appear, or the moment when fireworks will open. Alternatively, the moment of impact of a golf swing, which is relatively easy to predict, may be predicted. Furthermore, the moment of cell division, egg cleavage, eclosion, hatching, etc. may be predicted. It is relatively difficult to confirm the moment of cell division, and it would be extremely useful if the moment of division could be predicted in minutes. Also, for example, by imaging the state of cooking, it is possible to predict the moment when the fire will be turned off.

Furthermore, in each of the above embodiments, an example of inferring time, image, position, and direction from images has been described, but it is also possible to perform these inferences based on sound. For example, it is possible to predict the courtship behavior from the cry of the animal requesting the courtship behavior. It is also possible to predict the moment of the cry until courtship behavior is started from the image of the animal, that is, it is also possible to predict the timing of sound generation from the image.
As described above, the concept of the present invention can be applied to all information that can be obtained, such as voice and images, and it is possible to comprehensively judge not only prediction from one to the other but also learning using both data. you can go For example, for each frame of the image, it is possible to learn by inserting fragmentary information of the sound at the corresponding acquisition time. In addition to courtship behavior, there are also decisive moments such as egg laying, emergence, and hatching. In addition, it is also possible to develop medical care that predicts the disease that the patient will develop later by making inferences based on heart sounds, breath sounds, intestinal peristaltic sounds, and the like. The condition of wheezing and breathing in asthma is easy to detect at an early stage because the condition changes depending on the deterioration. It can also be used for early treatment of infants and people with disabilities who cannot express themselves in words.

In the above embodiment, the imaging device requests the external device to create and transfer the inference model, but the inference model can be created by any device, for example, using a computer on the cloud. You may

In the above embodiments, a digital camera was used as an image capturing device, but the camera may be a digital single-lens reflex camera, a compact digital camera, or a video camera such as a video camera or movie camera. Alternatively, a camera built in a mobile information terminal (PDA: Personal Digital Assist) such as a mobile phone or smart phone may be used. Alternatively, the imaging unit may be separate from the imaging device.

The present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the constituent elements without departing from the scope of the present invention at the implementation stage. Also, various inventions can be formed by appropriate combinations of the plurality of constituent elements disclosed in the above embodiments. For example, some components of all components shown in the embodiments may be omitted. Furthermore, components across different embodiments may be combined as appropriate.

Regarding the operation flow in the claims, the specification, and the drawings, even if explanations are made using "first," "next," etc. for the sake of convenience, it is essential to implement them in this order. does not mean Further, it goes without saying that each step constituting these operation flows can be appropriately omitted as long as it does not affect the essence of the invention.

It should be noted that, among the techniques described here, the control described mainly in the flow charts can often be set by a program, and may be stored in a recording medium or a recording unit. The method of recording in the recording medium and the recording unit may be recorded at the time of product shipment, using a distributed recording medium, or downloading via the Internet.

It should be noted that, in the embodiments, portions described as "parts" (sections or units) may be configured by combining a dedicated circuit or a plurality of general-purpose circuits, and if necessary, pre-programmed software A processor such as a microcomputer, a CPU, or a sequencer such as an FPGA may be combined to operate according to the above. It is also possible to design a part or all of the control by an external device, in which case a wired or wireless communication circuit is interposed. Communication may be performed by Bluetooth, WiFi, telephone line, or the like, and may be performed by USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrated into an ASIC. The moving part and the like are composed of various actuators and, if necessary, a connecting mechanism for movement, and the actuators are operated by a driver circuit. This drive circuit is also controlled by a microcomputer, ASIC, etc. according to a specific program. Such control may include detailed corrections and adjustments based on information output from various sensors and their peripheral circuits. In addition, we have explained an embodiment in which decisions are made based on learning results determined by artificial intelligence using the terms inference model and learned model, but this can also be replaced by simple flowcharts, conditional branching, or numerical judgments involving calculations. Sometimes. Further, machine learning may be performed within the imaging device by improving the computing power of the control circuit of the camera or by narrowing down to a specific situation or object.

[Appendix 1]
a time determination unit that obtains an image after a predetermined time from each image in a series of images having time information based on shooting time;
an object image determination unit for detecting an image state of a specific object in each image after a predetermined time, for each image in the series of images;
A teaching data creating apparatus, comprising: a control unit that combines each of the images and data of an image state of a specific object after a predetermined period of time obtained for each of the images and sets the data as teaching data.

[Appendix 2]
3. The training data creation apparatus according to claim 2, wherein the control unit selects an image similar to an image state of the specific object after a predetermined time from the series of images as a representative image.

[Appendix 3]
Inference model generation for generating an inference model for inferring an image state of a predetermined object after a predetermined time from an input image by machine learning using the teacher data created by the teacher data creation device according to additional item 1. A learning device characterized by comprising:

[Appendix 4]
an inference engine that realizes an inference model generated by the learning device of appendix 3;
an imaging unit;
and a setting control unit that provides an image captured by the image capturing unit to the inference engine and obtains an inference result of an image state of the predetermined object in the captured image after a predetermined time. .

[Appendix 5]
a time determination unit that obtains an image after a predetermined time from each image in a series of images having time information based on shooting time;
an object image determination unit for detecting an image position of a specific object in each image after a predetermined time, for each image in the series of images;
A teaching data creating apparatus, comprising: a control unit that combines each of the images and the data of the image position of the specific object after a predetermined time obtained for each of the images and sets the data as teaching data.

[Appendix 6]
Inference model generation for generating an inference model for inferring the image position of a predetermined object after a predetermined time from an input image by machine learning using the teacher data created by the teacher data creation device according to additional item 5. A learning device characterized by comprising:

[Appendix 7]
an inference engine that implements the inference model generated by the learning device of appendix 6;
an imaging unit;
and a setting control unit that provides an image captured by the image capturing unit to the inference engine and obtains an inference result of an image position of the predetermined object after a predetermined time in the captured image. .

[Appendix 8]
a time determination unit that obtains an image after a plurality of predetermined times for each image in a series of images having time information based on shooting time;
an object image determining unit for detecting, for each image in the series of images, an image position and moving direction of a specific object in each image after a plurality of predetermined times;
and a control unit that combines each of the images and data of image positions and moving directions after a plurality of predetermined times of the specific object obtained for each of the images and sets them as teacher data. creation device.

[Appendix 9]
An inference model that infers the image position and movement direction of a predetermined object after a plurality of predetermined times from the input image by machine learning using the teacher data created by the teacher data creation device according to additional item 8. A learning device comprising an inference model generation unit that generates an inference model.

[Appendix 10]
an inference engine that implements the inference model generated by the learning device of appendix 9;
an imaging unit;
a setting control unit that supplies an image captured by the imaging unit to the inference engine and obtains inference results of image positions and moving directions of the predetermined object after a plurality of predetermined times in the captured image. An imaging device characterized by:

DESCRIPTION OF SYMBOLS 11... Control part 11a... Imaging control part 11b... Image processing part 11c... Recording control part 11d... Setting control part 11e... Communication control part 11f... Display control part 12... Inference engine 12a... Network, DESCRIPTION OF SYMBOLS 13... Operation part 14, 31b, 31c, 33... Communication part, 15... Display part, 16... Recording part, 16a... Image data recording area, 16b... Setting data recording area, 20... Imaging device, 22... Imaging part, 22a... image sensor, 22b... optical system, 30... external device, 31... learning unit, 31a... mother set creation unit, 31a1... time determination unit, 31a2... object image determination unit, 31d... input/output modeling unit, 31e ... teaching data recording section, 31f ... display section, 32 ... external image DB.

Claims (11)

an object image determination unit that detects a specific state image, which is an image of a specific object in a specific state, from a series of images having time information based on shooting time;
a time determination unit that determines a time difference between the shooting time of each image of the series of images and the shooting time of an image including the specific state image among the series of images;
A teaching data creating apparatus, comprising: a control unit that sets each image and time difference data obtained for each image as teaching data. An inference model for generating an inference model for inferring the time at which a given object is in the specific state from an input image by machine learning using the teacher data created by the teacher data creation device according to claim 1. A learning device comprising a generator. an inference engine that implements an inference model generated by the learning device according to claim 2;
an imaging unit;
a setting control unit that provides an image captured by the imaging unit to the inference engine and obtains an inference result of the time required for the predetermined object in the captured image to reach the specific state. imaging device. 4. The imaging apparatus according to claim 3, further comprising a display control section for performing display control for displaying the inference result of the time on the display section. The captured image is a live view image,
5. The imaging apparatus according to claim 4, wherein the display control unit displays the inference result of the time by superimposing it on the live image displayed on the display unit. The setting control unit acquires reliability information of the inference result together with the inference result of the time,
4. The imaging apparatus according to claim 3, wherein the display control unit changes a display form of the time inference result based on the reliability information. The display control unit displays the time when the reliability of the inference result is equal to or higher than a predetermined threshold, and displays the time width when the reliability of the inference result is lower than the predetermined threshold. 7. The imaging device according to claim 6. a detection step of detecting a specific state image, which is an image of a specific object in a specific state, from a series of images having time information based on shooting time;
Determining a time difference between the photographing time of each image of the series of images and the photographing time of an image including the specific state image among the series of images;
and a generation step of generating training data by pairing each of the images with the time difference data obtained for each of the images. 9. The teacher data creation method according to claim 8, wherein the detection step detects the specific object by manual operation or recognition processing, and detects the specific state by image analysis processing. The generation step excludes the series of images from the training data if the number of images containing the specific object is less than a predetermined number, and excludes images that do not contain the specific object from the training data. 9. The teaching data creation method according to claim 8, wherein: to the computer,
a detection step of detecting a specific state image, which is an image of a specific object in a specific state, from a series of images having time information based on shooting time;
Determining a time difference between the photographing time of each image of the series of images and the photographing time of an image including the specific state image among the series of images;
A training data generation program for executing a generation step of generating training data by pairing each of the images and the time difference data obtained for each of the images.

Images