JP2020027982A - Teacher data preparation device, teacher data preparation method, teacher data preparation program, learning device, and imaging device - Google Patents

Abstract

To predict a desired timing from an image of a subject by machine learning.SOLUTION: A teacher data preparation device includes: a subject image determination section to detect a specific state image that is an image of a specific subject in a specific state from a series of images having time information based on time of shooting; a time determination section to determine a time difference between time of shooting of each image for each image of the series of images and time of shooting of an image including the specific state image among the series of images; and a control section to pair each image and data of time difference obtained for each image, as teacher data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a teacher data creation device, a teacher data creation method, a teacher data creation program, a learning device, and an imaging device for machine learning.

  2. Description of the Related Art In recent years, portable devices with a photographing function (photographing devices) such as digital cameras have become widespread. In this type of photographing device, there is a device in which various settings at the time of photographing are automated. For example, some digital cameras are equipped with an AF function that automates focusing and an automatic exposure (AE) function that automates exposure. In addition, photographing devices having a so-called continuous photographing function for continuously photographing have been widely used.

  By the way, a technique for obtaining a desired inference result by machine learning on a captured image acquired by such a photographing device has been developed. The machine learning is to obtain an inference result about an unknown matter by learning features, time-series information, spatial information, and the like of known input information and performing inference based on the learning result. That is, in machine learning, first, a learned model for enabling inference of a determinable output result from specific input information is obtained.

  In order to obtain an inference result with high reliability, a large amount of information having a known relationship between inputs and outputs is used as learning data when generating a learned model. For example, in deep learning, a network is designed and designed so that an expected output is obtained for a known input using a large amount of learning data. The learned model (hereinafter also referred to as an inference model) obtained by such a process can be used independently of the network that has performed the learning.

  For example, in Patent Document 1, in order to prevent the learning accuracy from deteriorating even when the number of learning data is small, a relationship between a set of a first content and a second content having a type different from the first content is provided. A generation unit that generates a new second learning device by using a part of the first learning device that deeply learns the gender; a first content that is generated by the second learning device generated by the generation unit; There is disclosed a technology including a learning unit that deeply learns the relationship of a set with a third content of a type different from the content.

Japanese Patent No. 6151404

  Conventionally, no device has been developed that performs machine learning for predicting a desired timing based on an image of a subject that moves in an unknown manner.

  The present invention provides a teacher data creation device, a teacher data creation method, a teacher data creation program, a learning device, and an imaging device capable of predicting a desired timing from an image of a subject by machine learning. With the goal.

  A teacher data creation device according to an aspect of the present invention is an object image determination unit that detects a specific state image that is an image in a specific state of a specific object from a series of images having time information based on a shooting time, A time determining unit that determines a time difference between a photographing time of each of the images of the series of images and a photographing time of an image including the specific state image in the series of images, and the respective images and the respective images. And a control unit that sets the data of the time difference obtained for (1) as teacher data.

  A learning device according to one aspect of the present invention is an inference model that infers a time when a predetermined target is in the specific state from an input image by machine learning using teacher data created by the teacher data creating device. Is provided.

  An imaging device according to an aspect of the present invention includes an inference engine that implements an inference model generated by the learning device, an imaging unit, and an image captured by the imaging unit, which is provided to the inference engine, and the A setting control unit that obtains an inference result of a time required for the predetermined object to reach the specific state.

  The teacher data creation method according to one aspect of the present invention includes a detecting step of detecting a specific state image that is an image in a specific state of a specific target object from a series of images having time information based on a shooting time; Determining a time difference between the photographing time of each of the images for each of the images and the photographing time of the image including the specific state image in the series of images; and And generating a set of data as teacher data.

  The teacher data creation program according to one aspect of the present invention, a computer, from a series of images having time information based on the shooting time, a detection step of detecting a specific state image that is an image in a specific state of a specific object, Determining a time difference between a photographing time of each of the images in the series of images and a photographing time of an image including the specific state image in the series of images; And generating a set of the time difference data as teacher data.

  According to the present invention, there is an effect that it is possible to predict a desired timing from an image of a subject by machine learning.

FIG. 1 is a block diagram showing a learning device and an imaging device according to a first embodiment of the present invention. FIG. 3 is an explanatory diagram for explaining a network 12a of the inference engine 12. Explanatory drawing which shows an example which captures each image of the image group 34a. FIG. 4 is an explanatory diagram showing a relationship between each image of an image group 34a and a photographing time. 9 is a flowchart for explaining a method of creating teacher data by the population creating unit 31a. FIG. 4 is an explanatory diagram for explaining the operation of the first embodiment. FIG. 4 is an explanatory diagram for explaining the operation of the first embodiment. 5 is a flowchart showing the operation of the imaging device 20. 5 is a flowchart showing the operation of the external device 30. FIG. 4 is an explanatory diagram for explaining the operation of the first embodiment. 9 is a flowchart showing an operation flow employed in the second embodiment of the present invention. Explanatory drawing which shows an example of the continuous image group taken in by the mother set preparation part 31a from the external image DB32. FIG. 3 is an explanatory diagram for explaining a method of generating a network 12a. FIG. 7 is an explanatory diagram illustrating a display example of an image displayed on a display screen of a display unit 15; 13 is a flowchart showing an operation flow employed in the third embodiment of the present invention. 9 is a flowchart illustrating control of the control unit 11 of the imaging device 20. FIG. 7 is an explanatory diagram illustrating a display example of an image displayed on a display screen of a display unit 15; FIG. 14 is an explanatory diagram for describing a fourth embodiment of the present invention. FIG. 14 is an explanatory diagram for describing a fourth embodiment of the present invention. FIG. 14 is an explanatory diagram for describing a fourth embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a learning device and an imaging device according to the first embodiment of the present invention. In the present embodiment, machine learning is realized in which an image having time information is used as learning data to predict a time until a predetermined moment (hereinafter, also referred to as a deterministic moment) is reached. As a specific example, an inference model that predicts the moment at which a bird takes off by machine learning is constructed, and the prediction result at the moment at which a bird takes off from a live view image can be displayed using the inference model.

  The imaging device 20 in FIG. 1 records an image obtained by imaging a subject. As the imaging device 20, not only a digital camera and a video camera, but also a camera built in a smartphone or a tablet terminal may be adopted. As described later, the imaging device 20 can use an inference model at the time of live view display. However, the imaging device 20 may use an inference model that is mounted in advance, and an external device. An inference model may be obtained from 30.

  The imaging device 20 includes a control unit 11 and an imaging unit 22. The control unit 11 may be configured by a processor using a CPU (Central Processing Unit) or the like, and may operate according to a program stored in a memory (not shown) to control each unit, or may be a hardware electronic circuit. May realize some or all of the functions.

  The imaging unit 22 has an imaging element 22a and an optical system 22b. The optical system 22b includes a lens and a stop (not shown) for zooming and focusing. The optical system 22b includes a zoom (magnification) mechanism (not shown) for driving these lenses, and a focus and aperture mechanism.

  The imaging device 22a is configured by a CCD, a CMOS sensor, or the like, and an optical system 22b guides a subject optical image to an imaging surface of the imaging device 22a. The image sensor 22a photoelectrically converts the subject optical image to obtain a captured image (imaging signal) of the subject.

  The imaging control unit 11a of the control unit 11 can control the zoom mechanism, the focus mechanism, and the aperture mechanism of the optical system 22b to adjust the zoom, the aperture, and the focus. The imaging unit 22 performs imaging under the control of the imaging control unit 11a, and outputs an imaging signal of a captured image (a moving image and a still image) to the control unit 11.

  An operation unit 13 is provided in the imaging device 20. The operation unit 13 includes a release button, a function button, various switches for setting a shooting mode, parameter operation, and the like, a dial, a ring member, and the like, and outputs an operation signal based on a user operation to the control unit 11. The control unit 11 controls each unit based on an operation signal from the operation unit 13.

  The control unit 11 captures captured images (moving images and still images) from the imaging unit 22. The image processing unit 11b of the control unit 11 performs predetermined signal processing, for example, color adjustment processing, matrix conversion processing, noise removal processing, and other various signal processing on the captured image.

  The display unit 15 is provided in the imaging device 20, and the display control unit 11f is provided in the control unit 11. The display unit 15 has, for example, a display screen such as an LCD (liquid crystal display), and the display screen is provided on, for example, the back of the housing of the imaging device 20. The display control unit 11f causes the display unit 15 to display the captured image signal-processed by the image processing unit 11b. In addition, the display control unit 11f can also display various menu displays, warning displays, and the like of the imaging device 20 on the display unit 15.

  The communication unit 14 is provided in the imaging device 20, and the communication control unit 11 e is provided in the control unit 11. The communication unit 14 can transmit and receive information to and from the external device 30 under the control of the communication control unit 11e. The communication unit 14 can perform short-range wireless communication such as Bluetooth (registered trademark) and wireless LAN communication such as Wi-Fi (registered trademark). Note that the communication unit 14 is not limited to Bluetooth and Wi-Fi, and can employ communication using various communication methods. The communication control unit 11e can receive the information of the inference model from the external device 30 via the communication unit 14.

  The control unit 11 is provided with a recording control unit 11c. The recording control unit 11c can compress the captured image after the signal processing and supply the compressed image to the recording unit 16 for recording. The recording unit 16 is configured by a predetermined recording medium, and can record information provided from the control unit 11 and output the recorded information to the control unit 11. The recording unit 16 may employ a card interface, for example. In this case, the recording unit 16 can record image data on a recording medium such as a memory card.

  The recording unit 16 has an image data recording area 16a, and the recording control unit 11c records image data in the image data recording area 16a. Further, the recording control unit 11c can also read and reproduce information recorded in the recording unit 16.

  The recording unit 16 has a setting data recording area 16b. In the setting data recording area 16b, setting information of the inference model is recorded.

  In the present embodiment, the imaging device 20 is provided with an inference engine 12 as an inference unit. The inference engine 12 has a network 12a. The network 12a is constructed using the set values recorded in the recording unit 16, and constitutes a network obtained by completing learning in machine learning, that is, an inference model.

  The recording control unit 11c receives information for forming an inference model from the learning unit 31 that is the external device 30 via the communication unit 14, and records the setting information in the setting data recording area 16b of the recording unit 16. You may be able to do it.

  2 to 4 are explanatory diagrams for explaining the network 12a of the inference engine 12. FIG. In FIG. 2, a large amount of data sets 31G corresponding to inputs and outputs are given as teacher data to a predetermined network N1. Thus, the network design of the network N1 is determined so that an output corresponding to the input is obtained. In the present embodiment, an image is used as an input, and an estimated time up to a decisive moment is obtained as output along with reliability information (reliability). Information on the determined network design of the network N1 is recorded as setting information in the setting data recording area 16b.

"Deep learning" is a multi-layered process of "machine learning" using a neural network. A “forward-propagation neural network” that sends information from the front to the back to make a determination is a typical example. In the simplest case, it is composed of an input layer composed of N1 neurons, an intermediate layer composed of N2 neurons given by parameters, and N3 neurons corresponding to the number of classes to be determined. Only three output layers are required. Each neuron of the input layer and the intermediate layer and the neuron of the intermediate layer and the output layer are connected by a connection weight, and a bias value is applied to the intermediate layer and the output layer, so that a logic gate can be easily formed. For simple discrimination, three layers may be used. However, if the number of intermediate layers is large, it becomes possible to learn how to combine a plurality of feature amounts in the process of machine learning. In recent years, those having 9 to 152 layers have become practical due to the relationship between the time required for learning, the determination accuracy, and energy consumption.
As the network N1 used for machine learning, various known networks may be used. For example, R-CNN (Regions with CNN features) using CNN (Convolution Neural Network) or FCN (Fully Convolutional Networks) may be used. This involves a process called “convolution” that compresses the feature amount of an image, moves with minimal processing, and is strong in pattern recognition. Further, a "recursive neural network" (a fully connected recurrent neural network) that can handle more complicated information and that has a bidirectional flow of information in response to information analysis whose meaning changes depending on the order may be used.
To implement these technologies, conventional general-purpose arithmetic processing circuits such as CPUs and FPGAs (Field Programmable Gate Arrays) may be used, but most of the processing of neural networks is matrix multiplication. For this reason, what is called a GPU (Graphic Processing Unit) or Tensor Processing Unit (TPU) specialized in matrix calculation may be used. In recent years, such a "neural network processing unit (NPU)" dedicated to artificial intelligence (AI) has been designed so that it can be integrated and incorporated with other circuits such as a CPU, and may be part of a processing circuit. is there.
Further, the inference model may be obtained by employing not only the deep learning but also various known machine learning techniques. For example, there is a technique called support vector machine or support vector regression. The learning here is to calculate the weight, filter coefficient, and offset of the classifier, and there is another method using logistic regression processing. When making a machine judge something, it is necessary for a human to teach the machine how to make the judgment.In this embodiment, the method of deriving the image judgment by machine learning was adopted. A rule-based method that applies rules acquired by humans through empirical rules and heuristics may be applied and used.

  The external device 30 includes a learning unit 31 that determines such a network design and an external image database (DB) 32 that records a large amount of learning data. The learning unit 31 has a communication unit 31b, and the external image DB 32 has a communication unit 33. The communication units 31b and 33 can communicate with each other. The communication unit 31c of the learning unit 31 can communicate with the communication unit 14.

  The learning unit 31 has a control unit 31g. The control unit 31g is configured by a processor using a CPU or the like, and operates according to a program stored in a memory (not shown) to control each unit. Alternatively, some or all of the functions may be realized by hardware electronic circuits. The entire learning unit 31 may be configured by a processor using a CPU, a GPU (Graphics Processing Unit), an FPGA, or the like, and may operate according to a program stored in a memory (not shown) to control learning. Alternatively, some or all of the functions may be realized by a hardware electronic circuit.

  The external image DB 32 includes an image classification recording unit 34. The image classification recording unit 34 is configured by a recording medium (not shown) such as a hard disk or a memory medium, and classifies and records a plurality of images for each type of object included in the images. Although the example of FIG. 1 shows an example in which the image classification recording unit 34 records only the image group 34a of the target object type A, the number of types to be classified can be set as appropriate.

  In the present embodiment, for example, a bird image group is recorded as the image group 34a. 3 and 4 are for explaining each image of the image group 34a. FIG. 3 is an explanatory diagram illustrating an example of capturing each image of the image group 34a, and FIG. 4 is an explanatory diagram illustrating a relationship between each image of the image group 34a and a shooting time.

  FIG. 3 shows a state in which a bird 46 stopping on a tree branch 45 is photographed by the camera 40. A display screen 42 composed of an LCD or the like is provided on the back of the camera 40, and indicates that an image of the bird 46 is displayed as a live view. The bird 46 can be photographed by operating the shutter button 41. In the present embodiment, a series of states until the bird 46 flies off the branch 45 is imaged. For example, when the camera 40 has a continuous shooting function, the continuous shooting function is used to take an image of a series of states from the state where the bird 46 stops on the branch 45 to the time when the bird 46 jumps off the branch 45 at a predetermined time interval. And may be obtained as a series of images. When the camera 40 has a moving image capturing function, a moving image obtained by capturing a moving image of a series of states from the state where the bird 46 is stopped at the branch 45 to the time it flies off the branch 45 may be obtained. . Further, by operating the shutter button 41 of the camera 40 at a predetermined time interval, a series of states from the state where the bird 46 stops at the branch 45 to the time it jumps off the branch 45 is imaged at a predetermined time interval and discretely. May be obtained.

  FIG. 4 shows six images P1 to P6 acquired by the camera 40 arranged in chronological order. Each of the images P1 to P6 is an image obtained by photographing the bird 46 in FIG. Each of the images P1 to P6 includes time information, and the number below each image in FIG. 4 indicates the time obtained based on the image P5. In the example of FIG. 4, the images P1 to P6 are captured at −20 seconds (2 seconds before), −15 seconds, −10 seconds, −5 seconds, 0 seconds, and +5 seconds with the image P5 as a time reference. is there. In general, a timer is built in the camera 40, and time information is added to each of the images P1 to P6 by the timer. Information of relative time based on the time may be added to each of the images P1 to P6. For example, in the example of FIG. 4, the shooting time of the image P1 is 15:30:27, and the relative acquisition times of the images P2 to P6 may be stored as time information based on this time.

  For example, it is assumed that the image that the user wants to photograph most is an image P5 including an image (hereinafter, also referred to as a specific state image) at the moment when the bird 46 is about to fly right off the branch 45. This moment is defined as the decisive moment, and the image P5 is referred to as the decisive moment image. In the present embodiment, a series of images having time information as shown in FIG. 4 is recorded in the image classification recording unit 34 as an image group 34a. The image classification recording unit 34 may record an image of a bird of the same type or an image group of birds classified as having substantially the same size. In addition to these images, a group of images of different types or bird images classified into different sizes may be recorded.

  As shown in FIG. 4, the bird sometimes performs a preparatory operation to extend his bent legs and spread his wings at a predetermined time before taking off. For example, depending on the type of birds, the size of the birds, or whether or not they are aiming for prey, the preparatory movements may be slightly different, but a huge image of birds taking off If learning is performed on the data, it is possible to predict the time at the moment of taking off from the state before taking off.

  In the image group 34a, for example, an enormous data group of a series of images having time information as shown in FIG. 4 is recorded by shooting as shown in FIG. The mother set creation unit 31a of the learning unit 31 reads images from the external image DB 32 and creates a mother set as a source of learning.

  The learning data to be provided to the learning unit 31 can be acquired from the imaging device 20. In this case, the imaging device 20 adds time information from a timer (not shown) built in the control unit 11 to the captured image acquired by the imaging unit 22 and transmits the captured image to the learning unit 31 via the communication unit 14. I do.

In the present embodiment, the population creating unit 31a includes a time determining unit 31a1 and an object image determining unit 31a2. The population creation unit 31a is controlled by the control unit 31g to create teacher data used for generating an inference model. The object image determination unit 31a2 determines an image portion (hereinafter, referred to as an object image) of the object to be determined at the definitive moment, and determines the definitive moment when the object image is in a specific state of the object. It is determined whether or not the image at the time of reaching (specific state image) has been reached. Further, the time determination unit 31a1 determines the time until each image reaches a decisive moment (specific state) based on the time information added to each image. That is, the time determination unit 31a1 determines, for each image in the series of images, the time difference between the shooting time of each image and the imaging time of the image including the specific state image at the decisive moment in the series of images. The control unit 31g sets information of each image and the determined time difference as a set and sets it as teacher data.
Here, the description of the specific state refers to the state in which the shape of the object itself being imaged and following changes over time to a specific posture and orientation, and the color and size of the object. It also includes changes and the like, and indicates that the shape, position, orientation, and the like in the imaging screen have changed. The state may include a time change of a sound emitted from the object. In addition, the image and the sound may be comprehensively determined so that the user can set and designate the start and end of an event, such as performance or dance, or the start or end of an event, or the climax or highlight of the event. Examples of the designation method include character input, voice input, item selection, and similar information input. In addition, even if not specified one by one, what many people feel is a decisive moment may be determined automatically, or it may occur not only at one specific timing but also at a plurality of timings. The time difference may be plural as a numerical value.

  FIG. 5 is a flowchart for explaining a method of creating teacher data by the mother set creating unit 31a. In step S1 of FIG. 5, the population creating unit 31a acquires a series of image groups to which time information has been added. For example, images P1 to P6 in FIG. 4 are obtained. The target image determining unit 31a2 of the population creating unit 31a determines a target image from each image of the acquired series of images.

  First, the object image determining unit 31a2 determines whether manual selection has been instructed. The manual selection is performed by an operation in which a user specifies a target image from the images. The learning unit 31 is provided with a display unit 31f configured by an LCD or the like, and the display unit 31f is provided with a touch panel (not shown) for receiving a user operation. The target image determining unit 31a2 displays a series of images on the display unit 31f (step S9), and determines the subject specified by the user's touch operation as a manually selected target image. If a manual selection has been made by the user, the series of images is set as teacher data candidates (step S10).

  If manual selection has not been specified, the object image determining unit 31a2 determines an object image in step S3. For example, the target image determining unit 31a2 may determine a subject located at the center of the image with a size equal to or larger than a predetermined size as the target, and determine the image portion as the target image. Further, what should be the target object may be specified in advance by a user operation. For example, when a bird is specified as the target, the target image determining unit 31a2 may determine the bird as the target by a known recognition process on the captured image.

  In the next step S4, the target image determining unit 31a2 excludes images that do not include the target image from the series of image groups, and determines whether the number of remaining images is equal to or greater than a predetermined number (step S5). . When the number of images including the target object image in the series of images is smaller than a predetermined number, it is considered that it is difficult to determine the decisive moment or the time until the decisive moment. The image group is excluded from candidates for the teacher data group in step S11.

  When the number of images including the target image is equal to or more than the predetermined number, the target image determining unit 31a2 shifts the processing to step S6, selects an image group including the target image, and in step S7, Image candidates for selecting a crucial instant image are performed. In the example of FIG. 4, an example in which the moment when the bird takes off is set as the definitive moment is shown. For example, the target object image determination unit 31a2 has the target object image spread most vertically, horizontally, and horizontally in the image by the image analysis processing. The image is detected as a specific state image, and an image including the specific state image is determined as a definitive instantaneous image candidate. In the example of FIG. 4, the image P5 is a candidate for the decisive instantaneous image.

  In the next step S8, the target object image determining unit 31a2 determines whether or not there is a predetermined number or more of images before the crucial instant image candidate. If the number of images before the decisive instant image in the image group including the target image is less than the predetermined number of images, the time to the definitive instant is too short to be easily used. Such an image group is excluded from candidates for the teacher data group in step S11.

  If there is a predetermined number or more of images including the target image before the crucial instant image, the target image determining unit 31a2 shifts the processing to step S12, and determines a candidate for the deterministic instant image. An instantaneous image is determined, and the acquisition time of the decisive instantaneous image is standardized.

  In the next step S13, the time determination unit 31a1 records the image including the target object image in the series of image groups with the relative time based on the acquisition time of the definitive instantaneous image. In this way, as shown in FIG. 4, a series of images to which information of a relative time difference from the acquisition time of another image is added with reference to the image P5 which is the decisive instantaneous image is used as teacher data as the teacher data recording unit. 31e.

  The input / output modeling unit 31d as the inference model generation unit reads the teacher data created by the population creation unit 31a from the teacher data recording unit 31e by, for example, the method shown in FIG. 2, and obtains an image and a definitive instantaneous image. A learning model (inference model) that has learned the relationship with the time until it is obtained, that is, the network 12a and its setting information are obtained.

  When there is a request from the control unit 11 of the imaging device 20, the learning unit 31 transmits the generated inference model to the imaging device 20 via the communication units 31c and 14. The control unit 11 records the setting information acquired via the communication unit 14 in the setting data recording area 16b, and uses the setting information for setting the network 12a of the inference engine 12. Thus, the inference model generated in the learning unit 31 can be used in the imaging device 20.

  The control unit 11 is provided with a setting control unit 11d, and the setting control unit 11d can control the inference engine 12 to cause the inference using the inference engine 12. That is, when the live view image is acquired by the imaging unit 22, the setting control unit 11d performs inference (hereinafter, referred to as image time inference) to give the live view image to the inference engine 12 and obtain a time until a decisive moment. Let it run. As a result, when the information of the time until the decisive moment is obtained from the inference engine 12, the setting control unit 11d controls the display control unit 11f to display the result of the inference by the inference engine 12 on the display unit 15. It can be displayed on the display screen. That is, in this case, the time until the decisive moment is displayed over the live view image.

  Note that the setting control unit 11d may be configured to be able to present the result of the inference by the inference engine 12 to the user in various ways, without being limited to the display. For example, the setting control unit 11d may present the inference result by voice, or may present the inference result by mechanical control of the driving unit.

  Next, the operation of the embodiment configured as described above will be described with reference to FIGS. 6, 7, and 10 are explanatory diagrams for explaining the operation of the first embodiment. FIGS. 8 and 9 are flowcharts for explaining the operation of the first embodiment. FIG. 8 shows the operation of the imaging device 20, and FIG. 9 shows the operation of the external device 30.

  FIG. 6 shows a state in which a subject is imaged by the imaging device 20 of FIG. Each part of the imaging device 20 in FIG. 1 is housed in a housing 20a in FIG. A display screen 15a constituting the display unit 15 is provided on the back of the housing 20a. A lens (not shown) that forms the optical system 22b is provided on the front surface of the housing 20a, and a shutter button 13a that makes up the operation unit 13 is provided on the upper surface of the housing 20a.

  FIG. 6 shows an example in which a bird 46 that has stopped on a tree branch 45 is photographed as a subject. The user 47 holds the housing 20 a with the right hand 48 and looks at the display screen 15 a of the display unit 15, for example. While holding the bird 46 in the visual field range, the shutter button 13a is pressed down with the finger of the right hand 48 to perform photographing.

  In the present embodiment, a decisive moment that is a photo opportunity is determined using an inference model. That is, the inference engine 12 forms an inference model for inferring a predicted time until a crucial moment comes to an image (live view image). Such an inference model can be generated by the external device 30.

  FIG. 9 shows the operation of the external device 30. In step S41 of FIG. 9, the external device 30 determines whether a learning request has been made, and enters a standby state until a learning request is made. When a learning request is generated, the external device 30 reads the learning data from the external image DB 32 and creates teacher data in step S42. Note that the teacher data creation step of step S42 may be performed according to the flow of FIG.

  When the teacher data is created and recorded in the teacher data recording unit 31e, in step S43, the input / output modeling unit 31d reads the teacher data from the teacher data recording unit 31e, performs learning, and creates an inference model. In the next step S44, the input / output modeling unit 31d sets a practice exercise and verifies the created inference model. In step S45, the input / output modeling unit 31d determines whether or not the reliability of the inference is equal to or more than a predetermined value as a result of the verification using the exercise. If the value is equal to or larger than the predetermined value, the input / output modeling unit 31d determines that the inference model has been correctly generated, and transmits the inference model to the imaging device 20 via the communication unit 31c (step S49). .

  If the reliability is not equal to or more than the predetermined value, the input / output modeling unit 31d shifts the processing from step S45 to step S46, resets the teacher data, and then resets the data at step S47 for a predetermined number of times or more. Is determined. If the reset has not been performed a predetermined number of times or more, the input / output modeling unit 31d returns the process to step S43. If the resetting has been performed a predetermined number of times or more, the input / output modeling unit 31d shifts the processing from step S47 to step S48, and determines that the target object image is a poor image unsuitable for inference. Then, after transmitting the weak image information to the imaging device 20, the process proceeds to step S49.

  On the other hand, the control unit 11 of the imaging device 20 determines whether or not the shooting mode is specified in step S21 of FIG. If the shooting mode is designated, the control unit 11 performs image input and display in step S22. That is, the imaging unit 22 captures an image of a subject, and the control unit 11 captures the captured image from the imaging unit 22 and gives the captured image as a live view image to the display unit 15 to display the captured image as shown in FIG.

  Next, in step S23, the setting control unit 11d causes the inference engine 12 to execute image time inference for displaying the time until the decisive moment. The inference engine 12 uses an inference model implemented by the network 12a to infer the time until each live view image being captured becomes a definitive instantaneous image. The inference engine 12 outputs a result of the inference to the control unit 11. Note that the inference result includes information on the time until the live view image being displayed changes to the definitive instantaneous image and the reliability thereof.

  In step S24, the setting control unit 11d determines whether or not the inference engine 12 has an inference model related to the live view image. For example, when the reliability (reliability) of the inference result from the inference engine 12 is lower than a predetermined first threshold, the setting control unit 11d has an inference model related to the live view image. It may be determined that it has not been performed. Further, the setting control unit 11d may recognize the subject in the live view image by a known recognition process and determine whether or not there is an inference model related to the recognized subject.

  If the setting control unit 11d does not have a related inference model, the process proceeds to step S29. If the setting control unit 11d determines that there is a related inference model, in the next step S25, it displays a display indicating that there is a related inference model in the current live view image.

  Next, the setting control unit 11d determines whether or not the reliability (reliability) of the inference result from the inference engine 12 is sufficiently high, for example, whether or not the reliability is higher than a predetermined second threshold. If the reliability is equal to or greater than the second threshold, the setting control unit 11d shifts the processing to step S27 to display a time difference display with high reliability, and if the reliability is smaller than the second threshold, Then, the process shifts to step S28 to display a display of a time difference width having relatively high reliability.

  FIG. 7 is an explanatory diagram illustrating a captured image displayed on the display screen 15a of the display unit 15. As described above, the user 47 is trying to photograph the bird 46 on the branch 45. In particular, it is assumed that the user 47 desires the photographing, considering the moment when the bird 46 jumps off the branch 45 as the decisive moment. Images P11 to P14 in FIG. 7 indicate live view images at a predetermined time, and the time elapses in the order of the images P11 to P14.

  The image 46a in the image P11 shows the bird 46 stopping on the branch. This image P11 is displayed as a live view image on the display screen 15a. The image P12, which is a live view image acquired after a predetermined time from the image P11, has a display 51 with a circle indicating that an inference model related to the subject in the image P12 exists. The image 46b of the bird 46 in the image P12 shows that the bird 46 is about to fly off. Further, in the image P12, as a result of the image time inference by the inference engine 12, a time difference width display 52b indicating that the time until the decisive moment is 5 seconds to 2 seconds is displayed. The time difference width display 52b displays the inference result with a predetermined width since the reliability of the inference result of the image time inference is not sufficiently high. , 65-84%).

  On the other hand, in the image P13 which is a live view image acquired a predetermined time after the image P12, an image 46c of the bird 46 immediately before flying up is displayed. Further, in the image P13, as a result of the image time inference by the inference engine 12, a time difference display 52c indicating that the time until the decisive moment is one second is displayed. The time difference display 52c shows one inference result having the highest reliability, with the reliability of the inference result of the image time inference being sufficiently high (for example, 85% or more). The time difference display 52c in the image P13 indicates that the bird 46, which is the subject, is likely to fly one second after the start of the display of the time difference display 52c.

  When the user 47 presses the shutter button 13a one second after the display of the time difference display 52c, there is a high possibility that the decisive moment at which the bird 46 will fly can be captured. The control unit 11 determines whether or not a moving image or still image shooting operation has been performed in step S29. When the photographing operation is not performed, the control unit 11 returns the process to step S21. When the photographing operation is performed, the control unit 11 executes the photographing and recording process in step S30, and returns the process to step S21. That is, the control unit 11 causes the recording control unit 11c to record the captured image acquired by the imaging unit 22 in the image data recording area 16a of the recording unit 16. At the time of moving image recording, a moving image file is recorded in the image data recording area 16a at the time of a shooting end operation.

  FIG. 10 is an explanatory diagram for explaining the captured image thus captured, and shows an example of a rec view image displayed on the display screen 15a. The left side of FIG. 10 shows a rec view image 55 displayed on the display screen 15a during continuous shooting. For example, after the user checks the time difference width display 52b and the time difference display 52c, and starts continuous shooting, the rec view image 55 is obtained. The thick frame indicates that one continuously shot image is the decisive instantaneous image 55a.

  The right side of FIG. 10 shows a rec view image 57 displayed on the display screen 15a during single shooting. For example, after the user confirms the time difference display 52c and presses the shutter button 13a after the displayed time, a decisive instantaneous image shown by the rec view image 57 is obtained.

  When determining in step S21 that the shooting mode has not been specified, the control unit 11 shifts the processing to step S31 and determines whether acquisition of an inference model has been specified. If acquisition of an inference model has not been specified, the control unit 11 returns the process to step S21. If it has been specified, the control unit 11 performs setting of a target object and setting of a relearning object in step S32.

  For example, the control unit 11 causes the display control unit 11f to display a menu for setting a dictionary on the display screen 15a, and further displays a target object setting screen and a relearning object setting screen in response to a user operation. In this way, the user may be able to specify the target object and the relearning object. In step S33, the control unit 11 requests the external device 30 to perform a learning request or a re-learning request for the target object or the re-learned object specified by the user.

  In step S34, the control unit 11 receives, via the communication unit 14, the inference model whose reliability has reached a predetermined value or more or the inference model corresponding to the weak image information from the learning unit 31. The control unit 11 sets the received inference model in the inference engine 12 and records the weak image information in the recording unit 16.

  In the description of FIG. 8, an example has been described in which the time difference display or the time difference width display is switched depending on whether or not the reliability of the inference result of the image time inference is sufficiently high. Can be variously considered. For example, the time difference of the inference result may be displayed by numerical values indicating the reliability or color coding, and the degree of shading of the display may be increased as the reliability of the inference result increases. Further, regardless of the reliability, the time difference display or the time difference width display may always be performed.

  In the above description, the photographing operation is manually performed by the user. However, the photographing control unit can also control the photographing operation to be performed automatically at a decisive moment. In addition, when performing continuous shooting, it is possible to control so that the timing of continuous shooting is synchronized with the crucial instant so as to ensure that shooting is performed at the crucial instant.

  As described above, in the present embodiment, machine learning that performs image time inference for estimating a time until reaching a predetermined moment (determinative moment) using an image having time information as learning data is realized. By applying the inference model obtained by this machine learning to, for example, an imaging device, image time inference is performed on a live view image that changes from moment to moment, and, for example, the arrival time until a decisive moment when a bird takes off is predicted. And can be presented. The user can easily take an image of the decisive moment when the bird jumps, for example, by operating the shutter button in consideration of the presented arrival time. In addition, an image having time information used as learning data can be obtained very easily, and teacher data can be obtained from the learning data by relatively simple processing, thereby enabling image time inference. Inference models can be easily created.

(Second embodiment)
FIG. 11 is a flowchart showing an operation flow employed in the second embodiment of the present invention. The hardware configuration of this embodiment is the same as that of the first embodiment. In FIG. 11, the same steps as those in FIG. 9 are denoted by the same reference numerals, and description thereof will be omitted.

  In the first embodiment, an image having time information is used as learning data, and teacher data is created using the learning data. An example of generating and using an inference model for performing inference has been described. On the other hand, in the present embodiment, an image having time information is used as learning data, and a set of a plurality of images having a predetermined time interval is created as teacher data by using the learning data. This is an example of generating and using an inference model for performing image image inference for predicting an image and an image after a predetermined time.

  The flow of FIG. 11 differs from the flow of FIG. 5 in the method of creating teacher data. That is, the population generating unit 31a of the external device 30 acquires a continuous image group of similar objects from the external image DB 32 or the like in step S51 of FIG. In step S52, the mother set creating unit 31a gives two images having a specific time difference to the network as teacher data to be trained.

  FIG. 12 is an explanatory diagram showing an example of a continuous image group taken into the mother set creation unit 31a from the external image DB 32. As in FIG. 4, the images P21 to P29 are images obtained by photographing a series of states before and after the bird takes off and arranged in chronological order. Note that these images P21 to P29 may be obtained by using the continuous shooting function or the moving image function, or may be obtained by single shooting at predetermined time intervals.

  The population generating unit 31a selects a pair of two images before and after a predetermined time as teacher data by the time determining unit 31a1 and the object image determining unit 31a2. For example, the arrow in FIG. 12 indicates an image after N seconds, and indicates that the images P21 and P24, the images P22 and P25, the images P23 and P26, and the images P24 and P27 are pairs. The mother set creation unit 31a records the created teacher data in the teacher data recording unit 31e.

  FIG. 13 is an explanatory diagram for explaining a method of generating the network 12a by the same description method as that of FIG. In FIG. 13, a large amount of learning data set input to the predetermined network N1 is a set of images read from the teacher data recording unit 31e. In the present embodiment, when an image is provided to network N1 as an input, a network design is determined such that an image after N seconds is obtained as an output. Information on the network design determined in this way is transmitted from the communication unit 31c to the imaging device 20, and is recorded as setting information in the setting data recording area 16b by the setting control unit 11d.

  Furthermore, in the present embodiment, when the learning unit 31 determines that the reliability is equal to or more than the predetermined value in step S45, in the next step S53, the output image estimated from the input image, that is, It is determined which of the images acquired after a predetermined time has elapsed from the acquisition time of the input image is output. As will be described later, this image is recorded on a recording medium (not shown) in order to use it as a representative image used for composite display. The learning unit 31 also transmits the representative image to the imaging device 20 when transmitting the network design information. The recording control unit 11c of the imaging device 20 records the representative image in the image data recording area 16a.

  Next, an operation of the imaging apparatus 20 according to the embodiment configured as described above will be described with reference to FIG. FIG. 14 is an explanatory diagram illustrating a display example of an image displayed on the display screen of the display unit 15.

  In the present embodiment, the inference engine 12 configures an inference model that performs the above-described image image inference that outputs a predicted image after a predetermined time has elapsed with respect to a predetermined image input. Further, the control unit 11 of the imaging device 20 causes the inference engine 12 to execute image image inference instead of steps S25 to S28 in FIG. 8 and performs display based on the inference result.

  Now, as in FIG. 6, it is assumed that the user 47 captures an image of the bird 46 stopping on the branch 45. Each of the images P31a to P31d in FIG. 14 shows a live view image immediately before the bird 46 takes off. The setting control unit 11d supplies the live view image from the imaging unit 22 to the inference engine 12, and executes the image image inference. The inference engine 12 predicts, as a result of the image image inference, an image that will be captured a predetermined time after the capturing time of the input live view image, and outputs the prediction result to the setting control unit 11d.

  The setting control unit 11d reads the representative image stored in the image data recording area 16a based on the prediction result, and gives the representative image to the display control unit 11f. In this way, the display control unit 11f superimposes and displays the representative image that will be captured after a predetermined time on the current live view image. The image P31a in FIG. 14 shows one display example in this case. In the image P31a, an image 2 seconds later is displayed outside the image portion 61 immediately before the bird 46 included in the live view image jumps up. The predicted representative image 62a and the predicted representative image 62b three seconds later are displayed. In addition, the display control unit 11f determines that the time at which these images will be acquired is two seconds or three seconds after the preview images are acquired near the images 62a and 63a. Are displayed on the display 62b and 63b.

  As described above, the representative image is an image determined and recorded by the external device 30, and is an image transferred from the external device 30 to the imaging device 20. Therefore, there is a possibility that the representative image does not always exist. Therefore, in this case, the image portion 61 may be copied and displayed at the display position of the representative image. An image P31b in FIG. 14 shows a display example in this case. In the image P31b, an image portion 61 immediately before the bird 46 included in the live view image immediately before jumping out is predicted as an image 2 seconds later. An image 64a generated by copying the image portion 61 instead of the representative image is displayed. In addition, the display control unit 11f displays a time display 64b near the image 64a, indicating that the time at which the representative image will be acquired is two seconds later, based on the acquisition time of the preview image. ing.

  Further, an image P31c in FIG. 14 illustrates an example in which the display control unit 11f displays time displays 65b and 66b instead of the time displays 62b and 63b of the image P31a. Time displays 65b and 66b imitating the shape of the hands of a clock and the display of arrows indicate that the expected acquisition times of the representative images 62a and 63a are respectively 2 seconds or 3 seconds after the shooting time of the preview image. I have.

  An image P31d in FIG. 14 illustrates an example in which a plurality of representative images that are likely to be captured at the same time are displayed simultaneously. In the above description of step S53 in FIG. 11, an example in which only one representative image is selected has been described, but a plurality of images may be selected and recorded as the representative image. In this case, the setting control unit 11d records the plurality of representative images transferred from the external device 30 in the image data recording area 16a.

  The setting control unit 11d reads a representative image recorded in the image data recording area 16a based on the prediction result of the inference engine 12, and supplies the representative image to the display control unit 11f. When there are a plurality of representative images, the display control unit 11f superimposes and displays the plurality of representative images. The image P31d shows one display example in this case. In the image P31d, an image 2 seconds later is predicted as an image outside the image portion 61 immediately before the bird 46 included in the live view image jumps up. The representative images 67a to 67c are displayed. Further, the display control unit 11f displays a time display near these images 67a to 67c indicating that the time at which these images will be obtained is two seconds later, based on the time at which the preview images are obtained. 67d is displayed.

  As described above, in the present embodiment, machine learning that performs image image inference for predicting an image after a predetermined time is realized using an image having time information as learning data. By applying the inference model obtained by this machine learning to, for example, an imaging device, image inference is performed on a live view image that changes every moment, and for example, how a bird is photographed after a predetermined time is determined. Can be predicted and presented. The user can easily take a picture of a bird flying, for example, by performing a shooting operation in consideration of the presented image. In addition, an image having time information used as learning data can be obtained extremely easily. Teacher data can be obtained from the learning data by relatively simple processing, and image image inference can be performed. Inference models can be easily created.

(Third embodiment)
FIG. 15 is a flowchart showing an operation flow employed in the third embodiment of the present invention. The hardware configuration of this embodiment is the same as that of the first embodiment. 15, the same steps as those in FIG. 11 are denoted by the same reference numerals, and description thereof will be omitted.

  In the second embodiment, an example of generating an inference model of image image inference has been described. In the present embodiment, an image having time information is used as learning data, and a predetermined time is determined using the learning data. This is an example of generating and using an inference model for performing image position inference for estimating an image position after a predetermined time from each image by creating a set of a plurality of images having intervals and their position differences as teacher data. .

  The flow of FIG. 15 differs from that of FIG. 11 in that the method of generating teacher data employs step S61 instead of step S52 and omits the processing of step S53. It is assumed that, for example, a continuous image group shown in FIG. 12 and the like is recorded in the external image DB 32 as in the second embodiment.

  The population generating unit 31a uses a pair of two images before and after a predetermined time as teacher data by the time determining unit 31a1 and the object image determining unit 31a2, as in the second embodiment. For example, it indicates that the images P21 and P24, the images P22 and P25, the images P23 and P26, and the images P24 and P27 in FIG. The population creating unit 31a obtains information on the position of the target object image in the image. For example, when the target object image is the image of the bird in FIG. 12, the position of the face of the bird or the position of the tip of the tip of the brim may be obtained. Then, the population generating unit 31a obtains the difference between the positions of the respective object images in the set of images, and determines the position of the object in an image acquired earlier in time (hereinafter referred to as a previous image) as a reference. Next, the position difference of the target object in an image acquired later (hereinafter, referred to as a later image) is obtained. The population creating unit 31a records the relationship between the previous image and the position difference as teacher data in the teacher data recording unit 31e.

  The method of generating the network 12a in this case is the same as that of the second embodiment, and the position difference of the post-image based on the position of the previous image is obtained instead of the output image in FIG. , The network design is decided. That is, in the present embodiment, an inference model for inputting the previous image and performing image position inference for predicting the position of the subsequent image is obtained. Information on the network design determined in this way is transmitted from the communication unit 31c to the imaging device 20, and is recorded as setting information in the setting data recording area 16b by the setting control unit 11d.

  Next, the operation of the imaging apparatus 20 according to the embodiment configured as described above will be described with reference to FIGS. FIG. 16 is a flowchart showing the control of the control unit 11 of the imaging device 20, and FIG. 17 is an explanatory diagram showing a display example of an image displayed on the display screen of the display unit 15. The control flow of the control unit 11 shown in FIG. 16 is substantially the same as that of FIG. 8 except that steps S65 and S66 are employed instead of steps S27 and S28 in FIG.

  In the present embodiment, the inference engine 12 constitutes an inference model that performs the above-described image position inference that outputs a predicted position after a predetermined time with respect to a predetermined image input.

  Now, as in FIG. 6, it is assumed that the user 47 captures an image of the bird 46 stopping on the branch 45. Each of the images P32a to P32d in FIG. 17 shows a live view image immediately before the bird 46 takes off. The setting control unit 11d supplies the live view image from the imaging unit 22 to the inference engine 12, and executes the image position inference. As a result of the image position inference, the inference engine 12 is imaged after a predetermined time from the photographing time of the live view image with reference to the position difference from the input image, that is, the position of the target object image in the input live view image. The position of the target object image in the image that is likely to be predicted is predicted, and the prediction result is output to the setting control unit 11d.

  If the control unit 11 of the imaging device 20 determines in step S26 of FIG. 16 that the reliability of the prediction result by the inference engine 12 is sufficiently high, the control unit 11 performs a predetermined time based on the inference result in the next step S64. A display is performed to indicate the position of the target object image in the image that will be captured later.

  The image P32a in FIG. 17 illustrates one display example in this case, and the display control unit 11f includes, in the image P32a, an image (object image) portion 61 immediately before the bird 46 included in the live view image rises. , A position display 64a indicating a position predicted as an image position two seconds after the target object image is displayed. The position display 64a is an image obtained by copying the image portion 61. In addition, the display control unit 11f displays a time display 64b near the image 64a, indicating that the time at which this image will be obtained is two seconds later, based on the acquisition time of the preview image. ing.

  An image P32b in FIG. 17 illustrates an example in which the display control unit 11f displays a time display 71a instead of the time display 64a of the image P32a. In addition, the display control unit 11f displays a time display 71b near the image 71a, indicating that the time at which this image will be obtained is two seconds later, based on the acquisition time of the preview image. ing. The time display 71a indicates the range of the position in the image that the bird 46 will reach two seconds after the shooting time of the preview image due to the curved shape.

  If the reliability of the inference is not sufficiently high in step S45, the control unit 11 displays a range of positions where the reliability is relatively high in step S66. An image P32c in FIG. 17 shows a display example in this case, and the display control unit 11f displays a position range display 72a by two curves in the image P32c, and displays a range in which the subsequent image exists two seconds later. Is shown. In addition, the display control unit 11f displays a time display 72b near the position range display 72a, indicating that the time at which the preview image will be obtained is two seconds later, based on the time at which the preview image was obtained. are doing. When the reliability of the inference result of the image position inference is not sufficiently high, the position range display 72a is the closest among a plurality of inference results of relatively high reliability (for example, 65 to 84%). It shows the range between the position and the farthest position.

  The image P32d shows another display example in step S66. The display control unit 11f displays a position range display 73a by a circle in the image P32d to indicate a range where the subsequent image exists two seconds later. In addition, the display control unit 11f displays a time display 73b near the position range display 73a, which indicates that the time at which the preview image will be obtained is two seconds later, based on the acquisition time of the preview image. are doing.

  As described above, in the present embodiment, machine learning that performs image position inference for predicting the position of an image after a predetermined time is realized using an image having time information as learning data. By applying the inference model obtained by this machine learning to, for example, an imaging device, image position inference is performed on a live view image that changes from time to time, and for example, a position where a bird is photographed after a predetermined time is determined. The position can be predicted and presented. The user can take a picture in which the birds fly easily by performing, for example, a photographing operation in consideration of the presented position. Further, an image having time information used as learning data can be obtained very easily, and teacher data can be obtained from the learning data by relatively simple processing, and image position inference can be performed. Inference models can be easily created.

(Fourth embodiment)
FIG. 18 to FIG. 20 are explanatory diagrams for explaining the fourth embodiment of the present invention. The hardware configuration of this embodiment is the same as that of the first embodiment. FIG. 18 is an explanatory diagram showing a shooting scene in the present embodiment.

  FIG. 18 shows a situation in which the user 47 takes an image of a bird 46 stopping on a tree branch 45. A river 81 flows between the user 47 and the tree, and a fish 49 is swimming in the river 81. It is conceivable that the bird 46 stopping on the branch 45 jumps up in the sky, jumps to a branch of an adjacent tree, or glides toward a fish 49 in the river 81. In FIG. 18, these states are indicated by reference numerals 46h, 46i, and 46j, respectively. In this case, if it is possible to predict in which direction the bird 46 will fly in advance, it may be possible to effectively shoot the bird 46. The example in FIG. 18 indicates that the probability that the bird 46 will be in the state of reference numeral 46h or 46i is 15%, and the probability that the bird 46 will be in the state of reference numeral 46j is 70%. The present embodiment enables such prediction.

  In the second embodiment, an image having time information is used as learning data, and a set of a plurality of images having a predetermined time interval is created as teacher data using the learning data. This is an example in which an inference model for performing image image inference for estimating an image (previous image) and an image after a predetermined time (post image) is generated and used. This embodiment is an example of generating and using an inference model for performing image image direction inference for estimating a moving direction with respect to a preceding image and a succeeding image after a plurality of times.

  In the present embodiment, for example, the continuous image group shown in FIG. 12 can be used as learning data. The mother set creation unit 31a generates teacher data in which a previous image and a plurality of subsequent images after a plurality of predetermined times from the acquisition time of the previous image are generated and stored in the teacher data recording unit 31e. The input / output modeling unit 31d obtains a rear image corresponding to the front image and the moving direction thereof at each predetermined time, and selects a representative image to be used as the rear image. In this case, a plurality of representative images are selected according to the moving direction. For example, a representative image may be selected every 15 degrees in the moving direction. The network design information and the representative image thus generated are transmitted to the imaging device 20, and the recording control unit 11c of the imaging device 20 records the network design information in the setting data recording area 16b, and stores the representative image in the image data recording area. 16a.

  Next, an embodiment configured as described above will be described with reference to FIGS. FIG. 19 and FIG. 20 are explanatory diagrams showing display examples of images displayed on the display screen of the display unit 15.

  In the present embodiment, the inference engine 12 constitutes an inference model that performs the above-described image image direction inference that outputs a predicted image and a moving direction thereof after a plurality of predetermined times with respect to a predetermined image input. The user 47 is trying to photograph the bird 46 after flying off the branch 45. Images P41 to P44 in FIG. 19 show live view images at a predetermined time, and the time elapses in the order of images P41 to P44.

  The image 46a in the image P41 shows the bird 46 stopping on the branch. This image P41 is displayed as a live view image on the display screen 15a. In the image P42 which is a live view image acquired after a predetermined time from the image P41, a display 51 of a circle indicating that an inference model related to the subject in the image P42 is displayed. The image 46b of the bird 46 in the image P42 shows that the bird 46 is about to fly off. For example, it is assumed that the reliability of the image image direction inference at this point is not sufficiently high. In this case, in the image P42, as a result of the image image direction inference by the inference engine 12, the display control unit 11f displays the time display 88b indicating that the image is predicted after 5 seconds to 2 seconds and the movement of the bird 46. A display indicating the prediction of the direction is displayed.

  In the example of FIG. 19, in the image P42, the probability display 85hp indicating that the probability that the bird 46 will jump upward is 15%, the representative image 85h in that case, and the possibility that the bird 46 jumps to the next branch are 15%. % And a representative image 85i in that case, a probability display 85jp indicating that the possibility of gliding horizontally or downward is 70%, and a representative image 85j in that case. The probabilities indicated by the respective probability displays are obtained based on the reliability values in the respective directions obtained as a result of the inference.

  Furthermore, in the image P43 which is a live view image acquired after a predetermined time from the image P42, a display 51 of a circle indicating that an inference model related to the subject in the image P43 is displayed. An image 46c of the bird 46 in the image P43 shows a state immediately before the bird 46 takes off. For example, it is assumed that the reliability of the image image direction inference at this point is sufficiently high. In this case, as a result of the image image direction inference by the inference engine 12, a display indicating the moving direction of the bird 46 after one second is displayed in the image P43 by the display control unit 11f.

  In the example of FIG. 19, in the image P43, the bird 46 can glide horizontally or downward, with the probability display 86ip indicating that the probability of jumping to the next branch is 5%, the representative image 86i in that case, and the like. A probability display 86jp indicating that the sex is 95% and a representative image 86j in that case are included. A time display 88c indicating that the prediction is one second after the present time is also displayed.

  By checking the image P43 displayed on the display screen 15a of the display unit 15, the user 47 can predict that the bird 46 will glide in a substantially horizontal direction after jumping off the branch 45.

  For example, the user 47 can relatively easily keep the bird 46 in the shooting range by checking the image P43 and estimating the moving direction of the bird 46. As a result, the user 47 can capture a desired decisive moment, that is, a moment at which the bird 46 captures the fish 49 or the like.

  FIG. 20 shows a display on the display screen 15a in this case, in which an image 49p of a fish and an image 46p of a bird holding the fish are displayed.

  Note that the image P44 in FIG. 19 shows a live view image one second after the shooting time of the image P43, and the image 46d of the actual bird 46, the representative image 87j of the prediction result, and the probability of the moving direction thereof are 100%. Is displayed.

  As described above, in the present embodiment, machine learning is performed, in which an image having time information is used as learning data to perform image image direction inference for predicting an image after a plurality of predetermined times and a moving direction thereof. By applying the inference model obtained by this machine learning to, for example, an imaging device, image image direction inference is performed on a live view image that changes every moment, for example, in which direction a bird is photographed after a predetermined time. Can be predicted and presented. The user can easily take a picture of a bird flying, for example, by performing a shooting operation in consideration of the presented image. In addition, an image having time information used as learning data can be obtained extremely easily. Teacher data can be obtained from the learning data by relatively simple processing, and image image direction inference can be performed. Inference models that enable it can be easily created.

  In each of the above embodiments, only an example in which a bird is assumed as the target object has been described. However, any target object may be used, and the decisive moment can be predicted from the preceding and succeeding images by learning. Anything may be used as long as it is appropriate. For example, the moment when a bird enters the water surface, the moment when a fish jumps from the water surface, or the moment when a cat or dog turns around may be predicted.

  In addition, the moment when the milk crown occurs is not limited to the animal, and the moment when the fireworks are opened may be predicted. Further, the moment of impact of a swing such as golf, which is relatively easy to predict, may be predicted. Further, the moment of cell division, cleavage, emergence, hatching, etc. may be predicted. It is relatively difficult to identify the moment of cell division, and it would be extremely useful to be able to predict the moment of division in minutes. Further, for example, the moment when the fire is stopped can be predicted by imaging the state of cooking.

Further, in each of the above-described embodiments, an example has been described in which time, image, position, and direction are inferred by an image, but these inferences can be made based on sound. For example, courtship behavior can be predicted from a call for animal courtship behavior. In addition, it is also possible to predict the instant of a squeak until a courtship action is performed from the image of the animal, that is, it is also possible to predict the timing of sound generation from the image.
As described above, the concept of the present invention can be applied to all information that can be obtained, such as audio and images, and learning using both data as well as prediction from one to the other can be made comprehensively. May go. For example, learning may be performed by inserting fragmentary information of sound at the corresponding acquisition time for each frame of an image. In addition to courtship behavior, there are decisive moments such as spawning, emergence and hatching. In addition, it is also possible to develop a medical treatment that predicts a disease that a patient will subsequently develop by making inferences based on heart sounds, breath sounds, intestinal peristaltic sounds, and the like. The state of wheezing and breathing, such as asthma, changes as the condition worsens, so it is easy to find them early. It can be useful for early treatment of infants and people with disabilities that cannot be expressed in words.

  In the above embodiment, the imaging apparatus requests the external device to create and transfer the inference model. However, the creation of the inference model may be performed in any apparatus, for example, using a computer on a cloud. May be.

  In the above embodiment, the description has been made using a digital camera as a device for imaging. However, the camera may be a digital single-lens reflex camera or a compact digital camera, and a video camera, a movie camera such as a movie camera. Alternatively, a camera built in a portable information terminal (PDA: Personal Digital Assist) such as a mobile phone or a smartphone may be used. Further, the imaging unit may be separate from the imaging device.

  The present invention is not limited to the above embodiments as they are, and can be embodied by modifying the constituent elements in an implementation stage without departing from the scope of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components of all the components shown in the embodiment may be deleted. Further, components of different embodiments may be appropriately combined.

  It should be noted that even if the operation flow in the claims, the specification, and the drawings is described using “first”, “next”, etc. for convenience, it is essential that the operation be performed in this order. It does not mean. In addition, it is needless to say that each step constituting the operation flow can be omitted as appropriate for portions that do not affect the essence of the invention.

  Of the techniques described here, the control mainly described in the flowchart can be set by a program in many cases, and may be stored in a recording medium or a recording unit. The recording method on the recording medium and the recording unit may be recorded when the product is shipped, a distributed recording medium may be used, or the recording medium may be downloaded via the Internet.

In the embodiments, the part described as a “unit” (section or unit) may be configured by combining a dedicated circuit or a plurality of general-purpose circuits, and if necessary, may be configured by a pre-programmed software. And a processor such as a CPU or a sequencer such as an FPGA that operates according to the above. It is also possible to design such that some or all of the control is undertaken by an external device. In this case, a wired or wireless communication circuit is interposed. The communication may be performed by Bluetooth, WiFi, a telephone line, or the like, and may be performed by USB or the like. A dedicated circuit, a general-purpose circuit, and a control unit may be integrally configured as an ASIC. The moving unit and the like are constituted by various actuators and, if necessary, a connecting mechanism for movement, and the actuators are operated by a driver circuit. This drive circuit is also controlled by a microcomputer or ASIC according to a specific program. For such control, detailed correction, adjustment, and the like may be performed based on information output from various sensors and their peripheral circuits. Also, the embodiment in which the judgment is made based on the learning result determined by the artificial intelligence using the words inference model or learned model has been described, but this can be replaced with a simple flowchart, conditional branching, or a numerical judgment involving an operation. There are cases. In addition, learning of machine learning may be performed in the imaging device by improving the arithmetic performance of the control circuit of the camera or by narrowing down to a specific situation or target.

[Appendix 1]
A time determination unit for obtaining an image after a predetermined time for each image in a series of images having time information based on the shooting time,
For each image in the series of images, an object image determination unit that detects an image state of the specific object in each image after a predetermined time,
A teacher data creation device, comprising: a controller that sets the respective images and data of an image state of a specific object obtained for the respective images after a predetermined time period as teacher data.

[Appendix 2]
3. The teacher data creating apparatus according to claim 2, wherein the control unit selects an image similar to an image state of the specific target object after a predetermined time from the series of images as a representative image.

[Appendix 3]
Inference model generation for generating an inference model for inferring an image state of a predetermined object after a predetermined time from an input image by machine learning using the teacher data created by the teacher data creation device described in Additional Item 1. A learning device comprising a unit.

[Appendix 4]
An inference engine that implements the inference model generated by the learning device according to claim 3;
An imaging unit;
An image pickup apparatus, comprising: a setting control unit that supplies an image captured by the image capturing unit to the inference engine to obtain an inference result of an image state of the predetermined object in the captured image after a predetermined time. .

[Appendix 5]
A time determination unit for obtaining an image after a predetermined time for each image in a series of images having time information based on the shooting time,
For each image in the series of images, an object image determination unit that detects an image position of the specific object in each image after a predetermined time,
A teacher data creating apparatus, comprising: a controller that sets the respective images and data of the image position of a specific object obtained for the respective images after a predetermined time and sets the data as teacher data.

[Appendix 6]
Inference model generation for generating an inference model for inferring an image position of a predetermined object after a predetermined time from an input image by machine learning using the teacher data generated by the teacher data generation device described in Additional Item 5. A learning device comprising a unit.

[Appendix 7]
An inference engine that implements the inference model generated by the learning device of claim 6;
An imaging unit;
An image pickup apparatus, comprising: a setting control unit that supplies an image captured by the imaging unit to the inference engine and obtains an inference result of an image position of the predetermined object in the captured image after a predetermined time. .

[Appendix 8]
A time determination unit that determines an image after a plurality of predetermined times for each image in a series of images having time information based on the shooting time,
For each image in the series of images, an object image determination unit that detects an image position and a moving direction of a specific object in each image after a plurality of predetermined times,
Teacher data comprising: a set of each of the images and data of the image position and the moving direction of the specific object obtained after a plurality of predetermined times obtained for each of the images as teacher data. Creating device.

[Appendix 9]
An inference model that infers an image position and a moving direction of a predetermined object after a plurality of predetermined times from an input image by machine learning using the teacher data created by the teacher data creation device described in Additional Item 8. A learning device, comprising: an inference model generation unit that generates an inference model.

[Appendix 10]
An inference engine that implements the inference model generated by the learning device of claim 9;
An imaging unit;
A setting control unit that gives an image captured by the imaging unit to the inference engine, and obtains an inference result of an image position and a moving direction of the predetermined object in the captured image after a plurality of predetermined times. Characteristic imaging device.

    11 control unit, 11a imaging control unit, 11b image processing unit, 11c recording control unit, 11d setting control unit, 11e communication control unit, 11f display control unit, 12 inference engine, 12a network 13: operation unit, 14, 31b, 31c, 33: communication unit, 15: display unit, 16: recording unit, 16a: image data recording area, 16b: setting data recording area, 20: imaging device, 22: imaging unit, 22a: imaging device, 22b: optical system, 30: external device, 31: learning unit, 31a: population creation unit, 31a1: time determination unit, 31a2: target image determination unit, 31d: input / output modeling unit, 31e ... teacher data recording unit, 31f ... display unit, 32 ... external image DB.

Claims (11)

From a series of images having time information based on the shooting time, a target image determination unit that detects a specific state image that is an image in a specific state of a specific target,
For each image of the series of images, the shooting time of each of the images, and a time determination unit that determines a time difference between the shooting times of the images including the specific state image of the series of images,
A teacher data creation device, comprising: a control unit that sets each image and data of the time difference obtained for each image as teacher data. An inference model for generating an inference model for inferring a time when a predetermined target is in the specific state from an input image by machine learning using the teacher data created by the teacher data creation device according to claim 1. A learning device comprising a generation unit. An inference engine that implements an inference model generated by the learning device according to claim 2;
An imaging unit;
And a setting control unit that gives an image captured by the image capturing unit to the inference engine and obtains an inference result of a time until the predetermined target in the captured image reaches the specific state. Imaging device. The imaging apparatus according to claim 3, further comprising a display control unit that performs display control for displaying the time inference result on a display unit. The captured image is a live view image,
The imaging apparatus according to claim 4, wherein the display control unit displays the inference result of the time on the live image displayed on the display unit. The setting control unit acquires reliability information of the inference result together with the inference result of the time,
The imaging apparatus according to claim 3, wherein the display control unit changes a display mode of the time inference result based on the reliability information. The display control unit displays the time when the reliability of the inference result is equal to or more than a predetermined threshold, and displays the time width when the reliability of the inference result is smaller than the predetermined threshold.
The imaging device according to claim 6, wherein: From a series of images having time information based on the shooting time, a detection step of detecting a specific state image that is an image in a specific state of a specific target,
For each image of the series of images, the shooting time of each of the images, a step of determining the time difference between the shooting time of the image including the specific state image of the series of images,
Generating a teacher data by combining each of the images and the data of the time difference obtained for each of the images as a set of teacher data. 9. The teacher data creating method according to claim 8, wherein the detecting step detects the specific object by manual operation or recognition processing, and detects the specific state by image analysis processing. The generating step excludes the series of images from the teacher data when the number of images including the specific object is less than a predetermined number, and excludes the images not including the specific object from the teacher data. The method according to claim 8, wherein the teacher data is created. On the computer,
From a series of images having time information based on the shooting time, a detection step of detecting a specific state image that is an image in a specific state of a specific target,
For each image of the series of images, the shooting time of each of the images, a step of determining the time difference between the shooting time of the image including the specific state image of the series of images,
Generating a teacher data by combining each image and the data of the time difference obtained for each image as teacher data.

Images