JP2019118098A - Imaging device, control method therefor, program, and storage medium - Google Patents

Abstract

To provide an imaging device capable of acquiring a video suitable for a user without performing a special operation by the user.SOLUTION: An imaging device includes an acquisition unit that acquires data related to a photographed image photographed by a photographing unit, a learning unit that learns conditions of an image preferred by the user on the basis of teacher data acquired by the acquiring unit, a control circuit that determines the automatic photographing by the photographing unit on the basis of the condition learned by the learning unit, and a registration unit that registers, as the teacher data, the data acquired by the acquisition unit for the photographed image for learning captured continuously to the photographing performed by the user's instruction.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an automatic imaging technique in an imaging device.

  In still image / moving image shooting with an imaging device such as a camera, it is common for a user to decide an object to be shot through a finder or the like, check the shooting situation by himself and adjust the framing of the shot image to shoot an image. is there. Such an imaging apparatus is provided with a function of detecting a user's operation error and notifying the user, or detecting an external environment and notifying the user when it is not suitable for photographing. In addition, there has conventionally been a mechanism for controlling the camera to be in a state suitable for photographing.

  There is a life log camera that shoots regularly and continuously without the user giving a shooting instruction to an imaging apparatus that performs shooting by such a user operation (Japanese Patent Application Laid-Open No. 2008-112118). The life log camera is used in a state of being attached to the user's body by a strap or the like, and records a scene that the user sees in daily life as an image at regular time intervals. The life log camera does not shoot at a timing intended by the user, for example, but shoots at fixed time intervals, so that unexpected moments that are not normally shot can be left as images.

Japanese Patent Publication No. 2016-536868 Publication JP, 2004-354251, A

  However, when the user wears the life log camera, if automatic shooting is performed periodically, there is a case where a video which is not preferred by the user is acquired, and it is not possible to acquire a video at the moment when it is desired to obtain.

  In addition, even if the life log camera has a learning function and can learn and automatically shoot the moment that the user really wants to shoot, a large amount of teacher data is required for the learning. In Patent Document 2, in a defect inspection apparatus for inspecting the presence or absence of a defect in a test object using a neural network, an artificial defect image of the test object is created by image processing to compensate for the lack of a learning pattern. Technology is disclosed. However, unlike the defect inspection in which the type of the object is limited, in the life log camera, the type of the object is infinite, and it is difficult to compensate for the lack of the learning pattern by image processing.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an imaging device capable of acquiring a video suitable for the user without the user performing a special operation.

  An imaging apparatus according to the present invention includes an acquisition unit that acquires data related to a photographed image captured by an imaging unit, and a learning unit that learns conditions of an image preferred by a user based on teacher data acquired by the acquisition unit. The control means for determining the automatic photographing by the photographing means based on the condition learned by the learning means, and the acquisition of the learning photographed image continuously photographed in the photographing performed by the user's instruction And registration means for registering data acquired by the means as the teacher data.

  According to the present invention, it is possible to provide an imaging device capable of acquiring a video suitable for the user without the user performing a special operation.

It is a figure which shows an imaging device typically. It is a figure showing composition of an imaging device. It is a figure which shows the structure of an imaging device and an external apparatus. It is a figure which shows the structure of an external apparatus. It is a figure which shows the structure of an imaging device and an external apparatus. It is a figure which shows the structure of an external apparatus. It is a flowchart explaining a 1st control circuit. It is a flowchart explaining a 2nd control circuit. 5 is a flowchart illustrating shooting mode processing. It is a figure explaining a neural network. It is a figure for demonstrating area division in a photography picture. It is a flowchart explaining a learning mode determination. It is a flowchart explaining a learning process. It is a figure explaining the display process which concerns on this embodiment.

First Embodiment
<Configuration of Imaging Device>
FIG. 1 is a view schematically showing an imaging device of the first embodiment.

  The imaging device 101 illustrated in FIG. 1A is provided with an operation member (hereinafter referred to as a power button, which may be a tap, flick, or swipe operation to a touch panel) that can operate the power switch. There is. A lens barrel 102, which is a housing including an imaging lens group for capturing an image and an imaging element, is attached to the imaging apparatus 101, and is provided with a rotation mechanism capable of rotationally driving the lens barrel 102 with respect to the fixed portion 103. The tilt rotation unit 104 is a motor drive mechanism capable of rotating the lens barrel 102 in the pitch direction shown in FIG. 1B, and the pan rotation unit 105 is a motor drive mechanism capable of rotating the lens barrel 102 in the yaw direction. Thus, the lens barrel 102 can rotate in one or more axes. FIG. 1B is an axis definition at the fixed portion 103 position. Both the angular velocity meter 106 and the accelerometer 107 are mounted on the fixing unit 103 of the imaging device 101. The vibration of the imaging apparatus 101 is detected based on the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit and the pan rotation unit are rotationally driven based on the detected swing angle. Thereby, the shake of the lens barrel 102 which is the movable portion is corrected, and the tilt is corrected.

  FIG. 2 is a block diagram showing the configuration of the imaging device of the present embodiment.

  In FIG. 2, the first control circuit 223 includes a processor (for example, a CPU, a GPU, a microprocessor, an MPU, and the like) and a memory (for example, a DRAM, an SRAM, and the like). These units execute various processes to control each block of the imaging apparatus 101 or to control data transfer between the blocks. A nonvolatile memory (EEPROM) 216 is an electrically erasable and recordable memory, and stores constants, programs and the like for the operation of the first control circuit 223.

  In FIG. 2, the zoom unit 201 includes a zoom lens that performs magnification change. The zoom drive control circuit 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens that performs focus adjustment. The focus drive control circuit 204 drives and controls the focus unit 203.

  The imaging unit 206 includes an imaging element and an A / D converter, and the imaging element receives light incident through each lens group, and outputs information of charge corresponding to the light amount to the image processing circuit 207 as analog image data. An image processing circuit 207 is an arithmetic circuit equipped with a plurality of ALUs (Arithmetic and Logic Units), and an image such as distortion correction, white balance adjustment, or color interpolation processing for digital image data output by A / D conversion. Apply processing and output digital image data after application. The digital image data output from the image processing circuit 207 is converted into a recording format such as JPEG format by the image recording circuit 208, and is transmitted to the memory 215 or a video output circuit 217 described later.

  The lens barrel rotation drive circuit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

  The device shake detection circuit 209 is mounted with, for example, an angular velocity meter (gyro sensor) 106 that detects angular velocity in the direction of three axes of the imaging device 101 and an accelerometer (acceleration sensor) 107 that detects an acceleration in the direction of three axes of the device. . The device shake detection circuit 209 calculates the rotation angle of the device, the shift amount of the device, and the like based on the detected signal.

  The audio input circuit 213 acquires an audio signal around the imaging device 101 from a microphone provided in the imaging device 101, performs analog-to-digital conversion, and transmits the signal to the audio processing circuit 214. The audio processing circuit 214 performs processing relating to audio such as optimization processing of the input digital audio signal. Then, the audio signal processed by the audio processing circuit 214 is transmitted to the memory 215 by the first control circuit 223. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing circuit 207 and the audio processing circuit 214.

  An image processing circuit 207 and an audio processing circuit 214 read out an image signal and an audio signal temporarily stored in the memory 215, encode the image signal, encode an audio signal, and the like, and generate a compressed image signal and a compressed audio signal. Generate The first control circuit 223 transmits the compressed image signal and the compressed audio signal to the recording and reproducing circuit 220.

  The recording / reproducing circuit 220 records, on the recording medium 221, the compressed image signal and the compressed audio signal generated by the image processing circuit 207 and the audio processing circuit 214, and other control data related to photographing. When the audio signal is not compressed and encoded, the first control circuit 223 sends the audio signal generated by the audio processing circuit 214 and the compressed image signal generated by the image processing circuit 207 to the recording and reproducing circuit 220. It transmits and makes the recording medium 221 record.

  The recording medium 221 may be a recording medium built in the imaging device 101 or a removable recording medium. The recording medium 221 can record various data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the imaging device 101, and a medium having a larger capacity than the non-volatile memory 216 is generally used. For example, the recording medium 221 includes a recording medium of any type such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a non-volatile semiconductor memory, and a flash memory.

  The recording and reproducing circuit 220 reads (reproduces) the compressed image signal, the compressed audio signal, the audio signal, various data, and the program recorded on the recording medium 221. Then, the first control circuit 223 transmits the read compressed image signal and compressed audio signal to the image processing circuit 207 and the audio processing circuit 214. The image processing circuit 207 and the audio processing circuit 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode the signal according to a predetermined procedure, and transmit the decoded signal to the video output circuit 217 and the audio output circuit 218. Do.

  The voice input circuit 213 has a plurality of microphones mounted on the imaging apparatus 101, and the voice processing circuit 214 can detect the direction of sound on a plane on which the plurality of microphones are installed. Used. Further, the audio processing circuit 214 detects a specific audio command. The voice command may be configured so that the user can register a specific voice in the imaging device in addition to some commands registered in advance. It also performs sound scene recognition. In sound scene recognition, sound scene determination is performed by a network learned in advance by machine learning based on a large amount of audio data. For example, a network for detecting a specific scene such as “Chaiing”, “Claping”, “Speaking” is set in the audio processing circuit 214. Then, when a specific sound scene or a specific voice command is detected, a detection trigger signal is output to the first control circuit 223 and the second control circuit 211.

  A second control circuit 211, which is provided separately from the first control circuit 223 that controls the entire main system of the imaging device 101, controls the power supplied from the first control circuit 223.

  The first power supply circuit 210 and the second power supply circuit 212 supply power for operating the first control circuit 223 and the second control circuit 211, respectively. Power is supplied to both the first control circuit 223 and the second control circuit 211 by pressing the power button provided on the imaging device 101, but as described later, the first control circuit 223 performs the first power supply. It is controlled to turn off its own power supply to the circuit 210. Even while the first control circuit 223 is not in operation, the second control circuit 211 is in operation, and information from the device fluctuation detection circuit 209 and the audio processing circuit 214 is input. The second control circuit is configured to determine whether to activate the first control circuit 223 based on various input information, and when the activation is determined, instructs the first power supply circuit to supply power. .

  The audio output circuit 218 outputs an audio pattern set in advance from a speaker incorporated in the imaging apparatus 101, for example, at the time of shooting.

  The LED control circuit 224 controls, for example, an LED provided in the imaging device 101 according to a preset lighting blink pattern at the time of shooting or the like.

  The video output circuit 217 includes, for example, a video output terminal, and transmits an image signal to display a video on a connected external display or the like. In addition, the audio output circuit 218 and the video output circuit 217 may be one coupled terminal, for example, a terminal such as a high-definition multimedia interface (HDMI (registered trademark)) terminal.

  The communication circuit 222 performs communication between the imaging apparatus 101 and an external device, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. Also, the imaging device 101 is driven based on an instruction from an external device that can mutually communicate with the imaging device 101 by receiving imaging start and end commands and control signals related to imaging such as pan / tilt and zoom driving. In addition, information such as various parameters related to learning processed by a learning processing circuit 219 described later is transmitted and received between the imaging apparatus 101 and an external apparatus. The communication circuit 222 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a wireless USB, or a GPS receiver.

<Configuration with external communication device>
FIG. 3 is a diagram showing a configuration example of a wireless communication system of the imaging apparatus 101 and the external apparatus 301. The imaging apparatus 101 is a digital camera having a photographing function, and the external apparatus 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

  The imaging apparatus 101 and the smart device 301 communicate, for example, with a wireless LAN conforming to the IEEE 802.11 standard series, and master-slave such as a control station and a dependent station such as Bluetooth Low Energy (hereinafter, referred to as "BLE"). It is possible to communicate by means of the communication 303 which has a relationship. The wireless LAN and BLE are an example of a communication method, and each communication apparatus has two or more communication functions, for example, one communication function that performs communication in the relationship between the control station and the dependent station. Other communication methods may be used as long as control of the other communication function can be performed. However, without loss of generality, the first communication such as a wireless LAN can perform faster communication than the second communication such as BLE, and the second communication consumes more than the first communication. It is assumed that the power is low and / or the communicable distance is short.

  The configuration of the smart device 301 will be described with reference to FIG.

  The smart device 301 has, for example, a public line control circuit 406 for public wireless communication in addition to a wireless LAN control circuit 401 for wireless LAN and a BLE control circuit 402 for BLE. The smart device 301 further includes a packet transmission / reception circuit 403. The wireless LAN control circuit 401 performs RF control of a wireless LAN, communication processing, a driver that performs various controls of communication by a wireless LAN conforming to the IEEE 802.11 standard series, and protocol processing regarding communication by a wireless LAN. The BLE control circuit 402 performs RF control of BLE, communication processing, and protocol processing regarding communication by a driver that performs various control of communication by BLE and BLE. The public line control circuit 406 performs RF control of public wireless communication, communication processing, a driver that performs various controls of public wireless communication, and protocol processing related to public wireless communication. The public wireless communication conforms to, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmission / reception circuit 403 performs processing for performing transmission and / or reception of packets related to communication by wireless LAN and BLE and public wireless communication. In this example, the smart device 301 is described as performing at least one of transmission and reception of a packet in communication, but other communication formats such as circuit switching may be used other than packet switching. Good.

  The smart device 301 further includes, for example, a control circuit 411, a storage circuit 404, a GPS reception unit 405, a display device 407, an operation member 408, an audio input audio processing circuit 409, and a power supply circuit 410. The control circuit 411 controls the entire smart device 301 by executing a control program stored in the storage circuit 404, for example. The storage circuit 404 stores, for example, a control program executed by the control circuit 411 and various information such as parameters required for communication. Various operations to be described later are realized by the control circuit 411 executing a control program stored in the memory circuit 404.

  The power supply circuit 410 supplies power to the smart device 301. The display device 407 has a function capable of outputting visually recognizable information such as an LCD or an LED, or a sound output such as a speaker, and displays various information. The operation member 408 is, for example, a button that receives an operation of the smart device 301 by the user. The display device 407 and the operation member 408 may be configured by a common member such as a touch panel, for example.

  The voice input voice processing circuit 409 may be configured to obtain a voice uttered by the user from, for example, a general-purpose microphone built in the smart device 301, and to obtain a user's operation command by voice recognition processing.

  Also, a voice command is acquired by the user's pronunciation via a dedicated application in the smart device. Then, it can be registered as a specific voice command for causing the voice processing circuit 214 of the imaging apparatus 101 to recognize a specific voice command via the communication 302 by wireless LAN.

  A GPS (Global Positioning System) 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude / latitude information) of the smart device 301. Alternatively, position estimation may estimate the current position of the smart device 301 based on information of a wireless network present in the vicinity using a Wi-Fi Positioning System (WPS) or the like. When the acquired current GPS position information is located in a position range (within a predetermined radius range) set in advance, movement information is notified to the imaging apparatus 101 via the BLE control circuit 402, and will be described later. Use as a parameter for automatic shooting and editing. In addition, when there is a position change of predetermined position or more in the GPS position information, the movement information is notified to the imaging apparatus 101 via the BLE control circuit 402, and used as a parameter for automatic photographing and automatic editing described later.

  As described above, the imaging apparatus 101 and the smart device 301 exchange data with the imaging apparatus 101 by communication using the wireless LAN control circuit 401 and the BLE control circuit 402. For example, it transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. Further, the smart device issues an operation instruction such as photographing of the imaging apparatus 101, transmission of voice command registration data, notification of predetermined position detection based on GPS position information, and notification of location movement. It also sends and receives learning data via a dedicated application in the smart device.

<Configuration of accessories>
FIG. 5 is a diagram showing a configuration example of an external apparatus 501 that can communicate with the imaging apparatus 101. The imaging apparatus 101 is a digital camera having a photographing function, and the external apparatus 501 is a wearable device including various sensing units capable of communicating with the imaging apparatus 101 by, for example, a Bluetooth communication module.

  The wearable device 501 is configured to be attached to, for example, a user's arm, and is a sensor that detects biological information such as the user's pulse, heart rate, blood flow, etc. at a predetermined cycle, and an acceleration sensor that can detect the user's exercise state. Etc. are mounted.

  The biological information detection circuit 502 detects, for example, a change in electric potential detected by contact of the skin with a conductive polymer, a pulse sensor that detects a pulse, a heartbeat sensor that detects a heartbeat, a blood flow sensor that detects blood flow, Include sensors that The present embodiment will be described using a heart rate sensor as the living body information detection circuit 502. The heart rate sensor irradiates the skin with infrared light using, for example, an LED or the like, and detects the heart rate of the user by detecting the infrared light transmitted through the body tissue with a light receiving sensor and processing the signal. The biological information detection circuit 502 outputs the detected biological information as a signal to a control circuit 607 described later.

  The shake detection circuit 503 for detecting the motion state of the user includes, for example, an acceleration sensor or a gyro sensor, and whether or not the user is moving based on acceleration information, or whether the user is swinging an arm And other motions can be detected.

  In addition, an operation member 505 that receives an operation of the wearable device 501 by the user, and a display device 504 that outputs visually recognizable information such as an LCD or an LED are mounted.

  The configuration of the wearable device 501 will be described with reference to FIG.

  The wearable device 501 includes, for example, a control circuit 607, a communication circuit 601, a biological information detection circuit 502, a shake detection circuit 503, a display device 504, an operation member 505, a power supply circuit 606, and a storage circuit 608.

  The control circuit 607 controls the entire wearable device 501, for example, by executing a control program stored in the storage circuit 608. The storage circuit 608 stores, for example, a control program executed by the control circuit 607 and various information such as parameters required for communication. Various operations to be described later are realized, for example, by the control circuit 607 executing a control program stored in the memory circuit 608.

  The power supply circuit 606 supplies power to the wearable device 501. The display device 504 has a function capable of outputting visually recognizable information, such as an LCD or an LED, or a sound output such as a speaker, and displays various types of information. The operation member 505 is, for example, a button that receives an operation of the wearable device 501 by the user. The display device 504 and the operation member 505 may be configured by a common member such as a touch panel, for example.

  In addition, the operation member acquires voice uttered by the user from, for example, a general-purpose microphone built in the wearable device 501, acquires voice uttered by the user by voice processing, and receives user's operation instruction by voice recognition processing. May be acquired.

  The various detection information processed by the control circuit 607 from the biological information detection circuit 502 and the shake detection circuit 503 is transmitted to the imaging device 101 by the communication circuit 601.

  For example, the detection information is transmitted to the imaging apparatus 101 at the timing at which a change in the user's heartbeat is detected, or the detection information is transmitted at the change timing of the movement state such as walking movement / running movement / stop. Also, for example, detection information is transmitted at a timing at which a pre-set arm-like motion is detected, or detection information is transmitted at a timing at which movement of a preset distance is detected.

<Sequence of imaging operation>
FIG. 7 is a flowchart for explaining an example of the operation of the first control circuit 223 of the imaging apparatus 101 according to the present embodiment.

  When the user operates a power button provided on the imaging apparatus 101, the first power supply circuit 210 causes the power supply unit to supply power to each block of the first control circuit 223 and the imaging apparatus 101.

  Similarly, in the second control circuit 211, power is supplied from the power supply unit to the second control circuit by the second power supply circuit 212. For details of the operation of the second control circuit, see the flowchart of FIG. It demonstrates using.

When power is supplied, the process of FIG. 7 starts. In step S701 (hereinafter, "step" is simply referred to as "S"), the activation condition is read. In the present embodiment, the activation condition is as follows.
(1) The power supply button is manually pressed and power supply activation (2) From the power supply activation (3) Sub processor (second control circuit 211) according to the instruction by the external communication (for example, BLE communication) from the external device (for example 301) Power Startup Here, in the case of power startup from the Sub processor of (3), the startup condition calculated in the Sub processor is read, but the details will be described later with reference to FIG.

  Also, the start condition read in here is used as one parameter element at the time of subject search and automatic photographing, which will be described later. When the start condition reading is completed, the process proceeds to S702.

  In S702, reading of various sensors is performed. The sensor read in here is a sensor for detecting vibration such as a gyro sensor from the apparatus shake detection circuit 209 or an acceleration sensor. In addition, the rotational position of the tilt rotation unit 104 or the pan rotation unit 105 may be set. In addition, it may be a detection trigger of a voice level detected by the voice processing circuit 214, a specific voice recognition, or a sound direction detection.

  Further, although not shown in FIGS. 1 to 6, a sensor that detects environmental information also acquires information.

  For example, there are a temperature sensor that detects the temperature around the imaging device 101 at a predetermined cycle, and an air pressure sensor that detects a change in air pressure around the imaging device 101. In addition, an illuminance sensor that detects the brightness around the imaging device 101, a humidity sensor that detects the humidity around the imaging device 101, a UV sensor that detects the amount of ultraviolet light around the imaging device 101, or the like may be provided. . In addition to the detected temperature information, barometric pressure information, brightness information, humidity information, and UV information, the temperature change amount, pressure change amount, brightness change amount, humidity change, etc. are calculated by calculating the change rate at predetermined time intervals from the detected various information. It is used for determination of automatic photographing etc. which mention the amount and the amount of change of ultraviolet rays later.

  When various sensors are read in S702, the process proceeds to S703.

  In S703, it is detected whether communication from the external device is instructed, and when communication is instructed, communication with the external device is performed.

  For example, data such as an audio signal, an image signal, a compressed audio signal, or a compressed image signal is transmitted or received from the smart device 301 via wireless LAN or BLE. In addition, whether there is an operation instruction such as shooting of the imaging device 101 from the smart device 301, transmission of voice command registration data, an instruction of transmission / reception of predetermined position detection notification based on GPS position information, location movement notification and learning data Do you read?

  Also, for example, when there is an update of biological information such as user's exercise information, arm action information, and heartbeat from the wearable device 501, information is read via BLE. The various sensors for detecting the environmental information described above may be mounted on the imaging apparatus 101, or may be mounted on the smart device 301 or the wearable device 501. In that case, the environmental information is read via BLE Do also. When communication reading from the external device is performed in S703, the process proceeds to S704.

  In S704, mode setting determination is performed. The mode set in step S704 is determined and selected from the following.

(1) Automatic shooting mode [mode judgment condition]
From each detection information (image, sound, time, vibration, location, change in the body, environmental change) set by learning described later, elapsed time after transition to the automatic shooting mode, and past shooting information, etc. When it is determined that the automatic shooting should be performed, the automatic shooting mode is set.

[In-mode processing]
In the automatic photographing mode processing (S710), pan / tilt and zoom are driven to automatically search for a subject based on each detection information (image, sound, time, vibration, place, body change, environment change). Then, when it is determined that it is the timing at which the user's favorite shooting can be performed, determination of the shooting method among various shooting methods such as single still image shooting, still image continuous shooting, moving image shooting, panoramic shooting, time-lapse shooting Processing is performed and photographing is performed automatically.

(2) Learning mode [mode judgment condition]
If it is determined that automatic learning should be performed from the elapsed time since the previous learning process and information associated with images that can be used for learning, the number of learning data, etc., the automatic learning mode is set. Be done. Alternatively, this mode is also set when there is an instruction to set learning data via communication from the smart device 301.

[In-mode processing]
In the automatic learning mode process (S712), learning is performed according to the user's preference. Based on information such as each operation in the smart device 301 and notification of learning information from the smart device 301, learning is performed according to the user's preference using a neural network. As information on each operation in the smart device 301, for example, image acquisition information from the imaging apparatus, information on which a manual editing instruction is given via a dedicated application, and a determination value input by the user with respect to the image in the imaging apparatus There is information.

  The details of the automatic shooting mode process and the learning mode process will be described later.

  In step S705, it is determined in step S704 whether the mode setting determination is set to the low power consumption mode. The low power consumption mode determination is made to be in the low power consumption mode when it is not the determination condition of either the “automatic photographing mode” or the “learning mode” described later. When the determination process is performed, the process proceeds to step S705.

  In S705, if it is determined that the low power consumption mode condition is satisfied, the process proceeds to S706.

  In S706, the Sub processor (second control circuit 211) is notified of various parameters related to the activation factor to be determined in the Sub processor (the shake detection determination parameter, the sound detection parameter, and the time lapse detection parameter). Various parameters change in value by being learned in a learning process described later. When the process of S706 is completed, the process proceeds to S707, the power of the Main processor (first control circuit 223) is turned off, and the process is completed.

  On the other hand, if it is determined in S705 that the mode is not the low power consumption mode, the process proceeds to S709 to determine whether the mode setting is the automatic shooting mode. If the automatic shooting mode is selected, the process proceeds to S710 and the automatic shooting mode processing is performed. It will be. When the process ends, the process returns to S702 and repeats the process. If it is determined in S709 that the automatic shooting mode is not set, the process proceeds to S711.

  In S711, it is determined whether or not the mode setting is the learning mode, and if it is the learning mode, the process proceeds to S712, and learning mode processing is performed. When the process ends, the process returns to S702 and repeats the process. If it is determined in S711 that the learning mode is not set, the process returns to S702 to repeat the process.

  FIG. 8 is a flowchart for explaining an example of the operation of the second control circuit 211 of the imaging apparatus 101 according to this embodiment.

  When the user operates the power button provided on the imaging apparatus 101, the second power supply is also supplied to the second control circuit 211 in the same manner as the first power supply circuit 210 supplies power from the power supply unit to the first control circuit 223. Power is supplied from the power supply unit to the second control circuit 211 by the circuit 212. When power is supplied, the Sub processor (second control circuit 211) is activated, and the process of FIG. 8 starts.

  In S801, it is determined whether or not a predetermined period which is a sampling cycle has elapsed. For example, if it is set to 10 msec, the process proceeds to S802 in a 10 msec cycle. If it is determined that the predetermined period has not elapsed, the Sub processor returns to S801 without performing any processing and waits for the predetermined period to elapse.

  In S802, a shake detection value is obtained. The shake detection value is an output value from a sensor that detects vibration such as a gyro sensor or an acceleration sensor from the device shake detection circuit 209.

  When the shake detection value is acquired in S802, the process proceeds to S803, and a process of detecting a shake state set in advance is performed. Some examples are described.

(1) Tap Detection A state (tap state) in which the user taps the imaging device 101 with, for example, a fingertip can be detected from an output value of an acceleration sensor attached to the imaging device 101. The signal area of the acceleration change due to the tap can be extracted by passing the output of the acceleration sensor of three axes through a band pass filter (BPF) set in a specific frequency area at a predetermined sampling cycle. Tap detection is performed based on whether the number of times the acceleration signal after BPF exceeds the predetermined threshold value ThreshA is the predetermined number of times CountA during the predetermined time TimeA. For double taps, CountA is set to 2, and for triple taps, CountA is set to 3.

(2) Detection of Swing State The swing state of the imaging device 101 can be detected from the output value of a gyro sensor or an acceleration sensor attached to the imaging device 101. The output of the gyro sensor or the acceleration sensor is cut by HPF and the low frequency component is cut by LPF, and then the absolute value conversion is performed. The vibration detection is performed based on whether or not the number of times the calculated absolute value exceeds the predetermined threshold value ThreshB is equal to or more than the predetermined number of times CountB during the predetermined time TimeB. For example, it is possible to determine whether the shake is small when the imaging apparatus 101 is placed on a desk or the like, or is large when the imaging apparatus 101 is worn and walked with a wearable. Moreover, it is also possible to detect a fine shaking state according to the shaking level by having a plurality of conditions of the judgment threshold value and the judgment count number.

  When the specific vibration state detection process is performed in S803, the process proceeds to S804, and a predetermined specific sound detection process is performed. Some examples are described.

(1) Specific voice command detection A specific voice command is detected. The voice command allows the user to register a specific voice in the imaging device, in addition to several commands registered in advance.

(2) Specific Sound Scene Recognition Sound scene determination is performed by a network learned in advance by machine learning based on a large amount of audio data. For example, it detects a specific scene such as "Cheers up", "Applause", "Speaking" or the like.

(3) Sound Level Determination Detection based on sound level determination is performed by adding time during which the magnitude of the sound level exceeds the level predetermined value during a predetermined time.

(4) Sound Direction Determination It is possible to detect the direction of the sound on the plane where a plurality of microphones are installed, and to detect the direction of the sound with respect to the sound level of a predetermined magnitude.

  The above determination process is performed in the audio processing circuit 214, and it is determined in S804 whether the specific sound is detected.

  When the specific sound detection process is performed in S804, the process proceeds to S805. In S805, it is determined whether or not the Main processor (first control circuit 223) is in the OFF state, and if the Main processor is in the OFF state, the process proceeds to S806, and the processing for detecting a lapse of time set in advance is performed. The elapsed time from when the main processor transits from ON to OFF is measured, and if the elapsed time is greater than or equal to the parameter TimeC, it is determined that the time has elapsed, and if smaller than TimeC, it is not determined that the time has elapsed.

When the time lapse detection process is performed in S806, the process proceeds to S807, and it is determined whether the low power consumption mode cancellation determination has been made. The low power consumption mode release condition is determined by the following.
(1) Judgment condition of specific vibration detection (2) Judgment condition of specific sound detection (3) Judgment condition of time lapse judgment Whether or not the specific vibration condition detection process in S803 entered the judgment condition of specific vibration detection Can be determined. In addition, it is possible to determine whether or not the determination condition of the specific sound detection has been entered by the specific sound detection process in S804. In addition, it is possible to determine whether or not the determination condition of time lapse detection has been entered by the time lapse detection processing in S806. Therefore, if any one or more conditions are entered, it is determined that the low power consumption mode is canceled.

  If the cancellation condition is determined in S807, the process proceeds to S808, the power of the Main processor is turned on, and the main processor is notified of the condition (shake, sound, time) determined to cancel the low power consumption mode in S809, and the process returns to S801. Loop the process.

  If it is determined in S807 that any release condition is not satisfied and it is determined that the low power consumption mode release determination is not made, the process returns to S801 and loops the processing.

  If it is determined in S805 that the Main processor is in the ON state, the information acquired in S802 to S805 is notified to the Main processor, and the processing returns to S801 and loops.

  In the present embodiment, even when the Main processor is in the ON state, vibration detection and specific sound detection are performed by the Sub processor, and the detection result is notified to the Main processor. However, when the Main processor is ON, the processing in S802 to 805 may not be performed, and the fluctuation detection or the specific sound detection may be detected in the processing in the Main processor (S702 in FIG. 7).

  Although the low power consumption mode cancellation method based on vibration detection, sound detection, and passage of time has been described in detail, the low power consumption mode cancellation may be performed by environmental information. The environmental information can be determined based on whether the absolute amount or the change amount of the temperature, the atmospheric pressure, the brightness, the humidity, and the ultraviolet light amount exceeds a predetermined threshold.

<Automatic shooting mode processing>
The details of the automatic shooting mode process will be described with reference to FIG. As described above, the following process is controlled by the first control circuit 223 of the imaging apparatus 101 in the present embodiment.

  In step S901, the image processing circuit 207 performs image processing on the signal acquired by the imaging unit 206 to generate an image for object recognition.

  From the generated image, subject recognition such as human and object recognition is performed.

  When recognizing a person, the face or human body of the subject is detected. In the face detection process, a pattern for determining the face of a person is determined in advance, and a portion matching the pattern included in the captured image can be detected as a face image of the person.

  Further, the reliability indicating the certainty as the face of the subject is also calculated at the same time, and the reliability is calculated from, for example, the size of the face area in the image, the degree of coincidence with the face pattern, and the like.

  Similarly for object recognition, it is possible to recognize an object that matches the pattern registered in advance.

  Further, there is also a method of extracting a characteristic subject by a method of using a histogram such as hue or saturation in a captured image. In this case, with respect to the image of the subject captured within the shooting angle of view, the process of dividing the distribution derived from the histogram of the hue, saturation, etc. into a plurality of sections and classifying the captured images into sections is performed. Be done.

  For example, a histogram of a plurality of color components is created for the captured image, divided by the mountain-shaped distribution range, the captured image is classified in the region belonging to the combination of the same sections, and the image region of the subject is Be recognized.

  By calculating the evaluation value for each image area of the recognized subject, it is possible to determine the image area of the subject having the highest evaluation value as the main subject area.

  Each subject information can be obtained from the imaging information by the above method.

  In S902, the image shake correction amount is calculated. Specifically, first, the absolute angle of the imaging device is calculated based on the angular velocity and acceleration information acquired by the device shake detection circuit 209. Then, an image stabilization angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angular direction that cancels out the absolute angle is determined, and is used as the image shake correction amount. In the image shake correction amount calculation processing here, the calculation method can be changed by learning processing described later.

  In step S903, the state determination of the imaging apparatus is performed. Based on angular velocity information, acceleration information, angle detected by GPS position information, etc., it is determined what kind of vibration / motion state of the imaging device is at present.

  For example, in the case where the imaging apparatus 101 is attached to a car and shooting is performed, subject information such as a surrounding scenery changes largely depending on the moved distance.

  Therefore, it is possible to determine whether it is a "vehicle moving state" which is attached to a car or the like and moving at a high speed, and can be used for automatic subject search described later.

  Further, it is determined whether or not the change in angle is large, and it is determined whether or not the imaging device 101 is in the “posting state” with almost no swing angle.

  In the case of the “post-taking state”, it may be considered that there is no change in the angle of the imaging device 101 itself, so that it is possible to perform a subject search for the post-taking.

  Also, if the change in angle is relatively large, it is determined that the "hand-held state", and it is possible to perform a hand-held subject search.

  In S904, a subject search process is performed. The subject search is configured by the following processing.

(1) Area Division The area division will be described with reference to FIG. As shown in FIG. 11A, area division is performed all around the position of the imaging device (the origin O is the imaging device position). In the example of FIG. 11A, division is performed at 22.5 degrees in each of the tilt direction and the pan direction. When division is performed as shown in FIG. 11A, as the angle in the tilt direction deviates from 0 degree, the circumference in the horizontal direction becomes smaller and the area area becomes smaller. Accordingly, as shown in FIG. 11B, when the tilt angle is 45 degrees or more, the area range in the horizontal direction is set larger than 22.5 degrees. FIGS. 11 (c) and 11 (d) show examples of area division within the shooting angle of view. An axis 1101 is the direction of the imaging device 101 at the time of initialization, and area division is performed with this direction angle as a reference position. Reference numeral 1102 denotes the angle of view area of the image being captured, and an example of the image at that time is shown in FIG. In the image projected at the angle of view, the image is divided as shown by 1103 to 1118 in FIG. 11D based on the area division.

(2) Calculation of Importance Level for Each Area For each area divided as described above, the importance level indicating the priority to be searched is calculated according to the scene status of the subject present in the area and the area. The importance level based on the condition of the subject is, for example, the number of persons present in the area, the size of the face of the person, the face direction, the certainty of face detection, the facial expression of the person, and the personal authentication result of the person. calculate. In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene discrimination result (blue sky, back light, sunset scene etc.), sound level from the direction of area and voice recognition result, motion detection in area It is information etc. In addition, the vibration state of the imaging device is detected in the state determination (S903) of the imaging device, and the importance level can be changed according to the vibration state. For example, when it is determined that the “deep-shooting state” is selected, a subject search is performed so that a subject search is performed centering on a high-priority subject (for example, the user of the imaging apparatus) registered in face authentication. When face recognition is detected, it is determined that the importance level is high. In addition, the automatic photographing described later is also performed with priority given to the face, and even if the user of the imaging apparatus wears the imaging apparatus and takes a lot of time taking pictures, the imaging apparatus is removed and the desk is By placing it on top, it is possible to leave many images captured by the user. At this time, it is possible to search by pan / tilt, so even if the installation angle of the imaging device is not considered, it is possible to leave an image photographed by the user or a group photograph with many faces photographed only by installing appropriately. it can. In addition, under the above conditions alone, as long as there is no change in each area, the area with the highest importance level is the same, and as a result, the area to be searched remains unchanged. Therefore, the importance level is changed according to the past shooting information. Specifically, the area continuously designated as the search area for a predetermined time is lowered in importance level, or in the area photographed in S910 to be described later, the importance level is lowered for a predetermined time. It is also good.

(3) Determination of Search Target Area When the importance level of each area is calculated as described above, an area having a high importance level is determined as a search target area. Then, the pan / tilt search target angle required to capture the search target area at the angle of view is calculated.

  In S905, pan and tilt driving is performed. Specifically, the pan / tilt drive amount is calculated by adding the drive shake amount at control sampling based on the image blur correction amount and the pan / tilt search target angle, and the lens barrel rotation drive circuit 205 performs tilt rotation. It drives and controls the unit 104 and the pan rotation unit 105 respectively.

  In step S906, the zoom unit 201 is controlled to perform zoom driving. Specifically, the zoom is driven according to the state of the search target subject determined in S904. For example, when the search target subject is the face of a person, if the face on the image is too small, detection is not possible because the size is smaller than the minimum detectable size, and there is a risk of losing sight. In such a case, control is performed to increase the size of the face on the image by zooming to the telephoto side. On the other hand, when the face on the image is too large, the subject or the imaging apparatus itself easily moves the subject out of the angle of view. In such a case, control is performed to reduce the size of the face on the screen by zooming to the wide angle side. By performing zoom control in this manner, it is possible to maintain a state suitable for tracking an object.

  In S904 to S906, although the method for performing subject search by pan / tilt or zoom driving has been described, the subject search may be performed with an imaging system that captures an image in all directions at once using a plurality of wide-angle lenses. In the case of an omnidirectional camera, if all signals obtained by imaging are used as an input image and image processing such as object detection is performed, a vast amount of processing is required. Therefore, a part of the image is cut out, and the search processing of the subject is performed in the cut out image range. The importance level for each area is calculated in the same manner as the method described above, the cutout position is changed based on the importance level, and the automatic photographing determination described later is performed. This makes it possible to reduce power consumption and perform high-speed object search by image processing.

  In step S 907, it is determined whether or not there is a photographing instruction from the user (manually). If there is a photographing instruction, the process advances to step S 910. At this time, the photographing instruction by the user (manually) may be performed by pressing the shutter button, tapping the housing of the imaging device with a finger or the like (tap), voice command input, an instruction from an external device, or the like. The shooting instruction by the tap operation is a shooting instruction method in which when the user taps the housing of the imaging device, the device shake detection circuit 209 detects the acceleration of high frequency continuous in a short period, and uses it as a trigger for shooting. The voice command input is a shooting instruction method of recognizing a voice by the voice processing circuit 214 and using it as a trigger of shooting when the user utters a slogan (for example, "photograph taking" etc.) instructing a predetermined shooting. The instruction from the external device is, for example, a photographing instruction method using as a trigger a shutter instruction signal transmitted through a dedicated application from a smartphone or the like connected to the imaging device and BlueTooth.

  If the user issues a shooting instruction in step S 907, the process advances to step S 914. The process of S 914 and the subsequent S 915 will be described in detail later.

  If there is no shooting instruction in S907, the process advances to S908 to make automatic shooting determination. In the automatic shooting determination, it is determined whether automatic shooting is to be performed and the shooting method (any one of still image shooting, still image continuous shooting (continuous shooting), moving image shooting, panorama shooting, time-lapse shooting, etc. Make a decision.

(1) Determination of Whether to Perform Automatic Shooting Determination of whether to perform automatic shooting is performed based on the following two determinations. First, based on the importance level for each area obtained in S904, when the importance level exceeds a predetermined value, it is determined to perform automatic shooting. The second is determination based on a neural network which is one of machine learning. An example of a multi-layer perceptron network is shown in FIG. 10 as an example of a neural network. A neural network is used to predict an output value from an input value, and learns in advance an input value and an output value that is an exemplar for the input to obtain a new input value. On the other hand, it is possible to estimate an output value that follows the learned model. The method of learning will be described later. In FIG. 10, reference numeral 1001 in FIG. 10 and the vertically arranged circles are neurons in the input layer, 1003 and the vertically arranged circles are neurons in the intermediate layer, and 1004 are neurons in the output layer. Arrows such as 1002 indicate connections connecting each neuron. In the judgment based on the neural network, to the neurons of the input layer, an object based on the current view angle, a feature value based on the scene or the state of the imaging device is given as an input, and an operation based on the forward propagation law of the multilayer perceptron To obtain the value output from the output layer. Then, if the value of the output is equal to or greater than the threshold value, it is determined to perform automatic photographing. The characteristics of the subject include the current zoom magnification, the result of general object recognition at the current angle of view, the result of face detection, the number of faces in the current angle of view, the degree of smile and closing of the face, face angle, and face authentication ID The number, the gaze angle of the subject person, the scene discrimination result, the detection result of the specific composition, and the like are used. Also, use the elapsed time since the last shooting, the current time, GPS position information, the amount of change from the last shooting position, the current voice level, the person making a voice, whether applause or cheering has risen, etc. May be Also, vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet light) or the like may be used. Furthermore, when there is information notification from the wearable device 501, notification information (user's exercise information, arm's action information, biological information such as heart beat, etc.) may also be used as a feature. This feature is converted into a numerical value of a predetermined range, and given to each neuron of the input layer as a feature amount. Therefore, each neuron of the input layer is required as many as the number of feature quantities to be used.

  The judgment based on this neural network can change the output value by changing the connection weight between the neurons by learning processing described later, and the result of the judgment can be adapted to the learning result.

  In addition, the determination as to the automatic photographing changes according to the start condition of the Main processor read in S702 of FIG. For example, in the case of activation by tap detection or activation by a specific voice command, there is a very high possibility that the operation is for the user to want to take a picture at present. Therefore, the frequency of photographing is set to be increased.

(2) Determination of shooting method In the determination of the shooting method, any one of still image shooting, moving image shooting, continuous shooting, panorama shooting and the like is detected based on the state of the imaging apparatus and the surrounding objects detected in S901 to S904. To determine whether to execute. For example, when the subject (person) is stationary, still image shooting is performed, and when the subject is moving, moving image shooting or continuous shooting is performed. In the case where a plurality of subjects exist so as to surround the imaging apparatus, or when it is determined that the scenery is scenic based on the above-described GPS information, images taken sequentially while operating the pan and tilt are A panoramic imaging process may be performed to generate a panoramic image by combining.

  In step S909, if it is determined in step S908 that shooting is determined by the automatic shooting determination in step S908, the process advances to step S910; otherwise, the process advances to end of shooting mode processing.

  In S910, shooting is started. At this time, in the case of manual shooting, shooting of a still image or shooting according to a shooting method manually set by the user is performed, and in the case of automatic shooting, shooting according to the shooting method determined in S908 is started. At this time, auto focus control is performed by the focus drive control circuit 204. In addition, exposure control is performed so that the subject has appropriate brightness by using an aperture control circuit, a sensor gain control circuit, and a shutter control circuit (not shown). Furthermore, after image capturing, the image processing circuit 207 performs various image processing such as auto white balance processing, noise reduction processing, gamma correction processing, and the like to generate an image.

  In the automatic photographing, when the predetermined condition is satisfied, the image pickup apparatus may perform the photographing process after notifying that the person to be photographed is to be photographed, and then the photographing may be performed. The method of notification may use, for example, voice from the audio output circuit 218 or LED lighting light by the LED control circuit 224, or a motion operation that visually guides the subject's line of sight by driving pan and tilt. You may The predetermined conditions are, for example, the number of faces within the angle of view, the degree of smile / eye closure of the face, the gaze angle or face angle of the subject person, the face authentication ID number, the number of persons registered for personal authentication, etc. . In addition, whether the current position based on general object recognition result at the time of shooting, scene discrimination result, shooting time at the previous shooting time, shooting time, GPS information is scenic position, voice level at shooting, voice out Whether the person is present, applause, cheers are rising, etc. Also, the information includes vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet light), and the like. By performing informing shooting based on these conditions, it is possible to leave a preferable image of the camera gaze in a scene of high importance.

  In addition, there are multiple predetermined conditions, change the sound according to each condition, change the lighting method of LED (color, blink time, etc.), or pan / tilt motion method (motion or driving speed) You may change it.

  In step S911, editing processing such as processing the image generated in step S910 or adding it to a moving image is performed. More specifically, image processing is trimming processing based on the face of a person or the in-focus position, image rotation processing, HDR (high dynamic range) effect, blur effect, color conversion filter effect, and the like. A plurality of image processes may be generated based on the image generated in S910 by a combination of the above-described processes, and may be stored separately from the image generated in S910. In addition, in the moving image processing, processing may be performed such that the captured moving image or still image is added to the generated edited moving image while performing special effect processing of slide, zoom, and fade. With regard to the editing in S911, the method of image processing can also be determined by judgment based on information of the photographed image or various information detected before photographing based on the neural network, and this judgment processing is the learning processing described later. The judgment conditions can be changed.

  In S912, learning information generation processing of the photographed image is performed. Here, learning information to be used for learning processing described later is generated and recorded. Specifically, the zoom magnification at the time of shooting, the result of general object recognition at the time of shooting, the result of face detection, the number of faces included in the shot image, the degree of smile / eye closing of face, face angle, face The authentication ID number, the gaze angle of the subject person, and the like. In addition, as a result of scene discrimination, elapsed time since the last shooting, shooting time, GPS position information and change amount from the last shooting position, voice level at the time of shooting, voiced person, applause, cheering or not It is In addition, vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, ultraviolet light amount), moving image shooting time, whether by manual shooting instruction, etc. Furthermore, a score, which is the output of a learning model that quantifies the preferences of the user's image, is also calculated.

  These pieces of information are generated and recorded as tag information in the photographed image file. Alternatively, the information may be written to the non-volatile memory 216 or may be stored in the recording medium 221 as so-called catalog data in the form of a list of information of each photographed image.

  In step S913, the past shooting information is updated. Specifically, the number of shots for each area described in the description of S908, the number of shots for each person registered for personal identification, the number of shots for each subject recognized by general object recognition, the number of shots for each scene determination scene The count of the number of images corresponding to the image taken this time is incremented by one.

<Learning mode processing>
Next, learning in accordance with the preference of the user in the present embodiment will be described.

  In this embodiment, using a neural network as shown in FIG. 10, using a machine learning algorithm, the learning processing circuit 219 performs learning in accordance with the user's preference to generate a learning model. The learning processing circuit 219 uses, for example, Jetson TX2 manufactured by NVIDIA. The neural network is used to predict the output value from the input value, and learns in advance the actual value of the input value and the actual value of the output value to output an output for the new input value. The value can be estimated. By using a neural network, learning is performed according to the user's preference for the above-described automatic shooting and subject search.

  In addition, registration of an object (such as face recognition or general object recognition), which is also feature data to be input to a neural network, is performed.

  In the present embodiment, the elements to be learned by the learning process are as follows.

(1) Automatic shooting The learning for automatic shooting will be described. In automatic shooting, learning is performed to automatically take an image that suits the user's preference. As described above in the explanation using the flow of FIG. 9, the learning information generation process is performed after the photographing (S912). An image to be learned is selected by a method to be described later, and learning is performed by changing weights of the neural network based on learning information included in the image. The learning is performed by changing a neural network that performs automatic shooting timing determination and determining a shooting method (still image shooting, moving image shooting, continuous shooting, panoramic shooting, etc.).

(2) Automatic Editing Describe the learning for automatic editing. In automatic editing, learning is performed on editing immediately after shooting in S911 of FIG. The editing immediately after shooting will be described. An image to be learned is selected by a method to be described later, and learning is performed by changing weights of the neural network based on learning information included in the image. Various detection information obtained by shooting or information immediately before shooting is input to a neural network, and determination of editing method (trimming process, image rotation process, HDR (high dynamic range) effect, blur effect, color conversion filter effect, etc.) I do.

(3) Subject Search Learning for subject search will be described. In the subject search, learning is performed to automatically search for a subject matching the user's preference. As described above in the description using the flow of FIG. 9, in the subject search process (S904), the importance level of each area is calculated, pan / tilt and zoom are driven, and the subject search is performed. The learning is performed based on the photographed image and the detection information during the search, and learning is performed by changing the weight of the neural network. Various detection information during the search operation is input to the neural network, the importance level is calculated, and the pan / tilt angle is set based on the importance level to perform the object search reflecting the learning. In addition to the setting of the pan and tilt angles based on the importance level, for example, learning of pan and tilt driving (speed, acceleration, frequency of movement) is also performed.

(4) Subject Registration The learning for subject registration will be described. In subject registration, learning is performed to automatically register and rank subjects according to the user's preference. As learning, for example, registration of face recognition registration and registration of general object recognition, registration of gesture and voice recognition, and scene recognition by sound are performed. In authentication registration, authentication registration is performed for a person and an object, and rank setting is performed based on the number and frequency of image acquisition, the number and frequency of manual photographing, and the frequency of appearance of a subject under search. The registered information is registered as an input of determination using each neural network.

  Next, the learning method will be described.

  As a learning method, there are “learning in the imaging device” and “learning in cooperation with the communication device”.

  The method of in-imaging device learning will be described below. There are the following methods for in-imaging device learning in the present embodiment.

(1) Learning based on detection information at the time of shooting instruction by the user As described in S 907 to S 913 in FIG. 9, in the present embodiment, the imaging apparatus 101 can perform two shootings: manual shooting and automatic shooting. If there is a shooting instruction by manual operation (made based on three determinations as described above) in S 907, in S 912, information is added that the captured image is an image captured manually. In addition, when it is determined that the automatic imaging is ON in S909, the information is added that the captured image is an image captured automatically in S912.

  In the case of manual shooting here, the possibility of shooting based on the user's favorite subject, favorite scene, favorite place and time interval is very high. Therefore, learning based on learning of each feature data and captured image obtained at the time of manual imaging is performed.

  Further, from detection information at the time of manual imaging, learning is performed regarding extraction of feature amounts in the captured image, registration of personal authentication, registration of individual facial expressions, and registration of human combinations. In addition, from the detection information at the time of subject search, for example, learning is performed to change the importance of a nearby person or object from the expression of the subject registered individually.

(2) Learning subject based on detection information at the time of subject search During the subject search operation, it is judged what kind of person, object or scene the subject registered with personal identification is taken at the same time, and the time taken simultaneously within the angle of view Calculate the ratio.

  For example, it can be determined that the importance is high when the time ratio in which the person A of the personal identification registration subject is shown simultaneously with the person B of the personal identification registration subject is higher than a predetermined threshold. For this reason, when the person A and the person B fall within the angle of view, various types of detection information are stored as learning data so as to increase the score of the automatic photographing determination, and learning is performed in the learning mode processing 716.

  In another example, it can be determined that the importance is high when the time ratio at which the person A of the personal identification registration subject simultaneously appears with the subject "cat" determined by general object recognition is higher than a predetermined threshold. Therefore, when the person A and the "cat" fall within the angle of view, various detection information is stored as learning data so that the score of the automatic photographing determination becomes high. Then, learning is performed in a learning mode process 716.

  As described above, when the number of occurrences of the subject under search is high, if the score of the automatic photographing determination is increased, the importance of the person or object near the subject registered for personal identification is also increased. It can be changed.

  In addition, when the degree of smile of the person A of the personal identification registration subject is detected or “joy” or “surprise” is detected by the detection of the expression, the subject that is shown at the same time is learned to be important To be done. Also, processing such as not learning is performed because the subject appearing at the same time when an expression such as "anger" or "true face" is detected is unlikely to be important.

  Next, learning in cooperation with an external communication device in the present embodiment will be described. There are the following methods for learning in cooperation with an external communication device in the present embodiment.

(3) Learning Based on Acquisition of Image by External Communication Device As described in FIG. 3, the imaging apparatus 101 and the external device 301 have communication means of the communication 302 and 303. The image transmission / reception is mainly performed by the communication 302, and the image in the imaging apparatus 101 can be acquired by communication with the external device 301 via a dedicated application in the external device 301. Also, the thumbnail image of the image data stored in the imaging apparatus 101 can be browsed through a dedicated application in the external device 301. In this way, the user can select an image that he / she likes from the thumbnail images, check the image, and operate the image acquisition instruction to acquire the image on the external device 301.

  At this time, since the user has selected an image and instructed to transmit and acquire it, the acquired image is very likely to be an image preferred by the user. Therefore, it is possible to perform various types of learning of the user's preference by determining that the acquired image is an image to be learned and learning based on the learning information of the acquired image.

  An operation example will be described. An example in which an image in the imaging apparatus 101 is browsed via an application dedicated to the external device 301 which is a smart device is shown in FIG. The thumbnail images (1404 to 1409) of the image data stored in the imaging apparatus are displayed on the display device 407, and the user can select an image that he / she likes and can acquire an image. At this time, a display method change unit (1401, 1402, 1403) for changing the display method is provided. When 1401 is pressed, the display order is changed to the date and time priority display mode, and the images are displayed on the display device 407 in the order of the shooting date and time of the image in the imaging device 101. (For example, the date and time is displayed as 1404 is new, and 1409 is displayed as the date and time are old.) When 1402 is pressed, the recommended image priority display mode is changed. The images are displayed on the display device 407 in the descending order of the scores of the images in the imaging apparatus 101 based on the scores obtained by determining the preference of the user for each image calculated in FIG. 9S912. (For example, 1404 indicates that the score is high and 1409 indicates that the score is low.) Pressing 1403 can designate a person or an object subject, and subsequently designating a specific person or an object subject displays only a specific subject. You can also

  1401 to 1403 can simultaneously turn on the settings. For example, when all the settings are turned on, only a designated subject is displayed, and an image with a new shooting date and time is prioritized, and an image with a high score is displayed. Will take precedence and will be displayed.

  As described above, since the user's preference is learned also for the photographed image, it is possible to easily extract only the user's favorite image from a large number of photographed images by a simple confirmation operation. .

(4) Learning by inputting a determination value to an image through an external communication device As described above, the imaging device 101 and the external device 301 have communication means, and are stored in the imaging device 101. These images can be browsed through a dedicated application in the external device 301. Here, the user may be configured to score each image. A high score (for example, 5 points) can be added to an image that the user thinks is preferred, or a low score (for example, 1 point) can be added to an image thought to be unpreferable. Will be structured as you learn. The score of each image is used for relearning along with learning information in the imaging device. Learning is performed so that the output of the neural network, which has feature data from specified image information as an input, approaches a point specified by the user.

  In the present embodiment, the user inputs the determination value to the photographed image via the communication device 301. However, the imaging device 101 may be operated to directly input the determination value to the image. . In that case, for example, the imaging device 101 is provided with a touch panel display, and the user presses a GUI button displayed on the touch panel display screen display device to set a mode for displaying a photographed image. Then, the user can perform similar learning by a method of inputting a determination value to each image while confirming the photographed image.

(5) Learning by changing parameters in an external communication device As described above, the imaging device 101 and the external device 301 have communication means, and learning parameters currently set in the imaging device 101 Can be communicated to the external device 301 and stored in the storage circuit 404 of the external device 301. As learning parameters, for example, weights of a neural network, selection of an object to be input to the neural network, and the like can be considered. In addition, the learning parameter set in the dedicated server can be acquired via the public line control circuit 406 via the dedicated application in the external device 301, and can be set as the learning parameter in the imaging apparatus 101. I assume. In this way, the parameters at a certain point in time can be stored in the external device 301, and learning parameters can be returned by setting them in the imaging apparatus 101, or learning parameters possessed by other users can be obtained via a dedicated server. It is also possible to acquire and set the image capturing apparatus 101 of its own.

  Next, the learning processing sequence will be described.

  In the mode setting determination in S704 of FIG. 7, it is determined whether or not the learning process should be performed. When the learning process is performed, it is determined that the learning mode is set, and the learning mode process in S712 is performed.

  The determination conditions of the learning mode will be described. Whether or not to shift to the learning mode is determined based on the elapsed time since the previous learning process, the number of information that can be used for learning, and whether there is a learning process instruction via the communication device. FIG. 12 shows a flow of processing for determining whether or not to shift to the learning mode, which is determined in the mode setting determination processing of S704.

  When the start of the learning mode determination is instructed in the mode setting determination process of S704, the process of FIG. 12 starts. In S1201, it is determined whether there is a registration instruction from the external device 301. The registration here is a determination as to whether or not there is a registration instruction for learning as described above. For example, there are <learning based on image information acquired by the communication device> and <learning based on inputting a determination value to an image through the communication device>. If there is a registration instruction from the external device in S1201, the process advances to S1208, the learning mode determination is set to TRUE, and the process of S712 is set to be performed. If there is no registration instruction from the external device in S1201, the process proceeds to S1202. In S1202, it is determined whether there is a learning instruction from an external device. The learning instruction here is a determination as to whether or not there is an instruction to set a learning parameter, as in <Learning by changing the imaging device parameter in the communication device>. If there is a learning instruction from the external device in S1202, the process advances to S1208, the learning mode determination is set to TRUE, and the process of S712 is set to be performed, and the learning mode determination process is ended. If there is no learning instruction from the external device in S1202, the process proceeds to S1203.

  In S1203, an elapsed time Time N after the previous learning process (recalculation of weights of the neural network) is obtained, and the process proceeds to S1204. In S1204, the number of new data to be learned DN (the number of images designated to be learned during the elapsed time Time N since the previous learning process was performed) is acquired, and the process proceeds to S1205. In S1205, the threshold value DT is calculated from TimeN. For example, the threshold value DTa when TimeN is smaller than a predetermined value is set larger than the threshold value DTb when larger than the predetermined value, and the threshold value is set to be smaller with the passage of time. As a result, even when the amount of learning data is small, the image pickup apparatus is made to easily change learning according to the time of use by learning again if the time elapsed is large.

  When the threshold value DT is calculated in S1205, the process proceeds to S1206, and it is determined whether the number of data to be learned DN is larger than the threshold value DT. If the DN is larger than the threshold value DT, the process proceeds to S1207, sets the DN to 0, and then proceeds to S1208, sets the learning mode determination to TRUE, performs the process of S712, and ends the learning mode determination process. Do.

  If the DN is less than the threshold value DT in S1206, the process proceeds to S1209. Since there is neither a registration instruction from an external device nor a learning instruction from an external device, and the number of learning data is less than a predetermined value, the learning mode determination is set to FALSE and the process of S712 is not performed, and the learning mode determination is made. End the process.

  Next, the process in the learning mode process (S712) will be described. The detailed flow of the learning mode process is shown in FIG.

  When it is determined in S711 of FIG. 7 that the learning mode is set, and the process proceeds to S712, the process of FIG. 13 starts. In S1301, it is determined whether there is a registration instruction from the external device 301. If there is a registration instruction from the external device in S1301, the process proceeds to S1302. In S1302, various registration processing is performed.

  The various registrations are registrations of features to be input to the neural network, such as registration of face recognition, registration of general object recognition, registration of sound information, registration of location information, and the like.

  When the registration process is completed, the process proceeds to S1303, and the element to be input to the neural network is changed from the information registered in S1302.

  When the process of S1303 ends, the process proceeds to S1307.

  If there is no registration instruction from the external device 301 in S1301, the process advances to S1304 to determine whether there is a learning instruction from the external device 301. If there is a learning instruction from the external device, the process advances to step S1305 to set the learning parameters communicated from the external device in each determiner (such as the weight of the neural network), and the process advances to step S1307.

  If there is no learning instruction from the external device in S1304, learning (recalculation of weight of neural network) is performed in S1306. As described with reference to FIG. 12, the process of S1306 is a condition under which the number of data to be learned DN exceeds the threshold, and the respective learners can perform relearning. Retraining is performed using a method such as error back propagation method or gradient descent method, and weights of the neural network are recalculated to change parameters of each decision unit. When the learning parameter is set, the process proceeds to S1307.

  At S1307, the images in the file are re-scored. In the present embodiment, all photographed images stored in the file (recording medium 221) are scored based on learning results, and automatic editing and automatic file deletion are performed according to the scored scores. It is a structure. Therefore, when re-learning or setting of learning parameters from an external device is performed, it is also necessary to update the scores of captured images. Therefore, in S1307, recalculation for adding a new score to the photographed image stored in the file is performed, and when the process ends, the learning mode process ends.

  In the present embodiment, although the explanation has been made based on the configuration for learning in the imaging apparatus 101, the external device 301 has a learning process, data necessary for learning is communicated to the external device 301, and only the external device The same learning effect can be realized even with a configuration that executes learning. In that case, as described above in <Learning by changing parameters in the communication device>, learning is performed by setting parameters such as weights of the neural network learned on the external device side in the imaging apparatus 101 by communication. It may be configured.

  Further, both the inside of the imaging apparatus 101 and the inside of the external device 301 may have a learning process. For example, learning information possessed by the external device 301 may be communicated to the imaging apparatus 101 at the timing when the learning mode processing 716 is performed in the imaging apparatus 101, and learning may be performed by merging learning parameters.

  Next, a method of compensating for the shortage of teacher data in learning of a neural network will be described.

  A sufficient number of training data is required to accurately estimate the output value from the input value in the neural network. It is difficult to increase the estimation accuracy if the model of the neural network is complex and has a high degree of freedom with respect to the number of teacher data. In the field of machine learning, in order to be able to estimate robustly even data that is slightly different from teacher data, a process called Data Augmentation may be performed. This is often performed by adding image processing such as change of aspect ratio, rotation (roll, pitch, yaw), blurring, addition of noise, shift and the like to teacher data (in this case, an image). However, it does not necessarily coincide with the image that can be taken by the camera. For example, even if blurring is added in image processing, equivalent blurring can not always be realized even if the camera actually opens the aperture or shifts the focus.

  If the data desired to be estimated by the neural network and the teacher data are not similar, this teacher data may be a factor to reduce the estimation accuracy of the neural network. Further, even if a predetermined rotation (roll, pitch, yaw) is added, it is not always possible to reproduce the angle at which a person actually shoots with a camera. Specifically, even if the image is simply rotated 45 degrees or 90 degrees from the center of the image, for example, the user does not have a chance to take a picture with the subject not standing upright. Have a low degree of contribution.

  As described above, it is difficult to compensate for the shortage of teacher data by image processing, and it is preferable to increase teacher data by actual photographing. Alternatively, when performing data augmentation by image processing, it is better to use an image closer to an image that can be captured by the camera than an image that can not be captured by the camera. Therefore, in the present embodiment, a method of automatically performing actual shooting for learning to increase teacher data will be described.

  As described above, when it is determined in S907 of FIG. 9 that there is a photographing instruction from the user, the process proceeds to S910 and S914.

  In S914, it is determined whether the current number of teacher data is smaller than a predetermined number N (N is a natural number). Then, only when the current number of teacher data is smaller than the predetermined number N, it is determined that the teacher data is insufficient, the process proceeds to S915, and learning automatic imaging for supplementing the teacher data is performed. This N may be changed according to the complexity and the degree of freedom of the neural network (the number of nodes and the number of layers in the middle layer). If the neural network is complex or has a high degree of freedom, the number of required teacher data increases, so N is increased. If it is determined in S914 that the current number of teacher data is N N and it is determined that sufficient teacher data has been stored, S915 is skipped and the process proceeds to S912.

  The automatic shooting for learning only when a manual shooting instruction is given in S 907 is, as described above, when shooting manually, shooting is performed based on the user's favorite subject, favorite scene, favorite place and time interval. Because the possibility is very high. Therefore, if automatic imaging for learning is performed at this time, there is a high possibility that teacher data reflecting the preference of the user can be acquired.

  In addition, since the automatic shooting for learning in step S915 and the manual shooting in step S910 can not be performed simultaneously, the timing is shifted. Either way may be ahead, but automatic shooting for learning and manual shooting should be performed continuously. If the automatic learning for learning is delayed, the user may move the camera and move away from the preferable composition at the time of manual photographing. On the other hand, if the manual shooting is delayed, the shutter timing will be shifted. In the following description, unless otherwise specified, manual imaging is performed first and automatic imaging for learning is performed immediately thereafter.

  In S915, learning automatic imaging is performed. There are several methods for automatic learning for learning. The first one is continuous shooting. After manual shooting, continuous shooting is performed automatically for learning, and continuous shooting images are acquired. If the timing is close to that of the manual shooting, it is possible to acquire a plurality of teacher data similar to the user's favorite image. In this case, an image obtained by manual shooting is treated as a recorded image, and thus is recorded in the recording medium 221. However, an image obtained by automatic learning for learning is used only for learning, and is used by the user's eyes. It does not attach.

  The second is moving image shooting. Before or after manual shooting, moving image shooting is automatically performed, and a function in which a still image and a moving image are combined and provided to the user may be installed in a general camera or a life log camera. Alternatively, the camera has a function of constantly overwriting a moving image for a predetermined period in a memory such as a ring buffer and providing the user with moving images of a predetermined period before and after the timing at which the still image was taken. There is a thing. This automatically acquired moving image is decomposed into still images and used as teacher data. This is also valuable as teacher data for the same reason as the first continuous shooting. In addition, when using this function, it is not necessarily restricted and you may shoot a moving image only for the purpose of learning. In that case, the video is not provided to the user.

  The third is bracket shooting. Bracket shooting is performed by gradually changing shooting conditions in manual shooting. The shooting conditions to be changed may be parameters that can be changed by the camera, such as focus, exposure, white balance, flash, zoom, and sharpness. By changing these shooting conditions, the same effect as Data Augmentation can be expected. If you use Data Augmentation, which can not be realized with a camera, and use it as teacher data, the neural network learned from it can only estimate things close to the teacher data well. This makes it unsuitable as a neural network for cameras. Therefore, if the increase in teacher data by bracket photography that can be realized by a camera, an effect as Data Augmentation can be expected.

  There are two types of bracket shooting, one that must be performed immediately after manual shooting, and one that can be performed even after a certain amount of time. The former involves mechanical operations such as focusing and zooming. If these are not taken continuously with manual shooting, the composition will change and can not be used as teacher data. On the other hand, the latter is based on image processing such as white balance, sharpness, development conditions of RAW image data, and the like. Even if these can not be performed continuously with the manual imaging, they can be generated based on the manually captured image. In this case, RAW data of a manually captured image may be recorded. When generating based on a manually captured image, it does not have to be at the time of shooting, and may be generated while the camera is on standby.

  As described above, there are some types of bracket imaging that need to be performed continuously with manual imaging and some that do not need to be performed continuously, so even if priority is given to the type of bracket imaging and automatic imaging is performed. Good. This means that bracket shooting, which needs to be performed continuously with manual shooting, is performed first.

  In addition, when it is determined from the information of the angular velocity sensor 106 of the camera and the accelerometer 107 etc. that the user moved the camera from manual imaging to automatic imaging for learning, automatic imaging for learning is to be canceled. May be

  In S912, learning information for manual shooting and automatic shooting for learning is generated, and teacher data is created. The learning information can also be generated in the same manner as the image obtained by the manual shooting, for the image obtained by the automatic shooting for learning. The images obtained by the manual imaging are highly likely to be the user's preference, so they are given a predetermined high score. Then, the score is also attached to teacher data generated from an image obtained by automatic learning for learning.

  Alternatively, the image obtained by the automatic learning for learning may be scored according to the relationship with the image obtained by the manual shooting. For example, if automatic imaging for learning is performed without leaving a space between manual imaging, an image obtained by automatic imaging for learning is also given a high score equivalent to an image obtained by manual imaging. . Then, as the interval between the manual imaging and the learning automatic imaging increases, it is possible to lower the score of the image obtained by the learning automatic imaging. As a result, the score of the manually shot image at the best shot timing instructed by the user is the highest, and the point becomes lower as it deviates from that point, so that the user's preference for shutter timing can be learned. Alternatively, the degree of similarity may be compared with that of the manually captured image for each of the images obtained by learning for automatic imaging, and a score may be added according to the degree of similarity. Furthermore, when the subject is a moving object or the scene including the subject is changed, the images before and after the image captured at the timing of manual shooting are dared to be used as learning data as negative teacher data. May be By doing this, it can be expected that the user's shutter timing preference can be learned more strictly. Further, instead of the images before and after, an image whose similarity with the image obtained by the manual imaging is lower than the threshold among the continuously captured images may be set as the negative teacher data.

  Also, with regard to the bracket image, in the same way, the score can be lowered as the shooting condition set by the bracket deviates from the shooting condition set by the manual shooting. For example, the highest score is given to the image obtained by manual shooting, the second highest score is given to the image set as exposure correction + 1 by bracket shooting, the third highest score to the image set as exposure correction + 2 etc. is there. This makes it possible to learn about the user's favorite imaging conditions.

  The learning information of the image obtained by automatic learning for learning may be diverted from the learning information of the image obtained by manual photographing. For example, there is a high possibility that the subject to be photographed is the same in automatic learning for learning and manual photographing, so general object recognition results and face detection results generated from images obtained by manual photographing are for learning It can be diverted as learning information of an image obtained by automatic photographing. Thus, the time for generating the learning information can be shortened.

  In addition, the shooting instruction from the user to be determined in S 907 may include the above-described voice command, the tap operation to the device 101, and the shooting instruction from the external devices 301 and 501.

  In addition, since automatic learning for learning itself is not instructed by the user, it is desirable to perform photographing using an electronic shutter with a small shutter sound in automatic learning for learning.

  Further, in the case of automatic learning for learning, since photographing is performed at a timing different from the timing intended by the user, there is a possibility that personal information not intended by the user may be stored. This can be an issue when considering privacy. Therefore, only the learning information generated from the image may be stored without storing the image obtained by the automatic learning for learning. The learning information is, for example, a parameter corresponding to the input layer of the neural network, and is in a form other than an image, so privacy information is difficult to identify. Alternatively, information related to a person such as a personal identification ID may not be recorded in the learning information, and instead may be replaced with a predetermined specified value.

  Further, the condition for performing automatic learning for learning in step S 914 may not be the number of teacher data. For example, if it is determined that the estimation accuracy of the neural network has increased, the determination in S914 may be NO. It is verified by the following method whether the estimation accuracy has increased. When teacher data is acquired by automatic imaging for learning, it is input to a neural network to obtain an output value. If the difference between the output value and the training value is smaller than a predetermined value, it can be determined that the accuracy of the neural network is enhanced. That is, even if new data is input, the output value is close to the model value, so it can be determined that the accuracy is high.

  Moreover, it is also possible to remove out of the teacher data acquired by automatic learning for learning as what is not suitable as teacher data as an outlier using this “difference between neural network output value and teacher value”. If the difference between the neural network output value and the teacher value is larger than a predetermined value, it means that estimation has not been made, and this teacher data can be said to be teacher data whose nature is significantly different from the teacher data learned in the past. In this case, the camera is already moved by the user immediately after the manual shooting, and it is highly likely that the camera is facing an unintended direction such as the sky or the ground, and is removed as an outlier. That is, it does not register as teacher data.

  In addition, outlier verification of learning automatically captured images is possible without passing through a neural network. If the difference between the automatically captured image for learning and the manually captured image is larger than a predetermined value in a feature amount vector obtained by combining the feature amounts of the input layer of the neural network, it may be removed as an outlier.

  These automatic learning for learning makes it possible to increase teacher data. These teacher data are used for learning when the next learning mode is executed. As the amount of teacher data increases, the estimation accuracy of the neural network can be improved.

(Other embodiments)
Furthermore, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by the process to be executed. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The present invention is not limited to shooting with digital cameras and digital video cameras, and can also be installed in shooting devices such as surveillance cameras, web cameras, and mobile phones.

101: imaging apparatus, 301: smart device, 501: wearable device, 104: tilt rotation unit, 105: pan rotation unit

Claims (21)

Acquisition means for acquiring data related to a photographed image photographed by the photographing means;
Learning means for learning conditions of an image preferred by the user based on the teacher data acquired by the acquisition means;
A control unit that determines the automatic photographing by the photographing unit based on the conditions learned by the learning unit;
Registration means for registering, as the teacher data, data acquired by the acquisition means with respect to a learning image for photography continuously taken in response to a user's instruction.
An imaging apparatus comprising:   The image processing apparatus further includes a developing unit that develops the photographed image, and the developing unit develops the photographed image for learning under conditions different from those of the photographed image obtained from the photographing performed by the user's instruction. The imaging device according to claim 1.   The imaging apparatus according to claim 1, wherein the user's instruction is performed by inputting a predetermined voice command.   The imaging device according to any one of claims 1 to 3, wherein the instruction of the user is performed through an external device capable of communicating with the imaging device.   5. The teaching value of the photographed image for learning is determined based on the teaching value of a recorded image which is a photographed image obtained from photographing performed by the user's instruction. An imaging device according to claim 1.   The teaching value of the taken image for learning is set so that the difference between the teaching value of the recorded image and the taken image for learning becomes larger as the difference between the photographing conditions of the recorded image and the taken image for learning increases. The imaging device according to claim 5, wherein the imaging device is determined.   The teaching value of the photographed image for learning is set so that the difference between the teaching value of the recorded image and the photographed image for learning increases as the difference between the shutter timings of the recorded image and the photographed image for learning increases. The imaging device according to claim 5, wherein the imaging device is determined.   The imaging device according to any one of claims 1 to 7, wherein the registration unit registers at least a part of the photographed image for learning as negative teacher data.   The registration unit is characterized in that, among the photographed images for learning, an image having a degree of similarity lower than a threshold with a photographed image obtained from the photographing performed by the user's instruction is registered as negative teacher data. The imaging device according to claim 8.   The photographed image for learning is photographed by changing the photographing condition of the photographed image obtained from the photographing performed by the user's instruction and at least any of the focus, exposure, white balance, strobe light emission, and zoom. The imaging device according to any one of claims 1 to 9, which is a captured image.   The imaging device according to any one of claims 1 to 10, wherein the learning photographed image is generated from a moving image photographed immediately before or after photographing performed by the user's instruction. .   The imaging device according to any one of claims 1 to 10, wherein the learning photographed image is a continuously shot image photographed immediately before or after photographing performed according to the user's instruction. .   The imaging apparatus according to any one of claims 1 to 12, wherein the imaging unit captures the learning image by using an electronic shutter.   14. The control method according to any one of claims 1 to 13, wherein when the number of the teacher data is smaller than a predetermined number, the control means causes the imaging means to capture the captured image for learning. Imaging device.   The control means determines the automatic photographing by a neural network based on a condition learned by the learning means, and the teacher outputs a value obtained by inputting the photographed image for learning to the neural network and the teacher The imaging apparatus according to any one of claims 1 to 13, wherein whether or not to cause the imaging unit to capture the learning image is determined based on a difference in teacher values of data.   The control means is configured to input the learning image to the imaging means when a difference between a value output by inputting the photographed image for learning into the neural network and an output value of the teacher data becomes smaller than a predetermined value. The imaging device according to claim 15, wherein the imaging image is not captured.   The control means is configured to teach the photographed image for learning when a difference between a value output by inputting the photographed image for learning into the neural network and a teacher value of the teacher data is equal to or more than a predetermined value. The imaging device according to claim 15, wherein the imaging device is not used as data. The determination of the automatic photographing is performed by a neural network based on the conditions learned by the learning means,
When the difference between the data of the photographed image obtained from the photographing performed by the user's instruction and the data of the photographed image for learning becomes a predetermined value or more in the vector combining the feature quantities of the input layer of the neural network The image pickup apparatus according to any one of claims 1 to 13, wherein the photographed image for learning is not used as teacher data. An acquisition step of acquiring data related to a photographed image photographed by the photographing means;
A learning step of learning conditions of an image preferred by the user based on the teacher data acquired in the acquiring step;
A control step of performing automatic photographing determination by the photographing means based on the conditions learned in the learning step;
A registration step of registering, as the teacher data, data acquired in the acquisition step on a learning photographed image continuously photographed in accordance with a user's instruction;
And a control method of an imaging apparatus.   The program for making a computer perform each process of the control method of Claim 19.   A computer readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 19.

Images