JP2019121855A - Imaging device and control method thereof - Google Patents

Abstract

To provide an imaging device capable of acquiring a user's favorite video without performing a special operation by the user.SOLUTION: The control method includes: a determination step of determining a photographing process on the basis of a detection result of the state of the imaging device and a detection result of the state of a subject; and a learning step of analyzing a plurality of data acquired in at least one of the detection result of the state of the imaging device, the detection result of the state of the subject, and the determined photographing process, extracting regularity, and learning a user's preference according to the extraction result. The determination of the photographing process in the determination step is changed in accordance with learning by the learning step.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an imaging device and a control method thereof.

  In still image / moving image shooting with an imaging device such as a camera, it is common for a user to decide an object to be shot through a finder or the like, check the shooting situation by himself and adjust the framing of the shot image to shoot an image. is there. Such an imaging apparatus is provided with a function of detecting a user's operation error and notifying the user, or detecting an external environment and notifying the user when it is not suitable for photographing. In addition, there has conventionally been a mechanism for controlling the camera to be in a state suitable for photographing.

  There is a life log camera that shoots regularly and continuously without the user giving a shooting instruction to an imaging apparatus that performs shooting by such a user operation (Japanese Patent Application Laid-Open No. 2008-112118). The life log camera is used in a state of being attached to the user's body by a strap or the like, and records a scene that the user sees in daily life as an image at regular time intervals. The life log camera does not shoot at a timing intended by the user, for example, but shoots at fixed time intervals, so that unexpected moments that are not normally shot can be left as images.

Japanese Patent Publication No. 2016-536868 Publication

  However, when the user wears the life log camera, if automatic shooting is performed periodically, there is a case where a video which is not preferred by the user is acquired, and it is not possible to acquire a video at the moment when it is desired to obtain.

  The present invention has been made in view of the above-described problems, and an object thereof is to provide an imaging device capable of acquiring a video favorite of the user without the user performing a special operation.

  In order to achieve the above object, as a technical feature of the present invention, there is provided a control method of an imaging device, which is a determination step of determining a photographing process based on a detection result of a state of an imaging device and a detection result of a subject state. And analyzing a plurality of data obtained in at least one of the detection result of the device state, the detection result of the object state, and the determined photographing process, and extracting regularity, According to the present invention, there is provided a learning step of learning the preference of the user, and the determination of the photographing process of the determination step is changed by the learning by the learning step.

  According to the present invention, it is possible to provide an imaging device capable of acquiring an image desired by the user without the user performing a special operation.

It is a figure which shows an imaging device typically. It is a figure showing composition of an imaging device. It is a figure which shows the structure of an imaging device and an external apparatus. It is a figure which shows the structure of an external apparatus. It is a figure which shows the structure of an imaging device and an external apparatus. It is a figure which shows the structure of an external apparatus. It is a flowchart explaining a 1st control part. It is a flowchart explaining a 2nd control part. 5 is a flowchart illustrating shooting mode processing. It is a flowchart explaining automatic edit mode determination. It is a flow chart explaining automatic edit processing. It is a figure explaining a neural network. It is a figure for demonstrating area division in a photography picture. It is a flowchart explaining a learning mode determination. It is a flowchart explaining a learning process. It is a figure explaining the display process which concerns on this embodiment. It is a figure which shows the structure of a barrel rotation drive part. It is a figure explaining the target position and real position of the to-be-photographed object on an image. 5 is a flowchart illustrating shooting direction change operation detection. FIG. 8 is a view schematically showing shooting direction change operation detection. FIG. 8 is a view schematically showing shooting direction change operation detection. FIG. 8 is a diagram for explaining a shooting direction change operation detection process. FIG. 8 is a diagram for explaining a shooting direction change operation detection process. It is a figure explaining the example of the photography method concerning this embodiment. It is a figure explaining an example of photography method judgment concerning this embodiment. It is a figure explaining the example of the acceleration change of the imaging device concerning this embodiment.

  The best mode for carrying out the present invention is as shown in the following examples.

  In the photographing method, for example, besides the normal photographing as shown in FIG. 24A, the panoramic photographing in the horizontal direction as shown in FIG. 24B and the panorama in the vertical direction as shown in FIG. There is also a method such as shooting. Among the various photographing methods described above, when the user designates the photographing method every time in determining which method to adopt, the user may cause extra time for the user and may miss the photographing opportunity. is there. In addition, there is a possibility that the width of the expression of the user who does not know such an option in the first place may be narrowed. On the other hand, if, for example, imaging is always performed by all imaging methods, extra images are increased, which increases the time and effort of the user's selection.

  Therefore, in the following, without changing the user's special operation, it is possible to reflect the preference of each individual without causing any image blurring or angle-of-field shift, and to perform the photographing operation only when necessary, by merely activating the imaging device. Shows an example of an imaging device that can be used for a long time and can acquire a user's favorite video.

First Embodiment
<Configuration of Imaging Device>
FIG. 1 is a view schematically showing an imaging device of the first embodiment.

  The imaging device 101 illustrated in FIG. 1A is provided with an operation member (hereinafter referred to as a power button, which may be a tap, flick, or swipe operation to a touch panel) that can operate the power switch. There is. A lens barrel 102, which is a housing including an imaging lens group for capturing an image and an imaging element, is attached to the imaging apparatus 101, and is provided with a rotation mechanism capable of rotationally driving the lens barrel 102 with respect to the fixed portion 103. The tilt rotation unit 104 is a motor drive mechanism capable of rotating the lens barrel 102 in the pitch direction shown in FIG. 1B, and the pan rotation unit 105 is a motor drive mechanism capable of rotating the lens barrel 102 in the yaw direction. Thus, the lens barrel 102 can rotate in one or more axes. FIG. 1B is an axis definition at the fixed portion 103 position. Both the angular velocity meter 106 and the accelerometer 107 are mounted on the fixing unit 103 of the imaging device 101. The vibration of the imaging apparatus 101 is detected based on the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit and the pan rotation unit are rotationally driven based on the detected swing angle. Thereby, the shake of the lens barrel 102 which is the movable portion is corrected, and the tilt is corrected.

  FIG. 2 is a block diagram showing the configuration of the imaging device of the present embodiment.

  In FIG. 2, the first control unit 223 includes a processor (for example, a CPU, a GPU, a microprocessor, an MPU, and the like) and a memory (for example, a DRAM, an SRAM, and the like). These units execute various processes to control each block of the imaging apparatus 101 or to control data transfer between the blocks. A non-volatile memory (EEPROM) 216 is an electrically erasable and recordable memory, and stores constants, programs and the like for the operation of the first control unit 223.

  In FIG. 2, the zoom unit 201 includes a zoom lens that performs magnification change. The zoom drive control unit 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens that performs focus adjustment. The focus drive control unit 204 drives and controls the focus unit 203.

  In the imaging unit 206, the imaging device receives light incident through each lens group, and outputs information on charge according to the light amount to the image processing unit 207 as analog image data. The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, color interpolation processing, and the like to digital image data output by A / D conversion, and outputs the applied digital image data. The digital image data output from the image processing unit 207 is converted into a recording format such as JPEG format by the image recording unit 208, and is transmitted to the memory 215 or a video output unit 217 described later.

  The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

  The device shake detection unit 209 is mounted with, for example, an angular velocity meter (gyro sensor) 106 that detects angular velocities in the three axis directions of the imaging device 101, and an accelerometer (acceleration sensor) 107 that detects an acceleration in the three axis directions of the device. . The device shake detection unit 209 calculates the rotation angle of the device, the shift amount of the device, and the like based on the detected signal.

  The audio input unit 213 acquires an audio signal around the imaging device 101 from a microphone provided in the imaging device 101, performs analog-to-digital conversion, and transmits the signal to the audio processing unit 214. The audio processing unit 214 performs processing related to audio such as optimization processing of the input digital audio signal. Then, the first control unit 223 transmits the audio signal processed by the audio processing unit 214 to the memory 215. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing unit 207 and the audio processing unit 214.

  An image processing unit 207 and an audio processing unit 214 read out an image signal and an audio signal temporarily stored in the memory 215 and perform encoding of the image signal, encoding of the audio signal, and the like, and a compressed image signal and a compressed audio signal. Generate The first control unit 223 transmits the compressed image signal and the compressed audio signal to the recording and reproducing unit 220.

  The recording and reproducing unit 220 records, on the recording medium 221, the compressed image signal and the compressed sound signal generated by the image processing unit 207 and the sound processing unit 214, and other control data related to photographing. When the audio signal is not compressed and encoded, the first control unit 223 causes the recording / reproducing unit 220 to transmit the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207. It transmits and makes the recording medium 221 record.

  The recording medium 221 may be a recording medium built in the imaging device 101 or a removable recording medium. The recording medium 221 can record various data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the imaging device 101, and a medium having a larger capacity than the non-volatile memory 216 is generally used. For example, the recording medium 221 includes a recording medium of any type such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a non-volatile semiconductor memory, and a flash memory.

  The recording and reproducing unit 220 reads (reproduces) the compressed image signal, the compressed audio signal, the audio signal, various data, and the program recorded on the recording medium 221. Then, the first control unit 223 transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing unit 207 and the audio processing unit 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode the signal according to a predetermined procedure, and transmit the decoded signal to the video output unit 217 and the audio output unit 218. Do.

  The voice input unit 213 has a plurality of microphones mounted on the imaging apparatus 101, and the voice processing unit 214 can detect the direction of the sound on the plane on which the plurality of microphones are installed. Used. Further, the voice processing unit 214 detects a specific voice command. The voice command may be configured so that the user can register a specific voice in the imaging device in addition to some commands registered in advance. It also performs sound scene recognition. In sound scene recognition, a plurality of data are analyzed in advance based on a large amount of voice data, regularity is extracted, and sound scene determination is performed by a network learned by machine learning according to the extraction result. For example, a network for detecting a specific scene such as “Chaiing,” “Applause,” “Speaking,” etc. is set in the audio processing unit 214. When a specific sound scene or a specific voice command is detected, a detection trigger signal is output to the first control unit 223 or the second control unit 211.

  The first control unit 223 that controls the entire main system of the imaging apparatus 101 is provided separately, and the second control unit 211 controls the power supply of the first control unit 223.

  The first power supply unit 210 and the second power supply unit 212 respectively supply power for operating the first control unit 223 and the second control unit 211. Power is supplied to both the first control unit 223 and the second control unit 211 by pressing the power button provided on the imaging apparatus 101. However, as described later, the first control unit 223 performs the first power supply. It is controlled to turn off its own power supply to the unit 210. Even while the first control unit 223 is not in operation, the second control unit 211 is in operation, and information from the device shake detection unit 209 and the audio processing unit 214 is input. The second control unit is configured to determine whether or not to start the first control unit 223 based on various input information, and when the start determination is made, instructs the first power supply unit to supply power. .

  The audio output unit 218 outputs an audio pattern set in advance from a speaker incorporated in the imaging apparatus 101, for example, at the time of shooting.

  The LED control unit 224 controls, for example, a lighting blink pattern in which an LED provided in the imaging device 101 is preset when photographing or the like.

  The video output unit 217 includes, for example, a video output terminal, and transmits an image signal to display a video on a connected external display or the like. Further, the audio output unit 218 and the video output unit 217 may be one combined terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

  The communication unit 222 performs communication between the imaging apparatus 101 and an external device, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. Also, the imaging device 101 is driven based on an instruction from an external device that can mutually communicate with the imaging device 101 by receiving imaging start and end commands and control signals related to imaging such as pan / tilt and zoom driving. In addition, information such as various parameters related to learning processed by a learning processing unit 219 described later is transmitted and received between the imaging apparatus 101 and an external apparatus. The communication unit 222 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a wireless USB, or a GPS receiver.

<Configuration with external communication device>
FIG. 3 is a diagram showing a configuration example of a wireless communication system of the imaging apparatus 101 and the external apparatus 301. The imaging apparatus 101 is a digital camera having a photographing function, and the external apparatus 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

  The imaging apparatus 101 and the smart device 301 communicate, for example, with a wireless LAN based on the IEEE 802.11 standard series, and control stations and dependent stations such as Bluetooth Low Energy (hereinafter, referred to as "BLE") BLE. It can communicate by the communication 303 which has master-slave relationship. The wireless LAN and BLE are an example of a communication method, and each communication apparatus has two or more communication functions, for example, one communication function that performs communication in the relationship between the control station and the dependent station. Other communication methods may be used as long as control of the other communication function can be performed. However, without loss of generality, the first communication such as a wireless LAN can perform faster communication than the second communication such as BLE, and the second communication consumes more than the first communication. It is assumed that the power is low and / or the communicable distance is short.

  The configuration of the smart device 301 will be described with reference to FIG.

  The smart device 301 includes, for example, a public line control unit 406 for public wireless communication in addition to the wireless LAN control unit 401 for wireless LAN and the BLE control unit 402 for BLE. The smart device 301 further includes a packet transmitting / receiving unit 403. A wireless LAN control unit 401 performs RF control of a wireless LAN, communication processing, a driver that performs various controls of communication by a wireless LAN conforming to the IEEE 802.11 standard series, and protocol processing regarding communication by a wireless LAN. The BLE control unit 402 performs RF control of BLE, communication processing, and protocol processing regarding communication by a driver that performs various controls of communication by BLE and BLE. A public line control unit 406 performs RF control of public wireless communication, communication processing, a driver that performs various controls of public wireless communication, and protocol processing related to public wireless communication. The public wireless communication conforms to, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmission / reception unit 403 performs processing for performing transmission and / or reception of packets related to communication by wireless LAN and BLE and public wireless communication. In this example, the smart device 301 is described as performing at least one of transmission and reception of a packet in communication, but other communication formats such as circuit switching may be used other than packet switching. Good.

  The smart device 301 further includes, for example, a control unit 411, a storage unit 404, a GPS reception unit 405, a display unit 407, an operation unit 408, an audio input audio processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301, for example, by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program executed by the control unit 411 and various information such as parameters required for communication. Various operations described later are realized by the control unit 411 executing a control program stored in the storage unit 404.

  The power supply unit 410 supplies power to the smart device 301. The display unit 407 has a function capable of outputting visually recognizable information, such as an LCD or an LED, or a sound output such as a speaker, and displays various information. The operation unit 408 is, for example, a button for receiving an operation of the smart device 301 by the user. The display unit 407 and the operation unit 408 may be configured by a common member such as a touch panel, for example.

  The voice input voice processing unit 409 may be configured to obtain a voice uttered by the user from, for example, a general-purpose microphone built in the smart device 301, and to obtain a user's operation command by voice recognition processing.

  Also, a voice command is acquired by the user's pronunciation via a dedicated application in the smart device. Then, it can be registered as a specific voice command for causing the voice processing unit 214 of the imaging apparatus 101 to recognize a specific voice command via the communication 302 by wireless LAN.

  A GPS (Global Positioning System) 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude / latitude information) of the smart device 301. Alternatively, position estimation may estimate the current position of the smart device 301 based on information of a wireless network present in the vicinity using a Wi-Fi Positioning System (WPS) or the like. When the acquired current GPS position information is located in a position range (within a predetermined radius range) set in advance, movement information is notified to the imaging apparatus 101 via the BLE control unit 402, and will be described later. Use as a parameter for automatic shooting and editing. In addition, when there is a position change of predetermined position or more in the GPS position information, the movement information is notified to the imaging apparatus 101 via the BLE control unit 402, and used as a parameter for automatic photographing and automatic editing described later.

  As described above, the imaging apparatus 101 and the smart device 301 exchange data with the imaging apparatus 101 by communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, it transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. Further, the smart device issues an operation instruction such as photographing of the imaging apparatus 101, transmission of voice command registration data, notification of predetermined position detection based on GPS position information, and notification of location movement. It also sends and receives learning data via a dedicated application in the smart device.

<Configuration of accessories>
FIG. 5 is a diagram showing a configuration example of an external apparatus 501 that can communicate with the imaging apparatus 101. The imaging apparatus 101 is a digital camera having a photographing function, and the external apparatus 501 is a wearable device including various sensing units capable of communicating with the imaging apparatus 101 by, for example, a Bluetooth communication module.

  The wearable device 501 is configured to be attached to, for example, a user's arm, and is a sensor that detects biological information such as the user's pulse, heart rate, blood flow, etc. at a predetermined cycle, and an acceleration sensor that can detect the user's exercise state. Etc. are mounted.

  The biological information detection unit 502 detects, for example, a change in electric potential detected by contact of the skin with a conductive polymer, a pulse sensor that detects a pulse, a heartbeat sensor that detects a heartbeat, a blood flow sensor that detects blood flow, Include sensors that The present embodiment will be described using a heart rate sensor as the biological information detection unit 502. The heart rate sensor irradiates the skin with infrared light using, for example, an LED or the like, and detects the heart rate of the user by detecting the infrared light transmitted through the body tissue with a light receiving sensor and processing the signal. The biological information detection unit 502 outputs the detected biological information as a signal to a control unit 607 described later.

  For example, an acceleration sensor or a gyro sensor is mounted on the shake detection unit 503 that detects the motion state of the user, and it is determined whether the user is moving based on the acceleration information or whether the user is swinging an arm And other motions can be detected.

  In addition, an operation unit 505 that receives an operation of the wearable device 501 by the user, and a display unit 504 that outputs visually recognizable information such as an LCD or an LED are mounted.

  The configuration of the wearable device 501 will be described with reference to FIG.

  The wearable device 501 includes, for example, a control unit 607, a communication unit 601, a biological information detection unit 502, a shake detection unit 503, a display unit 504, an operation unit 505, a power supply unit 606, and a storage unit 608.

  The control unit 607 controls the entire wearable device 501, for example, by executing a control program stored in the storage unit 608. The storage unit 608 stores, for example, a control program executed by the control unit 607 and various information such as parameters required for communication. Various operations described later are realized, for example, by the control unit 607 executing a control program stored in the storage unit 608.

  The power supply unit 606 supplies power to the wearable device 501. The display unit 504 has a function capable of outputting visually recognizable information such as an LCD or an LED, or a sound output such as a speaker, and displays various information. The operation unit 505 is, for example, a button for receiving an operation of the wearable device 501 by the user. The display unit 504 and the operation unit 505 may be configured by a common member such as a touch panel, for example.

  In addition, the operation unit acquires voice uttered by the user from, for example, a general-purpose microphone built in the wearable device 501, acquires voice uttered by the user by voice processing, and receives user's operation instruction by voice recognition processing. May be acquired.

  The various detection information processed by the control unit 607 from the biological information detection unit 502 or the shake detection unit 503 transmits the detection information to the imaging apparatus 101 by the communication unit 601.

  For example, the detection information is transmitted to the imaging apparatus 101 at the timing at which a change in the user's heartbeat is detected, or the detection information is transmitted at the change timing of the movement state such as walking movement / running movement / stop. Also, for example, detection information is transmitted at a timing at which a pre-set arm-like motion is detected, or detection information is transmitted at a timing at which movement of a preset distance is detected.

<Sequence of imaging operation>
FIG. 7 is a flowchart for explaining an example of the operation of the first control unit 223 of the imaging apparatus 101 according to the present embodiment.

  When the user operates a power button provided on the imaging apparatus 101, the first power supply unit 210 causes the power supply unit to supply power to each block of the first control unit 223 and the imaging apparatus 101.

  Similarly, in the second control unit 211, power is supplied to the second control unit from the power supply unit by the second power supply unit 212. The details of the operation of the second control unit will be described later with reference to the flowchart of FIG. This will be described using

When the power is supplied, the process of FIG. 7 starts. In step 701, the start condition is read. In the present embodiment, the activation condition is as follows.
(1) The power button is manually pressed to start power (2) Power from external device (for example, 301) according to an instruction from external communication (for example, BLE communication) (3) From sub processor (second control unit 211) Power on

  Here, in the case of power activation from the Sub processor (3), the activation condition calculated in the Sub processor is read, but the details will be described later with reference to FIG.

  Also, the start condition read in here is used as one parameter element at the time of subject search and automatic photographing, which will be described later. When the start condition reading is completed, the process proceeds to step 702.

  In step 702, various sensors are read. The sensor read in here is a sensor that detects vibration such as a gyro sensor from the apparatus shake detection unit 209 or an acceleration sensor. In addition, the rotational position of the tilt rotation unit 104 or the pan rotation unit 105 may be set. In addition, it may be a voice level detected by the voice processing unit 214, a detection trigger of specific voice recognition, or a sound direction detection.

  Further, although not shown in FIGS. 1 to 6, a sensor that detects environmental information also acquires information.

  For example, there are a temperature sensor that detects the temperature around the imaging device 101 at a predetermined cycle, and an air pressure sensor that detects a change in air pressure around the imaging device 101. In addition, an illuminance sensor that detects the brightness around the imaging device 101, a humidity sensor that detects the humidity around the imaging device 101, a UV sensor that detects the amount of ultraviolet light around the imaging device 101, or the like may be provided. . In addition to the detected temperature information, barometric pressure information, brightness information, humidity information, and UV information, the temperature change amount, pressure change amount, brightness change amount, humidity change, etc. are calculated by calculating the change rate at predetermined time intervals from the detected various information. It is used for determination of automatic photographing etc. which mention an amount, an ultraviolet ray variation amount, etc. later.

  If various sensors are read in step 702, the process proceeds to step 703.

  In step 703, it is detected whether communication from the external device is instructed, and when communication is instructed, communication with the external device is performed.

  For example, data such as an audio signal, an image signal, a compressed audio signal, or a compressed image signal is transmitted or received from the smart device 301 via wireless LAN or BLE. In addition, whether there is an operation instruction such as shooting of the imaging device 101 from the smart device 301, transmission of voice command registration data, an instruction of transmission / reception of predetermined position detection notification based on GPS position information, location movement notification and learning data Do you read?

  Also, for example, when there is an update of biological information such as user's exercise information, arm action information, and heartbeat from the wearable device 501, information is read via BLE. The various sensors for detecting the environmental information described above may be mounted on the imaging apparatus 101, or may be mounted on the smart device 301 or the wearable device 501. In that case, the environmental information is read via BLE Do also. When communication reading from the external device is performed in step 703, the process proceeds to step S704.

  At step 704, mode setting determination is performed. The mode set in step 704 is determined and selected from the following.

(1) Automatic shooting mode [mode judgment condition]
From each detection information (image, sound, time, vibration, location, change in the body, environmental change) set by learning described later, elapsed time after transition to the automatic shooting mode, and past shooting information, etc. When it is determined that the automatic shooting should be performed, the automatic shooting mode is set.

[In-mode processing]
In the automatic photographing mode processing (step 710), pan / tilt and zoom are driven to automatically search for a subject based on each detection information (image, sound, time, vibration, place, body change, environment change). Then, when it is determined that it is the timing at which the user's favorite shooting can be performed, determination of the shooting method among various shooting methods such as single still image shooting, still image continuous shooting, moving image shooting, panoramic shooting, time-lapse shooting Processing is performed and photographing is performed automatically.

(2) Automatic editing mode [mode judgment condition]
If it is determined that automatic editing should be performed from the elapsed time since the previous automatic editing and past captured image information, the automatic editing mode is set.

[In-mode processing]
In the automatic editing mode processing (step 712), still image and moving image selection processing based on learning is performed, and a highlight moving image summarized into one moving image by image effect and time of edited moving image based on learning, etc. The automatic editing process to be created is performed.

(3) Image transfer mode [mode judgment condition]
If it is determined that automatic image transfer should be performed based on the elapsed time since the previous image transfer and the past captured image information if the automatic image transfer mode is set by an instruction via a dedicated application in the smart device , Automatic image transfer mode is set.

[In-mode processing]
In the image automatic transfer mode processing (step 714), the imaging apparatus 101 automatically extracts an image that may be the user's preference, automatically extracts the user's favorite image to the smart device 301, and image transfer is performed. The user's preference image extraction is performed using a score that determines the user preference added to each image described later.

(4) Learning mode [mode judgment condition]
If it is determined that automatic learning should be performed from the elapsed time since the previous learning process and information associated with images that can be used for learning, the number of learning data, etc., the automatic learning mode is set. Be done. Alternatively, this mode is also set when there is an instruction to set learning data via communication from the smart device 301.

[In-mode processing]
In the automatic learning mode process (step 716), learning is performed according to the user's preference. A plurality of data are analyzed using a neural network based on information such as each operation in the smart device 301, and learning information notification from the smart device 301, regularity is extracted, and user's preference according to the extraction result Learning is conducted in accordance with As information on each operation in the smart device 301, for example, image acquisition information from the imaging apparatus, information on which a manual editing instruction is given via a dedicated application, and a determination value input by the user with respect to the image in the imaging apparatus There is information.

  Further, learning about detection such as registration of personal authentication, voice registration, sound scene registration, general object recognition registration, etc., and learning of the conditions of the above-mentioned low power consumption mode, etc. are simultaneously performed.

(5) Automatic file deletion mode [Mode judgment condition]
If it is determined that automatic file deletion should be performed from the elapsed time since the previous automatic file deletion and the remaining capacity of the non-volatile memory 216 storing the image, the automatic file deletion mode is set.

[In-mode processing]
In the file automatic deletion mode processing (step 718), a file to be automatically deleted is specified (three-option processing) from the tag information of each image and the date and time of photographing from the images in the non-volatile memory 216.

  The details of the automatic shooting mode processing, the automatic editing mode processing, and the learning mode processing will be described later.

  In step 705, it is determined in step 704 whether the mode setting determination is set to the low power consumption mode. The low power consumption mode determination is not a judgment condition of any mode of "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode" and "automatic file deletion mode" described later. If so, it is determined to be in the low power consumption mode. When the determination process is performed, the process proceeds to step 705.

  In step 705, if it is determined that the low power consumption mode condition is satisfied, the process proceeds to step 706.

  In step 706, various parameters relating to the activation factor to be determined in the Sub processor (the shake detection determination parameter, the sound detection parameter, the time lapse detection parameter) are notified to the Sub processor (second control unit 211). Various parameters change in value by being learned in a learning process described later. When the process of step 706 is completed, the process proceeds to step 707, the power of the main processor (first control unit 223) is turned off, and the process is ended.

  If it is determined in step 705 that the mode is not the low power consumption mode, the process proceeds to step 709 to determine whether the mode setting is the automatic shooting mode. If the mode setting is the automatic shooting mode, the process proceeds to step 710; Processing is performed. When the process ends, the process returns to step 702 and the process is repeated. If it is determined in step 709 that the automatic photographing mode is not set, the process proceeds to step 711.

  In step 711, it is determined whether the mode setting is the automatic editing mode. If the mode is the automatic editing mode, the process proceeds to step 712 where the automatic editing mode processing is performed. When the process ends, the process returns to step 702 and the process is repeated. If it is determined in step 711 that the automatic editing mode is not set, the process proceeds to step 713.

  In step 713, it is determined whether the mode setting is the automatic image transfer mode. If the automatic image transfer mode is selected, the process proceeds to step 714, where the automatic image transfer mode processing is performed. When the process ends, the process returns to step 702 and the process is repeated. If it is determined in step 713 that the automatic image transfer mode is not set, the process proceeds to step 715.

  In step 715, it is determined whether the mode setting is the learning mode. If the mode setting is the learning mode, the process proceeds to step 716, and learning mode processing is performed. When the process ends, the process returns to step 702 and the process is repeated. If it is determined in step 715 that the learning mode is not set, the process proceeds to step 717.

  In step 717, it is determined whether the mode setting is the file automatic deletion mode. If the mode is the file automatic deletion mode, the process proceeds to step 718, where the file automatic deletion mode processing is performed. When the process ends, the process returns to step 702 and the process is repeated. If it is determined in step 717 that the learning mode is not set, the process returns to step 702 to repeat the processing.

  FIG. 8 is a flowchart for explaining an example of the operation of the second control unit 211 of the imaging apparatus 101 according to the present embodiment.

  When the user operates the power button provided on the imaging apparatus 101, the second power supply is also supplied to the second control unit 211 in the same manner as the first power supply unit 210 supplies power from the power supply unit to the first control unit 223. The power supply unit supplies power to the second control unit 211 by the unit 212. When power is supplied, the Sub processor (second control unit 211) is activated, and the process of FIG. 8 starts.

  In step 801, it is determined whether or not a predetermined period which is a sampling cycle has elapsed. For example, if it is set to 10 msec, the process proceeds to step 802 in a 10 msec cycle. If it is determined that the predetermined period has not elapsed, the Sub processor returns to step 801 without performing any processing, and waits for the predetermined period to elapse.

In step 802, learning information is read. The learning information is information transferred when performing information communication with the Sub processor in step 706 of FIG. 7, and, for example, the following information is read.
(1) Judgment condition of specific vibration detection (2) Judgment condition of specific sound detection (3) Judgment condition of time lapse judgment

  When the learning information is read in step 802, the process proceeds to step 803, where a shake detection value is obtained. The shake detection value is an output value from a sensor that detects vibration such as a gyro sensor or an acceleration sensor from the device shake detection unit 209.

  When the shake detection value is acquired in step 803, the process proceeds to step 804, and processing of shake state detection set in advance is performed. Here, the determination process is changed according to the learning information read in step S802. Some examples are described.

(1) Tap Detection A state (tap state) in which the user taps the imaging device 101 with, for example, a fingertip can be detected from an output value of an acceleration sensor attached to the imaging device 101. By passing the output of the three-axis acceleration sensor through a band pass filter (BPF) set to a specific frequency region by predetermined sampling, a signal region of acceleration change due to a tap can be extracted. Tap detection is performed based on whether the number of times the acceleration signal after BPF exceeds the predetermined threshold value ThreshA is the predetermined number of times CountA during the predetermined time TimeA. For double taps, CountA is set to 2, and for triple taps, CountA is set to 3. Further, Time A and Thresh A can also be changed by learning information.

(2) Detection of Swing State The swing state of the imaging device 101 can be detected from the output value of a gyro sensor or an acceleration sensor attached to the imaging device 101. The output of the gyro sensor or the acceleration sensor is cut by HPF and the low frequency component is cut by LPF, and then the absolute value conversion is performed. The vibration detection is performed based on whether or not the number of times the calculated absolute value exceeds the predetermined threshold value ThreshB is equal to or more than the predetermined number of times CountB during the predetermined time TimeB. For example, it is possible to determine whether the shake is small when the imaging apparatus 101 is placed on a desk or the like, or is large when the imaging apparatus 101 is worn and walked with a wearable. Moreover, it is also possible to detect a fine shaking state according to the shaking level by having a plurality of conditions of the judgment threshold value and the judgment count number.

  Time B, Thresh B, and Count B can also be changed by learning information.

  The above has described the method of specific swing state detection by condition determination of the swing detection sensor. However, from the data of the shake detection sensor sampled within a predetermined time, a specific shake state registered in advance by the learned neural network by inputting to the shake state determination unit using the neurus network It is also possible to detect. In that case, the learning information reading in step 802 is the weight parameter of the neural network.

  When the specific swing state detection process is performed in step 804, the process proceeds to step 805, and a predetermined specific sound detection process is performed. Here, the detection determination process is changed according to the learning information read in step S802. Some examples are described.

(1) Specific voice command detection A specific voice command is detected. The voice command allows the user to register a specific voice in the imaging device, in addition to several commands registered in advance.

(2) Specific Sound Scene Recognition A large amount of voice data is analyzed in advance to extract regularity, and sound scene determination is performed by a network learned by machine learning according to the extraction result. For example, it detects a specific scene such as "Cheers up", "Applause", "Speaking" or the like. The scene to detect changes with learning.

(3) Sound Level Determination Detection based on sound level determination is performed by adding time during which the magnitude of the sound level exceeds the level predetermined value during a predetermined time. The magnitude of the predetermined time, the level predetermined value, and the like change by learning.

(4) Sound Direction Determination It is possible to detect the direction of sound on a plane where a plurality of microphones are installed, and to detect the direction of sound with respect to a sound level of a predetermined magnitude.

  The above-described determination processing is performed in the voice processing unit 214, and it is determined in step 805 whether a specific sound is detected by each setting learned in advance.

  When the specific sound detection process is performed in step 805, the process proceeds to step 806. In step 806, it is determined whether the Main processor (first control unit 223) is in the OFF state. If the Main processor is in the OFF state, the process proceeds to step 807, and processing for detecting a lapse of time set in advance is performed. Here, the detection determination process is changed according to the learning information read in step S802. The learning information is information transferred when performing information communication with the Sub processor (second control unit 211) in step 706 described with reference to FIG. The elapsed time from when the main processor transits from ON to OFF is measured, and if the elapsed time is greater than or equal to the parameter TimeC, it is determined that the time has elapsed, and if smaller than TimeC, it is not determined that the time has elapsed. TimeC is a parameter that changes according to learning information.

When the time lapse detection process is performed in step 807, the process proceeds to step 808, in which it is determined whether the low power consumption mode cancellation determination has been made. The low power consumption mode release condition is determined by the following.
(1) Judgment condition of specific vibration detection (2) Judgment condition of specific sound detection (3) Judgment condition of time lapse judgment

  In each specific swing state detection process at step 804, it can be determined whether or not the determination condition of the specific swing detection has been entered. Further, by the specific sound detection process in step 805, it can be determined whether or not the determination condition of the specific sound detection is entered. Further, by the time lapse detection processing in step 807, it can be determined whether or not the determination condition of time lapse detection is entered. Therefore, if any one or more conditions are entered, it is determined that the low power consumption mode is canceled.

  When the release condition is determined in step 808, the process proceeds to step 809 to turn on the power of the main processor, and in step 810, the main processor is notified of the condition (shake, sound, time) determined to be the low power consumption mode release. The flow returns to step 801 to loop the process.

  If it is determined in step 808 that none of the release conditions are met and it is determined that the low power consumption mode release determination is not made, the process returns to step 801 to loop processing.

  If it is determined in step 806 that the main processor is in the ON state, the information acquired in steps 803 to 805 is notified to the main processor, and the process returns to step 801 to loop processing.

  In the present embodiment, even when the Main processor is in the ON state, vibration detection and specific sound detection are performed by the Sub processor, and the detection result is notified to the Main processor. However, when the Main processor is ON, the processing in steps 803 to 805 may not be performed, and the fluctuation detection or the specific sound detection may be detected in the processing in the main processor (step 702 in FIG. 7).

  By performing steps 704 to 707 of FIG. 7 and the process of FIG. 8 as described above, the conditions for transitioning to the low power consumption mode and the conditions for canceling the low power consumption mode are learned based on the user's operation. Ru. Thus, the imaging operation can be performed according to the convenience of the user who owns the imaging device 101. The method of learning will be described later.

  Although the low power consumption mode cancellation method based on vibration detection, sound detection, and passage of time has been described in detail, the low power consumption mode cancellation may be performed by environmental information. Environmental information can be determined based on whether the absolute amount or change amount of temperature, atmospheric pressure, brightness, humidity, or ultraviolet light amount exceeds a predetermined threshold, and the threshold can be changed by learning described later.

  In addition, it is possible to judge the low power consumption mode cancellation by judgment based on neural network based on judgment of vibration detection, sound detection, detection information of elapsed time, absolute value or change amount of each environment information, and this judgment processing is The determination conditions can be changed by learning described later.

<Automatic shooting mode processing>
The details of the automatic shooting mode process will be described with reference to FIG. As described above, the following processing is controlled by the first control unit 223 of the imaging apparatus 101 in the present embodiment.

  In step S901, the image processing unit 207 performs image processing on the signal acquired by the imaging unit 206 to generate an image for object recognition.

  From the generated image, subject recognition such as human and object recognition is performed.

  When recognizing a person, the face or human body of the subject is detected. In the face detection process, a pattern for determining the face of a person is determined in advance, and a portion matching the pattern included in the captured image can be detected as a face image of the person.

  Further, the reliability indicating the certainty as the face of the subject is also calculated at the same time, and the reliability is calculated from, for example, the size of the face area in the image, the degree of coincidence with the face pattern, and the like.

  Similarly for object recognition, it is possible to recognize an object that matches the pattern registered in advance.

  Further, there is also a method of extracting a characteristic subject by a method of using a histogram such as hue or saturation in a captured image. In this case, with respect to the image of the subject captured within the shooting angle of view, the process of dividing the distribution derived from the histogram of the hue, saturation, etc. into a plurality of sections and classifying the captured images into sections is performed. Be done.

  For example, a histogram of a plurality of color components is created for the captured image, divided by the mountain-shaped distribution range, the captured image is classified in the region belonging to the combination of the same sections, and the image region of the subject is Be recognized.

  By calculating the evaluation value for each image area of the recognized subject, it is possible to determine the image area of the subject having the highest evaluation value as the main subject area.

  Each subject information can be obtained from the imaging information by the above method.

  In S902, the image shake correction amount is calculated. Specifically, first, the absolute angle of the imaging device is calculated based on the angular velocity and the acceleration information acquired by the device shake detection unit 209. Then, an image stabilization angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angular direction that cancels out the absolute angle is determined, and is used as the image shake correction amount. In the image shake correction amount calculation processing here, the calculation method can be changed by learning processing described later.

  In step S903, the state determination of the imaging apparatus is performed. Based on angular velocity information, acceleration information, angle detected by GPS position information, etc., it is determined what kind of vibration / motion state of the imaging device is at present.

  For example, in the case where the imaging apparatus 101 is attached to a car and shooting is performed, subject information such as a surrounding scenery changes largely depending on the moved distance.

  Therefore, it is possible to determine whether it is a "vehicle moving state" which is attached to a car or the like and moving at a high speed, and can be used for automatic subject search described later.

  Further, it is determined whether or not the change in angle is large, and it is determined whether or not the imaging device 101 is in the “posting state” with almost no swing angle.

  In the case of the “post-taking state”, it may be considered that there is no change in the angle of the imaging device 101 itself, so that it is possible to perform a subject search for the post-taking.

  Also, if the change in angle is relatively large, it is determined that the "hand-held state", and it is possible to perform a hand-held subject search.

  In S904, a subject search process is performed. The subject search is configured by the following processing.

(1) Area Division The area division will be described with reference to FIG. As shown in FIG. 13A, area division is performed on the entire periphery around the position of the imaging device (the origin O is the imaging device position). In the example of FIG. 13A, division is performed at 22.5 degrees in each of the tilt direction and the pan direction. When division is performed as shown in FIG. 13A, as the angle in the tilt direction deviates from 0 degree, the circumference in the horizontal direction becomes smaller and the area area becomes smaller. Therefore, as shown in FIG. 13B, when the tilt angle is 45 degrees or more, the area range in the horizontal direction is set larger than 22.5 degrees. FIGS. 13 (c) and 13 (d) show examples of area division within the shooting angle of view. An axis 1301 is the direction of the imaging apparatus 101 at the time of initialization, and area division is performed with this direction angle as a reference position. Reference numeral 1302 denotes a field angle area of the image being captured, and an example of the image at that time is shown in FIG. In the image projected at the angle of view, the image is divided as shown by 1303 to 1318 in FIG. 13D based on the area division.

(2) Calculation of Importance Level for Each Area For each area divided as described above, the importance level indicating the priority to be searched is calculated according to the scene status of the subject present in the area and the area. The importance level based on the condition of the subject is, for example, the number of persons present in the area, the size of the face of the person, the face direction, the certainty of face detection, the facial expression of the person, and the personal authentication result of the person. calculate. In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene discrimination result (blue sky, back light, sunset scene etc.), sound level from the direction of area and voice recognition result, motion detection in area It is information etc. In addition, the vibration state of the imaging device is detected in the state determination (S903) of the imaging device, and the importance level can be changed according to the vibration state. For example, when it is determined that the “deep-shooting state” is selected, a subject search is performed so that a subject search is performed centering on a high-priority subject (for example, the user of the imaging apparatus) registered in face authentication. When face recognition is detected, it is determined that the importance level is high. In addition, the automatic photographing described later is also performed with priority given to the face, and even if the user of the imaging apparatus wears the imaging apparatus and takes a lot of time taking pictures, the imaging apparatus is removed and the desk is By placing it on top, it is possible to leave many images captured by the user. At this time, it is possible to search by pan / tilt, so even if the installation angle of the imaging device is not considered, it is possible to leave an image photographed by the user or a group photograph with many faces photographed only by installing appropriately. it can. In addition, under the above conditions alone, as long as there is no change in each area, the area with the highest importance level is the same, and as a result, the area to be searched remains unchanged. Therefore, the importance level is changed according to the past shooting information. Specifically, the area continuously designated as the search area for a predetermined time is lowered in importance level, or in the area photographed in S910 to be described later, the importance level is lowered for a predetermined time. It is also good.

(3) Determination of Search Target Area When the importance level of each area is calculated as described above, an area having a high importance level is determined as a search target area. Then, the pan / tilt search target angle required to capture the search target area at the angle of view is calculated.

  In S905, pan and tilt driving is performed. Specifically, the pan / tilt drive amount is calculated by adding the drive shake amount at control sampling based on the image shake correction amount and the pan / tilt search target angle, and the lens barrel rotation drive unit 205 performs tilt rotation. It drives and controls the unit 104 and the pan rotation unit 105 respectively.

  In step S906, the zoom unit 201 is controlled to perform zoom driving. Specifically, the zoom is driven according to the state of the search target subject determined in S904. For example, when the search target subject is the face of a person, if the face on the image is too small, detection is not possible because the size is smaller than the minimum detectable size, and there is a risk of losing sight. In such a case, control is performed to increase the size of the face on the image by zooming to the telephoto side. On the other hand, when the face on the image is too large, the subject or the imaging apparatus itself easily moves the subject out of the angle of view. In such a case, control is performed to reduce the size of the face on the screen by zooming to the wide angle side. By performing zoom control in this manner, it is possible to maintain a state suitable for tracking an object.

  In S904 to S906, although the method for performing subject search by pan / tilt or zoom driving has been described, the subject search may be performed with an imaging system that captures an image in all directions at once using a plurality of wide-angle lenses. In the case of an omnidirectional camera, if all signals obtained by imaging are used as an input image and image processing such as object detection is performed, a vast amount of processing is required. Therefore, a part of the image is cut out, and the search processing of the subject is performed in the cut out image range. Similar to the method described above, the important level for each area is calculated, the cutout position is changed based on the important level, and the automatic photographing determination described later is performed. This makes it possible to reduce power consumption and perform high-speed object search by image processing.

  In S907, it is determined whether or not there is a shooting instruction from the user (manual). If there is a shooting instruction, the process proceeds to S908a. If there is no shooting instruction, the process proceeds to S908b. At this time, the photographing instruction by the user (manually) may be performed by pressing the shutter button, tapping the housing of the imaging device with a finger or the like (tap), voice command input, an instruction from an external device, or the like. The shooting instruction by the tap operation is a shooting instruction method in which when the user taps the housing of the imaging device, the device shake detection unit 209 detects the acceleration of high frequency which has continued in a short period, and uses it as a trigger for shooting. The voice command input is a shooting instruction method of recognizing a voice by the voice processing unit 214 and using it as a trigger for shooting when the user utters a combination (for example, "take a photo" or the like) instructing a predetermined shooting. The instruction from the external device is, for example, a photographing instruction method using as a trigger a shutter instruction signal transmitted through a dedicated application from a smartphone or the like connected to the imaging device and BlueTooth.

  In S908b, automatic shooting determination is performed. In the automatic shooting determination, it is determined whether automatic shooting is to be performed and the shooting method (any one of still image shooting, still image continuous shooting (continuous shooting), moving image shooting, panorama shooting, time-lapse shooting, etc. Make a decision. In S 908 a, since the determination itself of whether or not to perform imaging is determined in S 907, only the determination of the imaging method described later is performed.

(1) Determination of Whether to Perform Automatic Shooting Determination of whether to perform automatic shooting is performed based on the following two determinations. First, based on the importance level for each area obtained in S904, when the importance level exceeds a predetermined value, it is determined to perform automatic shooting. The second is determination based on a neural network. An example of a multi-layer perceptron network is shown in FIG. 12 as an example of a neural network. A neural network is used to predict an output value from an input value, and learns in advance an input value and an output value that is an exemplar for the input to obtain a new input value. On the other hand, it is possible to estimate an output value that follows the learned model. The method of learning will be described later. In FIG. 12, 1201 and its vertically arranged circles are neurons in the input layer, 1203 and its vertically arranged circles are neurons in the intermediate layer, and 1204 are neurons in the output layer. Arrows such as 1202 indicate connections connecting each neuron. In the judgment based on the neural network, to the neurons of the input layer, an object based on the current view angle, a feature value based on the scene or the state of the imaging device is given as an input, and an operation based on the forward propagation law of the multilayer perceptron To obtain the value output from the output layer. Then, if the value of the output is equal to or greater than the threshold value, it is determined to perform automatic photographing. The characteristics of the subject include the current zoom magnification, the result of general object recognition at the current angle of view, the result of face detection, the number of faces in the current angle of view, the degree of smile and closing of the face, face angle, and face authentication ID The number, the gaze angle of the subject person, the scene discrimination result, the detection result of the specific composition, and the like are used. Also, use the elapsed time since the last shooting, the current time, GPS position information, the amount of change from the last shooting position, the current voice level, the person making a voice, whether applause or cheering has risen, etc. May be Also, vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet light) or the like may be used. Furthermore, when there is information notification from the wearable device 501, notification information (user's exercise information, arm's action information, biological information such as heart beat, etc.) may also be used as a feature. This feature is converted into a numerical value of a predetermined range, and given to each neuron of the input layer as a feature amount. Therefore, each neuron of the input layer is required as many as the number of feature quantities to be used.

  The judgment based on this neural network can change the output value by changing the connection weight between the neurons by learning processing described later, and the result of the judgment can be adapted to the learning result.

  Further, depending on the start condition of the Main processor read in step 702 of FIG. For example, in the case of activation by tap detection or activation by a specific voice command, there is a very high possibility that the operation is for the user to want to take a picture at present. Therefore, the frequency of photographing is set to be increased.

(2) Determination of shooting method In the determination of the shooting method, still image shooting, moving image shooting, continuous shooting, panorama shooting, etc. based on the state of the imaging apparatus and the state of the subject in the peripheral area detected in S901 to S904. Determine which one to execute. For example, when the subject (person) is stationary, still image shooting is performed, and when the subject is moving, moving image shooting or continuous shooting is performed. In the case where a plurality of subjects exist so as to surround the imaging apparatus, or when it is determined that the scenery is scenic based on the above-described GPS information, images taken sequentially while operating the pan and tilt are A panoramic imaging process may be performed to generate a panoramic image by combining. As in the determination method of <determination of whether or not to perform automatic imaging>, the imaging method can also be determined by determining various information detected before imaging based on a neural network. The determination conditions can be changed by learning processing described later.

  In addition, in the case of the determination in the photographing method, in the case of S908a, since it is performed based on the manual photographing instruction from the user, it is also possible to determine the photographing method according to the user's intention. Hereinafter, determination processing of the imaging method reflecting the user intention will be described with reference to FIG.

<Determination of the imaging method based on the manual imaging instruction by the user>
FIG. 25 shows an example of how to hold the camera and the state of the subject when the user issues a manual shooting instruction.

  First, FIG. 25 (a) shows a state in which the user wears the camera and shoots a little while projecting forward for shooting a thing in front of the eyes. This may be, for example, a scene in which the appearance of the visited store is photographed as a note. In such a scene, for example, it is desirable that the user usually shoot a still image. On the other hand, FIG. 25 (b) shows a state in which the user turns the camera lowered from the neck into a hand-held camera and shoots while raising the camera forward and upward in order to shoot a scenic site. This can be considered, for example, a scene that shoots an open scene such as a mountain observation deck. In such a scene, for example, even if panorama shooting is expected or not, the user knows such a shooting method by presenting the image obtained by the panorama shooting to the user, and for future shooting. It becomes possible to take advantage of it. Therefore, in order to perform these determinations, for example, by detecting the moving distance of the camera from the holding state to shooting, it is a scene as shown in FIG. 25 (a) or a scene as shown in FIG. 25 (b). It is possible to determine the

  As described above, the preferred photographing method may differ depending on the manner of holding the camera, so that it is possible to switch the photographing method according to the state of the camera at the time of photographing.

  Next, FIG. 25C will be described. In FIG. 25 (c), the way of holding the camera is similar to that of FIG. 25 (b). However, in FIG. 25C, it is assumed that the subject is slightly above, and a scene to be taken up and photographed to shoot it. As the purpose of the user in such a case, for example, as shown in FIG. 25 (a), it is desirable that the user usually shoots a still image. Therefore, in order to perform these determinations, it is possible to determine, for example, a scene as shown in FIG. 25B or a scene as shown in FIG. 25C by detecting the subject distance.

  As described above, since the preferred imaging method may differ depending on the state of the subject, the imaging method can be switched according to the state of the camera at the time of imaging and the state of the viewed object.

  Such a determination method can be further applied to the cases as shown in FIGS. 25 (d) and 25 (e).

  FIG. 25 (d) shows a state in which the user switches the camera lowered from the neck to the hand and shoots while holding the camera upward, in order to shoot a vertically long object in front of the eyes. This can be considered, for example, a scene where a high-rise building is photographed at a sightseeing spot. In such a scene, for example, even if vertical panoramic shooting is expected or not, the user can display such a photographing method by presenting the user with an image obtained by vertical panoramic shooting. Knowing, it will be possible to use for future shooting. Therefore, in such a case, for example, by detecting the holding angle of the camera, it is determined whether it is a scene as shown in FIGS. 25 (a) to 25 (c) or a scene as shown in FIG. 25 (d). It becomes possible. It should be noted that, by further determining, for example, the distance to the subject and the distance to the subject in the upper and lower, left and right regions of the subject as the state of the subject at this time, it is said which of panoramic portrait and landscape panoramic is preferred. It is also possible to improve the accuracy of discrimination. That is, if the distance between the subject and its upper and lower regions is at the same distance, it is possible to make a determination such as performing panoramic photography in the vertical direction.

  FIG. 25E shows a state in which the user turns the camera lowered from the neck into a hand-held camera and shoots while holding the camera directly above for shooting 360 degrees. This may be, for example, a scene in which an image is taken to look around at the top of a mountain. At this time, for example, it is conceivable that the user sets the 360-degree imaging mode using an external device and issues an imaging instruction. Therefore, for example, when it is attempted to operate the external device in such a case, it is possible to reduce the time and effort of the user's operation by presenting a UI that covers the transition to 360-degree imaging. Furthermore, while repeating such a thing several times, the user is expected to be able to take a 360-degree picture just by holding it up and pressing the shutter button without having to operate the external device. . Therefore, in such a case, for example, if the moving direction of the camera from the holding state to the shooting is the upward direction, 360-degree shooting is performed to reduce the time and effort of the user for shooting. It becomes possible.

  As described above, by switching the photographing method based on the state of the camera and the state of the subject, it is possible to reduce the time and effort of the user at the time of photographing and at the time of photographing image confirmation.

  Also in the determination of the expected photographing method according to the state of the camera and the state of the subject, the judgment based on the neural network is performed similarly to the determination method of <determination of whether or not to perform automatic photographing>. It is also possible to determine the shooting method. Further, in this determination processing, it is also possible to change the determination condition for each user by learning processing described later. In such a case, in the initial stage of learning, a plurality of images are recorded by a plurality of photographing methods, and in the learning process described later, the determination condition is changed according to which photographing method the user preferred. It is possible to

  As described above, although there has been described the determination of the imaging method according to the user's intention when there is a manual imaging instruction from the user, such processing is an automatic imaging determination when there is no manual imaging instruction such as S908b. It is possible to apply also in processing. That is, when it is determined that the user holds the camera, it is also possible to similarly determine the photographing method reflecting the user's intention by detecting how to hold the camera. Such a process of detecting how to hold the camera will be described with reference to FIG.

  FIG. 26 shows an example of the output of the acceleration sensor in the life log camera. In this figure, stationary (a) → walking (b) → stationary (c) → hold the camera (d) → camera holding and photographing (e) → camera holding (f) → holding the camera (g) → camera holding and photographing An example of the state of (h) and the state of acceleration change at that time is shown. For example, the period of (a) and (c) is a state in which the life log camera is worn, so it is a change in the degree of vibration of the body. During the period (b), large acceleration and deceleration occur at the start and end of walking, and during that time, the change is in response to the shaking of the body during walking. Also, the periods (d) and (g) represent the period in which the camera is held for shooting, and the acceleration becomes stable to some extent as shown in the figure, and in the periods (e) and (h) the camera shakes again It has become a degree of vibration. Further, the period (f) is a period in which the life log camera is switched from being put on the neck to being in the hand, and since this period temporarily becomes a random operation, it is a disorderly large change.

  Note that "hold the camera" indicates the above-mentioned "shoot while slightly sticking forward while wearing the camera", and "hold the camera" indicates that the camera "held from the neck" is held by the hand. It shows "shooting while switching up and down and upward". In such a case, the photographing in (e) and (h) is not based on the user's instruction, and the photographing can be performed by the determination as to whether or not the automatic photographing described above is performed. At this time, if the holding manner of the camera described above can be extracted, it is possible to carry out the determination of the photographing method described above. For this purpose, it is necessary to extract the periods (d) and (g) in which the camera is held among the changes in acceleration shown in FIG. For this purpose, it is possible to monitor, for example, the acceleration and to detect the period as holding the camera when acceleration above the threshold and deceleration occur. Note that this threshold value can be set to a value such as ± Th in the figure so as to detect, for example, a change larger than hand shake and walking shake. Furthermore, in order to improve the accuracy, it is also possible to set the following conditions.

  The camera shake degree is used as a threshold, and after acceleration or deceleration above the threshold occurs, it is determined whether or not the movement is within the threshold for a predetermined period (about (e) and (h) in the figure). Exclude the acceleration change period due to walking.

  The acceleration change period due to walking is excluded by determining whether or not acceleration and deceleration occur within a fixed period ((d) (g) or more and (b) or more in the figure).

  By excluding the period in which acceleration and deceleration frequently occur in a short time, a changeover period such as (f) is excluded.

  As described above, by monitoring the acceleration of the camera, it is possible to detect how to hold the camera and to determine the photographing method reflecting the user's intention even during automatic photographing.

  In S909, if it is determined that the photographing is determined by the automatic photographing determination in S908b, the process proceeds to S910. If not determined, the process proceeds to the end of the photographing mode processing.

  In S910, automatic shooting is started. At this time, imaging according to the imaging method determined in S 908 a or S 908 b is started. At this time, auto focus control is performed by the focus drive control unit 204. Also, exposure control is performed such that the subject has appropriate brightness by using an aperture control unit, a sensor gain control unit, and a shutter control unit (not shown). Furthermore, after image capturing, the image processing unit 207 performs various image processing such as auto white balance processing, noise reduction processing, gamma correction processing, and the like to generate an image.

  It should be noted that, at the time of photographing, when the predetermined condition is satisfied, the imaging device may perform a notification process to the effect that the person to be photographed is to be photographed before photographing. The method of notification may use, for example, voice from the audio output unit 218 or LED lighting light by the LED control unit 224, or a motion operation of visually guiding the line of sight of the subject by driving the pan / tilt. You may The predetermined conditions are, for example, the number of faces within the angle of view, the degree of smile / eye closure of the face, the gaze angle or face angle of the subject person, the face authentication ID number, the number of persons registered for personal authentication, . In addition, whether the current position based on general object recognition result at the time of shooting, scene discrimination result, shooting time at the previous shooting time, shooting time, GPS information is scenic position, voice level at shooting, voice out Whether the person is present, applause, cheers are rising, etc. Also, the information includes vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet light), and the like. By performing informing shooting based on these conditions, it is possible to leave a preferable image of the camera gaze in a scene of high importance.

  In addition, there are multiple predetermined conditions, change the sound according to each condition, change the lighting method of LED (color, blink time, etc.), or pan / tilt motion method (motion or driving speed) You may change it.

  With regard to such notification before shooting, the method and timing of the notification can also be determined by judgment based on neural network based on information of the shot image or various information detected before shooting. Further, in this determination process, the determination condition can be changed by a learning process described later.

  In step S911, editing processing such as processing the image generated in step S910 or adding it to a moving image is performed. More specifically, image processing is trimming processing based on the face of a person or the in-focus position, image rotation processing, HDR (high dynamic range) effect, blur effect, color conversion filter effect, and the like. A plurality of image processes may be generated based on the image generated in S910 by a combination of the above-described processes, and may be stored separately from the image generated in S910. In addition, in the moving image processing, processing may be performed such that the captured moving image or still image is added to the generated edited moving image while performing special effect processing of slide, zoom, and fade. With regard to the editing in S911, the method of image processing can also be determined by judgment based on information of the photographed image or various information detected before photographing based on the neural network, and this judgment processing is the learning processing described later. The judgment conditions can be changed.

  In S912, learning information generation processing of the photographed image is performed. Here, the information used for the learning process described later is generated and recorded. Specifically, the zoom magnification at the time of shooting, the result of general object recognition at the time of shooting, the result of face detection, the number of faces included in the shot image, the degree of smile / eye closing of face, face angle, face The authentication ID number, the gaze angle of the subject person, and the like. In addition, as a result of scene discrimination, elapsed time since the last shooting, shooting time, GPS position information and change amount from the last shooting position, voice level at the time of shooting, voiced person, applause, cheering or not It is In addition, vibration information (acceleration information, state of imaging device), environment information (temperature, atmospheric pressure, illuminance, humidity, ultraviolet light amount), moving image shooting time, whether by manual shooting instruction, etc. Furthermore, a score, which is the output of a neural network that quantifies the preferences of the user's image, is also calculated.

  These pieces of information are generated and recorded as tag information in the photographed image file. Alternatively, the information may be written to the non-volatile memory 216 or may be stored in the recording medium 221 as so-called catalog data in the form of a list of information of each photographed image.

  In step S913, the past shooting information is updated. Specifically, the number of shots for each area described in the description of S 908 b, the number of shots for each person registered for personal identification, the number of shots for each subject recognized by general object recognition, the number of shots for each scene determination scene The count of the number of images corresponding to the image taken this time is incremented by one.

<Auto Edit Mode Processing (Highlight Video)>
Next, automatic editing mode processing (highlight video) in the present embodiment will be described.

  In the mode setting determination in step 704 of FIG. 7, it is determined whether or not the automatic editing process (highlighted moving image) is to be performed. When the automatic editing process is to be performed, the automatic editing mode process in step 712 is performed.

  The determination conditions of the automatic editing mode will be described. Whether or not to shift to the automatic editing mode depends on the elapsed time since the previous editing process and the tag information (learning information and user's preference for each image taken after the last editing process). It is judged based on the numerical score etc.). FIG. 10 shows the process flow of determining whether or not to shift to the automatic editing mode, which is determined in the mode setting determination process of step 704.

  When start of automatic editing mode determination is instructed in the mode setting determination process of step 704, the process of FIG. 10 is started. In step 1001, an elapsed time TimeD after the previous automatic editing processing is performed is acquired, and the process proceeds to step 1002. In step 1002, learning information, scores, and the like corresponding to each image captured after the time of the previous editing process are acquired, and the process proceeds to step 1003. In step 1003, an evaluation value DB is calculated to determine whether or not automatic editing should be performed from the data acquired in step 1002. The calculation method of the evaluation value is, for example, to extract the feature of the image from each piece of image information, and to increase the score when there are many types of features. Further, as described in the above-described automatic photographing, the score for which the preference of the user is determined is calculated for each image, and the score is increased even when there are many images having high scores. In addition, the score is calculated to be high even if the number of captured images is large. As a result, the evaluation value depends on the height of the score, the number of images, and the feature type. Proceed to step 1004. At step 1004, a threshold value DA is calculated from TimeD. For example, the threshold value DAa when TimeD is smaller than a predetermined value is set to be larger than the threshold value DAb when larger than the predetermined value, and the threshold value is set to decrease with time. As a result, even when there is a small amount of shooting data, the automatic editing process is performed when the elapsed time is large, so that the imaging device automatically creates the highlight moving image according to the usage time. .

  When the process of step 1004 ends, the process proceeds to step 1005, and if the evaluation value DB is larger than the threshold value DA, the process proceeds to step 1006. Set the automatic editing mode to TRUE, because it is possible to obtain data to be automatically edited since the last time automatic editing was performed, or because it is determined that automatic editing should be performed with a large lapse of time. End the automatic editing mode determination. If it is determined in step 1005 that the evaluation value DB is equal to or less than the threshold value DA, it is determined that the data to be automatically edited is not complete, and the automatic editing mode determination is set to FALSE so that the automatic editing process is not performed. Finish.

  Next, processing in the automatic editing mode processing (step 712) will be described. The detailed flow of the automatic editing mode process is shown in FIG.

  In step S1101, the first control unit 223 executes selection processing of still images and moving images stored in the recording medium 221 to select an image to be used for editing, and the processing proceeds to step S1102.

  The image selection processing mentioned here is a threshold determined by extracting metadata such as the captured still image, the number of faces in the moving image, the size of the face, and the color group for each image, and converting it into an evaluation value. I will list the above. The selection ratio of the still image and the moving image is determined by learning described later, and selection is preferentially performed in consideration of the setting of the user, the photographing frequency, and each setting.

  In step S1102, the first control unit 223 and the image processing unit 207 apply image effects to the image selected in step S1101 and the process advances to step S1103.

  The image effect application here refers to trimming processing at the center of the person's face or in-focus position, image rotation processing, HDR (high dynamic range) effect, blur effect, slide, zoom, and fade special effects in still images. Effect processing, color filter effect, etc.

  Similarly, color filters are applied to moving images.

  In S1103, the first control unit 223 sets the image reproduction time, and the process advances to S1104. In order to create the highlight moving image described in S1105 using the image selected in S1101, an appropriate image reproduction time is set based on learning described later.

  In step S1104, the first control unit 223 sets music (BGM) to be added to the highlight moving image described in step S1105, and the process advances to step S1105. With regard to music (BGM) settings, the most appropriate one to be provided to the user is set from the learning results described later.

  In S1105, a first control unit 223 performs a series of highlight moving image creations using the results performed in S1101 to S1104. The created highlight moving image is stored in the recording medium 221.

  The selection of images, the addition of image effects, playback time and BGM selection described above can be performed from the tag information added to each image (information of the photographed image or various information detected before photographing), neural network It can be judged by the judgment based on Further, in this determination processing, the determination conditions can also be changed by learning processing described later.

<Learning mode processing>
Next, learning in accordance with the preference of the user in the present embodiment will be described.

  In the present embodiment, using a neural network as shown in FIG. 12 and a machine learning algorithm, the learning processing unit 219 performs learning in accordance with the user's preference. The neural network is used to predict the output value from the input value, and learns in advance the actual value of the input value and the actual value of the output value to output an output for the new input value. The value can be estimated. By using the neural network, learning is performed according to the user's preference for the above-mentioned automatic photographing, automatic editing, and subject search.

  It also registers subject registration (face recognition, general object recognition, etc.) which is also feature data to be input to a neural network, and changes shooting notification control, low power consumption mode control, and automatic file deletion through learning.

  In the present embodiment, the elements to be learned by the learning process are as follows.

(1) Automatic shooting The learning for automatic shooting will be described. In automatic shooting, learning is performed to automatically take an image that suits the user's preference. As described above in the explanation using the flow of FIG. 9, the learning information generation process is performed after the photographing (step S912). An image to be learned is selected by a method to be described later, and learning is performed by changing weights of the neural network based on learning information included in the image. The learning is performed by changing a neural network that performs automatic shooting timing determination and determining a shooting method (still image shooting, moving image shooting, continuous shooting, panoramic shooting, etc.).

(2) Automatic Editing Describe the learning for automatic editing. In the automatic editing, learning is performed for each of editing immediately after shooting in step 911 in FIG. 9 and editing of the highlight moving image described in FIG. The editing immediately after shooting will be described. An image to be learned is selected by a method to be described later, and learning is performed by changing weights of the neural network based on learning information included in the image. Various detection information obtained by shooting or information immediately before shooting is input to a neural network, and determination of editing method (trimming process, image rotation process, HDR (high dynamic range) effect, blur effect, color conversion filter effect, etc.) I do. Describe the editing of highlight videos. The highlight movie performs learning to automatically create an album movie that meets the user's preference. An image to be learned is selected by a method to be described later, and learning is performed by changing weights of the neural network based on learning information included in the image. Various detection information obtained by shooting or information immediately before shooting is input to a neural network, and image effects are applied (trimming processing, rotation processing, HDR effect, blur effect, slide, zoom, fade, color conversion filter effect, BGM, time And still image / moving image ratio).

(3) Subject Search Learning for subject search will be described. In the subject search, learning is performed to automatically search for a subject matching the user's preference. As described above in the description using the flow of FIG. 9, in the subject search process (step S904), the importance level of each area is calculated, pan / tilt and zoom are driven, and the subject search is performed. The learning is performed based on the photographed image and the detection information during the search, and learning is performed by changing the weight of the neural network. Various detection information during the search operation is input to the neural network, the importance level is calculated, and the pan / tilt angle is set based on the importance level to perform the object search reflecting the learning. In addition to the setting of the pan and tilt angles based on the importance level, for example, learning of pan and tilt driving (speed, acceleration, frequency of movement) is also performed.

(4) Subject Registration The learning for subject registration will be described. In subject registration, learning is performed to automatically register and rank subjects according to the user's preference. As learning, for example, registration of face recognition registration and registration of general object recognition, registration of gesture and voice recognition, and scene recognition by sound are performed. In authentication registration, authentication registration is performed for a person and an object, and rank setting is performed based on the number and frequency of image acquisition, the number and frequency of manual photographing, and the appearance frequency of a subject under search. The registered information is registered as an input of determination using each neural network.

(5) Shooting notification control The learning for the shooting notification will be described. As described in S910 of FIG. 9, when the predetermined condition is satisfied immediately before shooting, shooting is also performed after the imaging device notifies that a person to be shot is to be shot. For example, a motion for guiding the line of sight of the subject visually by driving pan and tilt, a speaker sound emitted from the audio output unit 218, and LED lighting light by the LED control unit 224 are used. Whether the detection information is used for learning is determined immediately after the notification whether the detection information of the subject (for example, smile level, eye line detection, gesture) is obtained, and learning is performed by changing the weight of the neural network. Do. Alternatively, an image to be learned is selected by a method described later, and learning is performed by changing weights of the neural network based on learning information included in the image. The image has embedded therein information on how the notification operation was performed immediately before shooting, and the detection information added to the selected image and the notification operation information immediately before the shooting are analyzed as teacher data, Extract sex and learn according to the extraction result. Each detection information immediately before shooting is input to the neural network, and it is judged whether or not to perform notification, each notification operation (sound (sound level / type of sound / timing), LED light (color, lighting time, blink interval) , And pan / tilt motion (motion, driving speed)). The learning of each informing operation may be performed by selecting which informing method to perform from among informing methods (sound, LED light, and pan / tilt motion combined operation) prepared in advance. Alternatively, separate neural networks may be provided for each notification operation of sound, LED light, and pan / tilt motion to learn each operation.

(6) Low Power Consumption Mode Control Although control to turn on / off the power supply of the Main processor (first control unit 223) is performed as described with reference to FIGS. 7 and 8, recovery from the low power consumption mode is performed. Learning of conditions and transition conditions to the low power consumption state is performed.

  First, learning of conditions for canceling the low power consumption mode will be described.

Tap detection
As described above, the predetermined time TimeA and the predetermined threshold ThreshA are changed by learning. Temporary tap detection is also performed in the state where the above-described tap detection threshold is lowered, and parameters of Time A and Thresh A are set so as to be easily detected depending on whether temporary tap detection has been determined before tap detection. Also, if it is determined from the detection information after tap detection that it is not an activation factor (no shooting target as a result of subject search or automatic shooting determination described above), it is difficult to detect TimeA and Thresh A parameters Do. The determination as to whether or not there is a shooting target at the time of activation varies with subject detection information embedded in an image learned by a learning method described later.

[Swaying state detection]
As described above, the predetermined time Time B, the predetermined threshold Thresh B, the predetermined number of times Count B, and the like are changed by learning. When the activation condition is entered due to the shaking state, activation is performed, but it is determined from the detection information for a predetermined time after activation that it is not an activation factor (as a result of subject search and automatic shooting determination described above, no shooting target). Then, the parameter of the swing state determination is changed, and learning is performed so that it is difficult to start. In addition, when it is determined that the photographing frequency in the state where the shaking is large is high, it is set to be easily activated by the shaking state determination. The determination of whether or not there is a shooting target at the time of activation, and the determination of whether the shooting frequency is high in a large shaking state are the subject detection information embedded in the image learned by the learning method described later and the shaking information at the time of shooting It will change due to

[Sound detection]
Learning can be performed by manually setting a specific voice, a specific sound scene, or a specific sound level that the user wants to detect, for example, through communication with a dedicated application of the external device 301. Further, a plurality of detections are set in advance in the voice processing unit, and an image to be learned is selected by a method to be described later, and learning is performed based on learning information such as sound information before and after shooting included in the image. As a result, it is possible to set a sound determination as an activation factor (a specific sound command or a sound scene such as "cheering" or "applause"), and it is possible to learn activation by sound detection.

[Environmental information detection]
For example, it is possible to learn by manually setting the condition of the environmental information change that the user wants to activate, for example, through communication with the dedicated application of the external device 301. For example, activation can be performed according to specific conditions of temperature, atmospheric pressure, brightness, absolute amount of humidity, and the amount of change of ultraviolet light. In addition, it is also possible to learn the determination threshold based on each piece of environmental information. If it is determined from the detection information after activation by the environment information that it is not an activation factor (no shooting target as a result of subject search or automatic shooting determination described above), it is difficult to detect parameters of each determination threshold Set up. Alternatively, by learning from the information of each environment embedded in the image learned by the learning method described later, it is possible to learn the activation based on the environment information. For example, when learning many images taken at the time of temperature rise, learning is performed so as to be easy to drive at the time of temperature rise. The above parameters also change depending on the remaining capacity of the battery. For example, when the battery remaining amount is low, it becomes difficult to enter various determinations, and when the battery remaining amount is large, it becomes easy to enter various determinations. Specifically, the sway state detection result and the sound scene detection of sound detection, which are conditions that do not require the user to always start the imaging device, change the ease of each detection determination depending on the battery remaining amount. Become.

  Next, learning of the transition condition to the low power consumption state will be described.

  As shown in FIG. 7, if it is determined that the mode setting determination 704 is neither “automatic shooting mode” “automatic editing mode” “automatic image transfer mode” “learning mode” “automatic file deletion mode”, it is low. Enter power consumption mode. The determination conditions of each mode are as described above, but the conditions under which each mode is determined also change by learning. In the automatic shooting mode, as described above, the importance level for each area is determined, and automatic shooting is performed while searching for a subject with pan and tilt, but if it is determined that there is no subject to be shot, the automatic shooting mode Release For example, when the importance level of all the areas or the value obtained by adding the importance levels of the respective areas becomes equal to or less than a predetermined threshold, the automatic photographing mode is canceled. At this time, the predetermined threshold may be lowered according to the elapsed time after the transition to the automatic photographing mode. The transition to the low power consumption mode is facilitated as the elapsed time after the transition to the automatic shooting mode increases. Further, by changing the predetermined threshold according to the remaining capacity of the battery, it is possible to perform the low power consumption mode control in consideration of the battery durability. For example, when the battery remaining amount is low, the threshold is large, and when the battery remaining amount is large, the threshold is small. Here, the parameter (elapsed time threshold TimeC) of the next low power consumption mode cancellation condition is set for the Sub processor based on the elapsed time since the transition to the automatic shooting mode last time and the number of shots.

  Each threshold mentioned above changes by learning. The learning can be performed by, for example, manually setting the imaging frequency, the activation frequency, and the like through communication with a dedicated application of the external device 301. Further, an average value of elapsed time from turning on the power button of the imaging apparatus 101 to turning off the power button or distribution data for each time zone may be accumulated to learn each parameter. In this case, for users with short time from power ON to OFF, the time interval for returning from the low power consumption mode and transition to the low power consumption state becomes short, and the time from power ON to OFF is long Is learned so that the interval is long. In addition, learning is also performed by the detection information under search. While it is determined that the number of important subjects set by learning is large, the time interval for returning from the low power consumption mode and the transition to the low power consumption state is short, and the number of important subjects is small. , Will be learned to increase the interval.

(7) Automatic File Deletion The learning for automatic file deletion will be described. In automatic file deletion, learning is performed on the free space of files and the selection of images to be deleted first. An image to be learned is selected by a method described later, and learning can be performed by changing weights of the neural network based on learning information included in the image. As described above, as described in the above-described automatic photographing, a score in which the preference of the user is determined is calculated for each image, and an image having a low score is preferentially deleted from the recording medium 221. Also, learning is performed not only by the score of the score, but also by the shooting date and time embedded in each image in the recording medium 221, and the editing content of the selected highlight moving image (moving image edited automatically) by the method described later. For example, when the acquired highlight moving image has many images captured at short time intervals, files with older capture dates and times are preferentially deleted, but include images captured at long time intervals. Files with high scores are learned not to be deleted even if the date and time are old. Alternatively, the score of each image in the recording medium 221 is recalculated at predetermined time intervals sequentially. The shooting date and time information is also input to the neural network at the time of score calculation, and when there are many images shot at short time intervals, the file with the old shooting date and time is learned so that the score becomes low. As a result, priority is given to deletion, and when an image shot at a long time interval is included, learning is performed so that the score is not lowered even if the date and time are old, so that the score is high even if the date and time is old Files are learned not to delete. In another example, an image to be learned is selected by the method described later, but in the case where the date and time of the selected image is often selected by concentrating on relatively new ones, files with older shot dates and times are given priority Are deleted. However, if the date and time of the image to be selected is often also selected, learning is performed so that a file with a high score is not deleted even if the date and time are old. In another example, if learning is performed so that the shooting frequency is high, the file is automatically deleted so as to take up a lot of free space of the file, and if it is learned so that the shooting frequency becomes low, the file Automatic deletion of files is performed to save space. In another example, when learning is performed so as to increase the shooting frequency of a moving image, the file is automatically deleted so as to take up a lot of free space of the file, and learning is performed so that the shooting frequency of still images is increased. Automatic deletion of files is performed so that free space of files is reduced.

(8) Image shake correction Learning for image shake correction will be described. The image shake correction is performed by calculating the correction amount in S902 of FIG. 9 and driving the pan and tilt in S905 based on the correction amount to perform the image shake correction. In image shake correction, learning is performed to perform correction in accordance with the characteristics of the user's shake. It is possible to estimate the direction and size of blur by, for example, estimating PSF (Point Spread Function) with respect to a captured image. In the generation of information for learning in S912 of FIG. 9, the direction and the size of the estimated blur are added to the image as information. In the learning mode processing in step 716 of FIG. 7, the estimated direction and size of blur are output, and each detection information at the time of shooting is input to learn weights of the neural network for vibration correction. Each detection information at the time of shooting includes motion vector information of an image at a predetermined time before shooting, motion information of a detected subject (person or object), vibration information (gyro output, acceleration output, state of imaging device), etc. . In addition, environmental information (temperature, barometric pressure, illuminance, humidity), sound information (sound scene determination, specific voice detection, sound level change), time information (elapsed time from startup, elapsed time from previous shooting), Location information (GPS position information, position movement change amount) or the like may be determined in addition to the input. At the time of correction amount calculation in S902, by inputting each detection information to the neural network, it is possible to estimate the size of blur at the moment of shooting, and when the estimated size of blur is large, Control such as shortening the shutter speed becomes possible. In addition, when the size of the estimated blur is large, a blurred image is generated, and therefore a method of prohibiting photographing or the like can be employed. In addition, since there is a limit to the pan / tilt drive angle, further correction can not be performed if the drive end is reached, but by estimating the magnitude and direction of blur at the time of shooting as described above, It is possible to estimate the range required for pan / tilt driving for correcting the shake during exposure. When there is no margin of the movable range during exposure, it is possible to suppress a large blurring by setting the cutoff frequency of the filter for calculating the shake correction amount so as not to exceed the movable range. If the movable range is likely to be exceeded, rotate the pan / tilt angle in the direction opposite to the direction likely to exceed the movable range immediately before exposure, and then start exposure to ensure the movable range. It is also possible to shoot without blurring. As a result, since the shake correction can be learned according to the characteristics and usage of the user at the time of shooting, it is possible to shoot an image without blurring. Further, in the above-described “determination of the photographing method”, it is determined whether or not the follow shot photographing is performed, and the pan / tilt driving speed for photographing the subject without blurring is estimated from detection information before photographing. Subject blur correction may be performed. Here, the panning shooting is a shooting in which a moving subject has no blur and a non-moving background flows. In that case, the drive speed during still image shooting is estimated by inputting the above detection information into a neural network. In learning, by dividing the image into blocks and estimating the PSF of each block, it is possible to estimate the direction and size of blurring in the block in which the main subject is located, and learn based on the information . In addition, it is possible to learn the background flow amount based on the flow amount of the background of the selected image by a learning method described later. In that case, it is possible to estimate the size of blur in a block where the main subject is not located in the selected image, and to learn the user's preference based on the information. By setting the shutter speed at the time of shooting based on the learned background flow amount of preference, it is possible to automatically perform shooting that can obtain a shooting effect that meets the user's preference.

  Furthermore, it is possible to learn an expected photographing method from the information of the state of the camera and the state of the subject in <Determination of the photographing method based on the manual photographing instruction by the user> described above. In this case, first, as information on the state of the camera, for example, the camera angle calculated based on the information obtained from the accelerometer 107, the vibration amount of the camera obtained from the angular velocity meter 106 or the accelerometer 107 It is possible to input. Furthermore, with respect to the position of the camera at a predetermined time before photographing, it is possible to use the movement amount and the relative position at the position of the camera at the time of actual photographing as the input of the neural network. A method of extracting such information in the life log camera will be described with reference to FIG.

  FIG. 26 shows an example of the output of the acceleration sensor in the life log camera. Here, as described above, the situation in which imaging is performed during the periods (e) and (h) is shown. Here, with regard to the amount of movement of the camera, it is possible to calculate the amount of movement between the start of holding the camera and the actual transfer to shooting, that is, the amount of movement in the area of FIGS. It is. For this reason, for example, the acceleration is monitored, and when photographing is performed, changes in the past acceleration are traced, and regions (d) and (g) in which stable acceleration and deceleration occur are detected. The movement amount can be calculated based on the acceleration of the area.

  Further, as described above, it is possible to use the neural network as an input for the information on the subject. Here, information such as, for example, the distance to the subject, and the distance to the subject in the upper and lower, right and left areas of the subject can be used. Also, information of the GPS receiver 405 may be used.

  This makes it possible to learn the photographing method according to the state of the camera and the state of the subject.

(9) Automatic Image Transfer The learning for automatic image transfer will be described. In automatic image transfer, learning is performed on the selection of an image to be transferred with priority and the transfer frequency among the images recorded on the recording medium 221. An image to be learned is selected by a method described later, and learning can be performed by changing weights of the neural network based on learning information included in the image. As described above, in each of the images, as described in the above-mentioned automatic photographing, the score for which the preference of the user is determined is calculated, and the image with the highest score is preferentially transferred. In addition, learning information corresponding to an image transferred in the past is also used for image transfer determination. When an image to be learned by a method to be described later is selected, what is regarded as important in the learning information (feature amount) included in the image is set, and an image transferred in the past includes similar feature amounts In many cases, it is set to transfer an image including another feature amount and having a high score. In addition, the image transfer frequency also changes according to each state of the imaging device. It changes with the remaining capacity of the battery. For example, when the battery remaining amount is low, image transfer is difficult, and when the battery remaining amount is large, image transfer is set to be easy. Specifically, for example, the elapsed time from the previous automatic transfer is multiplied by the highest score among the images taken during the elapsed time, and the image is transferred when the multiplied value exceeds the threshold. It is possible to realize a configuration in which the threshold value is changed depending on the remaining battery level. In another example, the frequency of automatic image transfer is changed according to the set imaging frequency of the imaging apparatus 101. When learning is done to increase the frequency of shooting, the frequency of automatic image transfer is set to increase, and when learning is performed to decrease the frequency of shooting, the frequency of automatic image transfer is also decreased Set to At this time, it is possible to change the image transfer frequency according to the setting of the imaging frequency by changing the threshold value according to the imaging frequency. In another example, the frequency of automatic image transfer is also changed according to the free space of the file (recording medium 221). When the free space of the file is large, the frequency of automatic image transfer is low, and when the free space of the file is small, the frequency of automatic image rotation is set to be high. At this time, the image transfer frequency can be changed according to the file free space by changing the above-mentioned threshold value according to the file free space.

  Next, the learning method will be described.

  As a learning method, there are “learning in the imaging device” and “learning in cooperation with the communication device”.

  The method of in-imaging device learning will be described below.

  There are following methods for in-imaging device learning in the present embodiment.

(1) Learning based on detection information at the time of shooting instruction by the user As described in steps S 907 to S 913 in FIG. 9, in the present embodiment, the imaging apparatus 101 can perform two shootings: manual shooting and automatic shooting. . In step S 907, when there is a photographing instruction by manual operation (as described above, it is performed based on three determinations), in step S 912, information is added that the photographed image is an image photographed manually. Ru. If it is determined in step S909 that automatic imaging is ON and imaging is performed, information is added in step S912 that the captured image is an image captured automatically.

  In the case of manual shooting here, the possibility of shooting based on the user's favorite subject, favorite scene, favorite place and time interval is very high. Therefore, learning based on learning of each feature data and captured image obtained at the time of manual imaging is performed.

  Further, from detection information at the time of manual imaging, learning is performed regarding extraction of feature amounts in the captured image, registration of personal authentication, registration of individual facial expressions, and registration of human combinations. In addition, from the detection information at the time of subject search, for example, learning is performed to change the importance of a nearby person or object from the expression of the subject registered individually. Also, when a change in the angle of view is instructed by “the user rotates the pan and tilt by hand” described later with reference to FIGS. 17 to 22, the subject present in the angle of view is learned after the rotation. . This is also part of learning by manual operation detection information.

  Furthermore, as described above, when the determination of the photographing method is performed, information necessary for the determination is similarly added in S912. Here, for example, as described above, it becomes possible to assign camera state information such as the camera holding type, holding angle, moving distance and movement direction of the camera up to shooting. Further, it becomes possible to assign subject information such as subject distance, and the distance to the subject in the upper and lower, right and left areas of the subject. The information does not necessarily have to be attached to the image. For example, the information may be stored in the memory 216 or the like in the imaging apparatus, or may be associated with the image in a file different from the image file. Storage is also possible.

(2) Learning based on detection information at the time of subject search During a subject search operation, it is determined what person, an object, a scene, and a subject are registered at the same time as the subject registered for personal identification. Calculate the ratio.

  For example, it can be determined that the importance is high when the time ratio in which the person A of the personal identification registration subject is shown simultaneously with the person B of the personal identification registration subject is higher than a predetermined threshold. For this reason, when the person A and the person B fall within the angle of view, various types of detection information are stored as learning data so as to increase the score of the automatic photographing determination, and learning is performed in the learning mode processing 716.

  In another example, it can be determined that the importance is high when the time ratio at which the person A of the personal identification registration subject simultaneously appears with the subject "cat" determined by general object recognition is higher than a predetermined threshold. Therefore, when the person A and the "cat" fall within the angle of view, various detection information is stored as learning data so that the score of the automatic photographing determination becomes high. Then, learning is performed in a learning mode process 716.

  As described above, when the number of occurrences of the subject under search is high, if the score of the automatic photographing determination is increased, the importance of the person or object near the subject registered for personal identification is also increased. It can be changed.

  In addition, when the degree of smile of the person A of the personal identification registration subject is detected or “joy” or “surprise” is detected by detecting the facial expression, the subject that is shown at the same time is learned to be important To be done. Also, processing such as not learning is performed because the subject appearing at the same time when an expression such as "anger" or "true face" is detected is unlikely to be important.

  Next, learning in cooperation with an external communication device in the present embodiment will be described.

  There are the following methods for learning in cooperation with an external communication device in the present embodiment.

(3) Learning Based on Acquisition of Image by External Communication Device As described in FIG. 3, the imaging apparatus 101 and the external device 301 have communication means of the communication 302 and 303. The image transmission / reception is mainly performed by the communication 302, and the image in the imaging apparatus 101 can be acquired by communication with the external device 301 via a dedicated application in the external device 301. Also, the thumbnail image of the image data stored in the imaging apparatus 101 can be browsed through a dedicated application in the external device 301. In this way, the user can select an image that he / she likes from the thumbnail images, check the image, and operate the image acquisition instruction to acquire the image on the external device 301.

  At this time, since the user has selected an image and instructed to transmit and acquire it, the acquired image is very likely to be an image preferred by the user. Therefore, it is possible to perform various types of learning of the user's preference by determining that the acquired image is an image to be learned and learning based on the learning information of the acquired image.

  An operation example will be described. An example in which an image in the imaging apparatus 101 is browsed through an application dedicated to the external device 301 which is a smart device is shown in FIG. The display unit 407 displays thumbnail images (1604 to 1609) of the image data stored in the imaging apparatus, and the user can select an image that he / she likes and can acquire an image. At this time, a display method change unit (1601, 1602, 1603) is provided to change the display method. When the user presses the button 1601, the display order is changed to the date and time priority display mode, and the images are displayed on the display unit 407 in the order of the shooting date and time of the image in the imaging device 101. (For example, the date is displayed as 1604 is new, and 1609 is displayed as old.) When 1602 is pressed, the recommended image priority display mode is changed. The images are displayed on the display unit 407 in descending order of the score of the image in the imaging apparatus 101 based on the scores obtained by determining the preference of the user for each image calculated in step S912 in FIG. (For example, 1604 indicates that the score is high and 1609 indicates that the score is low.) Pressing 1603 can designate a person or an object subject, and subsequently designating a specific person or an object subject displays only a specific subject You can also

  For example, when all settings are turned on, only the designated subject is displayed, and an image with a new shooting date and time is prioritized, and an image with a high score can be displayed. Will take precedence and will be displayed.

  As described above, since the user's preference is learned also for the photographed image, it is possible to easily extract only the user's favorite image from a large number of photographed images by a simple confirmation operation. .

(4) Learning by inputting a determination value to an image through an external communication device As described above, the imaging device 101 and the external device 301 have communication means, and are stored in the imaging device 101. These images can be browsed through a dedicated application in the external device 301. Here, the user may be configured to score each image. A high score (for example, 5 points) can be added to an image that the user thinks is preferred, or a low score (for example, 1 point) can be added to an image thought to be unpreferable. Will be structured as you learn. The score of each image is used for relearning along with learning information in the imaging device. Learning is performed so that the output of the neural network, which has feature data from specified image information as an input, approaches a point specified by the user.

  In the present embodiment, the user inputs the determination value to the photographed image via the communication device 301. However, the imaging device 101 may be operated to directly input the determination value to the image. . In that case, for example, the imaging device 101 is provided with a touch panel display, and the user presses a GUI button displayed on the touch panel display screen display unit to set a mode for displaying a photographed image. Then, the user can perform similar learning by a method of inputting a determination value to each image while confirming the photographed image.

(5) Learning by analyzing the image stored in the external communication device The external device 301 has a storage unit 404, and an image other than the image captured by the imaging device 101 is also stored in the storage unit 404. Configuration. At this time, it is easy for the user to browse the images stored in the external device 301, and it is also easy to upload the images to the shared server via the public line control unit 406. Very likely to be included.

  The external device 301 may be configured to be able to process the image stored in the storage unit 404 through the dedicated application by the control unit 411 with the same learning processing as the learning processing unit 219 in the imaging apparatus 101. . In this case, by communicating the processed data for learning to the imaging apparatus 101, the learning can be performed. In addition, an image or data to be learned by the imaging device 101 may be transmitted to be learned in the imaging device 101.

  Alternatively, the user may select an image to be learned from among the images stored in the storage unit 404 via a dedicated application, and may learn.

(6) Learning from information uploaded to the SNS server with external communication devices Social networking services (SNS), which are services and websites that can build social networks that focus on the connection between people I will explain how to use the information in I for learning. When uploading an image to an SNS, there is a technology of inputting a tag related to the image from a smart device and transmitting it along with the image. In addition, there is also a technology for inputting liking or dislike to images uploaded by another user, and it is possible to determine whether the images uploaded by the other user are favorite photos of the user who owns the external device 301.

  The dedicated SNS application downloaded into the external device 301 can acquire the image and the information about the image uploaded by the user as described above. Also, it is possible to obtain user's favorite image and tag information by inputting whether the user likes the image uploaded by another user. These images and tag information are analyzed, and a configuration is made such that a learning set can be made in the imaging device 101.

  An image uploaded by the user as described above or an image determined to be liked by the user may be acquired, and the control unit 411 may be capable of processing the same learning processing as the learning processing unit 219 in the imaging apparatus 101. . In this way, the processed learning data may be communicated to the imaging apparatus 101 to learn. Alternatively, an image to be learned by the imaging device 101 may be transmitted, and learning may be performed in the imaging device 101.

Further, from the tag information, from the image filter provided in the SNS, the color conversion filter effect of the automatic editing mode process 712 of FIG. 7 and the editing S 911 of FIG. 9 is learned to change.
Alternatively, learning is performed by estimating subject information which the user prefers from the subject information set in the tag information, and registering it as a subject to be detected which is input to the neural network. The subject information may be, for example, subject object information such as a dog or a cat, scene information such as a beach, or expression information such as a smile.

  In addition, image information currently in circulation in the world may be estimated from statistical values of tag information (image filter information and subject information) in the SNS so that learning setting can be performed in the imaging apparatus 101. it can.

(7) Learning by changing parameters in an external communication device As described above, the imaging device 101 and the external device 301 have communication means, and learning parameters currently set in the imaging device 101 Can be communicated to the external device 301 and stored in the storage unit 404 of the external device 301. As learning parameters, for example, weights of a neural network, selection of an object to be input to the neural network, and the like can be considered. In addition, the learning parameter set in the dedicated server can be acquired via the public line control unit 406 via the dedicated application in the external device 301, and can be set as the learning parameter in the imaging apparatus 101. I assume. In this way, the parameters at a certain point in time can be stored in the external device 301, and learning parameters can be returned by setting them in the imaging apparatus 101, or learning parameters possessed by other users can be obtained via a dedicated server. It is also possible to acquire and set the image capturing apparatus 101 of its own.

  In addition, voice commands, authentication registrations, gestures registered by the user may be registered via an application dedicated to the external device 301, or important places may be registered. The imaging trigger described in the automatic imaging mode process (FIG. 9) is treated as input data for automatic imaging determination.

  In addition, the shooting frequency, start interval, still image and moving image ratio, and a favorite image can be set, and the start interval described in <Low Power Consumption Mode Control>, and still image moving image ratio described in <Auto Edit> You may set it.

(8) Learning from manually edited image of an external communication device A dedicated application of the external device 301 is provided with a function that can be manually edited by the user's operation, and the contents of editing work can be fed back to learning. it can. For example, it is possible to learn an automatic editing neural network so that editing of the image effect can be performed and the manually edited image effect can be determined with respect to the learning information of the image. As the image effect, for example, trimming processing, rotation processing, slide, zoom, fade, color conversion filter effect, time, still image / moving image ratio, BGM can be considered.

  Here, in the case of learning in which the user has taken (1) or image selection ((3) to (8)), since the user intentionally operated, it is reliable for reflecting the preference of the user. There is a high possibility of learning. However, (2) learning based on detection information at the time of subject search is not learning due to the user's operation at the time of intention, so there is a possibility that learning not desired by the user may be performed. For example, while searching for another person, an object, or a scene captured at the same time as a subject registered for personal identification from image information during a search, the subject captured frequently at the same time is necessarily limited to the user's preference. Absent. Therefore, learning ((1), (3) to (8)) in the case where the user intentionally takes a picture or selects an image is better than the case where it is not learning due to the user's intentional operation (2) Give priority to learning.

  The data for learning is various data (image information, vibration information, environment information, sound information, place information, etc.) recorded as tag information at the time of shooting or searching, and when reflecting in learning, these various data Is stored in the form of a list. The number of data groups for learning is assumed to have a fixed number determined. The data group for learning is divided into two areas of an area which is learning data intentionally performed by the user and an area which is learning data not intentionally used by the user, and the ratio of the number of data in the area is The ratio is set so that the learning data area intentionally performed by the user is larger. When a new learning reflection instruction is given, it is deleted from the learning data corresponding to each area, and new learning data is added. For example, when adding two learning data intentionally performed by the user, two data are deleted from the learning data area intentionally performed by the user, and two new data are added and re-learned. .

  With such a configuration, learning when the user intentionally shoots or selects an image than when learning is not performed by the user's intentional operation (2) ((1), (3) (8) can be learned preferentially.

  Alternatively, in various data for learning, the date and time when the learning data is generated is managed, and a weighting coefficient La according to the elapsed time from the date and time when the learning data is generated is calculated. The weighting factor La is updated to be smaller as the elapsed time becomes larger. In addition, a weighting factor Lb depending on whether the learning data intentionally performed by the user or the learning data not intended by the user is also managed corresponding to each learning data. The weighting coefficient Lb is set to be larger in the case of the learning data intentionally performed by the user than the learning data in which the user is not intentional. In addition, the weighting coefficient Lb may be changed depending on which learning of (1), (3) to (8) the learning data intentionally performed by the user.

  When new learning data is added, the data is preferentially deleted from the learning data with the smallest value obtained by multiplying the weighting coefficients La and Lb in the current learning data group, and then the additional data is inserted. Perform machine learning based on the updated learning data group.

  With such a configuration, learning when the user intentionally shoots or selects an image than when learning is not performed by the user's intentional operation (2) ((1), (3) (8) can be learned preferentially.

  If the configuration of FIGS. 1 to 6 has no screen on the imaging device side, setting of the priority is difficult, and it takes time and effort for the user to set in the menu on the external device side. However, if the priority is automatically flagged based on the operation for photographing processing and the operation for editing processing, not the operation for flagging priority, the time and effort of the user can be reduced. Can. Further, when automatically evaluating the priority using the photographed image, for example, when evaluating an image photographed at a predetermined time interval, only the same one is photographed, but it may not necessarily be a main subject. , The user's intention may not be reflected. On the other hand, in the present embodiment, there is a high possibility that the intention of the user is sufficiently reflected because the priority is flagged based on the process intentionally performed by the user.

  Next, the learning processing sequence will be described.

  In the mode setting determination in step 704 of FIG. 7, it is determined whether or not learning processing should be performed. If learning processing is performed, it is determined that the learning mode is set, and learning mode processing in step 716 is performed.

  The determination conditions of the learning mode will be described. Whether or not to shift to the learning mode is determined based on the elapsed time since the previous learning process, the number of information that can be used for learning, and whether there is a learning process instruction via the communication device. FIG. 14 shows a flow of processing for determining whether or not to shift to the learning mode, which is determined in the mode setting determination processing of step 704.

  When start of learning mode determination is instructed in the mode setting determination process of step 704, the process of FIG. 14 is started. In step 1401, it is determined whether there is a registration instruction from the external device 301. The registration here is a determination as to whether or not there is a registration instruction for learning as described above. For example, there are <learning based on image information acquired by the communication device> and <learning based on inputting a determination value to an image through the communication device>. Also, there are <learning by analyzing the image stored in the communication device>, <learning by analyzing the image stored in the communication device> and the like. If there is a registration instruction from the external device in step S1401, the process advances to step S1408 to set the learning mode determination to TRUE and perform processing of step S716. If there is no registration instruction from the external device in step S1401, the process advances to step S1402. In step 1402, it is determined whether there is a learning instruction from an external device. The learning instruction here is a determination as to whether or not there is an instruction to set a learning parameter, as in <Learning by changing the imaging device parameter in the communication device>. If it is determined in step S1402 that there is a learning instruction from an external device, the process advances to step S1408 to set the learning mode determination to TRUE, perform processing of step S716, and end the learning mode determination process. If it is determined in step 1402 that there is no learning instruction from the external device, the process advances to step 1403.

  In step S1403, an elapsed time TimeN from the previous learning process (recalculation of weights of neural networks) is acquired, and the process proceeds to step S1404. In step 1404, the number of new data to be learned DN (the number of images designated to be learned during the elapsed time Time N since the previous learning process was performed) is acquired, and the process proceeds to step 1405. At step 1405, the threshold value DT is calculated from TimeN. For example, the threshold value DTa when TimeN is smaller than a predetermined value is set larger than the threshold value DTb when larger than the predetermined value, and the threshold value is set to be smaller with the passage of time. As a result, even when the amount of learning data is small, the image pickup apparatus is made to easily change learning according to the time of use by learning again if the time elapsed is large.

  After calculating the threshold value DT in step 1405, the process proceeds to step 1406, where it is determined whether the number of data to be learned DN is larger than the threshold value DT. If the DN is larger than the threshold DT, the process proceeds to step 1407, sets DN to 0, and then proceeds to step 1408 to set the learning mode determination to TRUE and perform the process of step 716 to determine the learning mode End the process.

  If it is determined in step 1406 that the DN is smaller than the threshold DT, the process proceeds to step 1409. Since there is neither a registration instruction from an external device nor a learning instruction from an external device, and the number of learning data is less than a predetermined value, the learning mode determination is set to FALSE and the process of step 716 is set not to be performed. The determination process ends.

  Next, the process in the learning mode process (step 716) will be described. The detailed flow of the learning mode process is shown in FIG.

  When it is determined that the learning mode is in step 715 of FIG. 7 and the process proceeds to step 716, the process of FIG. 15 starts. In step 1501, it is determined whether there is a registration instruction from the external device 301. If it is determined in step 1501 that there is a registration instruction from an external device, the process advances to step 1502. At step 1502, various registration processing is performed.

  The various registrations are registrations of features to be input to the neural network, such as registration of face recognition, registration of general object recognition, registration of sound information, registration of location information, and the like.

  When the registration process is completed, the process proceeds to step 1503, and the information to be input to the neural network is changed from the information registered in step 1502.

  When the process of step 1503 ends, the process proceeds to step 1507.

  If there is no registration instruction from the external device 301 in step 1501, the process proceeds to step 1504, and it is determined whether there is a learning instruction from the external device 301. If there is a learning instruction from the external device, the process proceeds to step 1505, the learning parameter communicated from the external device is set in each determiner (such as weight of neural network), and the process proceeds to step 1507.

  If it is determined in step 1504 that there is no learning instruction from an external device, learning (recalculation of neural network weights) is performed in step 1506. The process of step 1506 is a condition under which the number of data to be learned DN exceeds the threshold and the respective learners can relearn, as described with reference to FIG. Retraining is performed using a method such as error back propagation method or gradient descent method, and weights of the neural network are recalculated to change parameters of each decision unit. Once the learning parameters are set, the process proceeds to step 1507.

  At step 1507, the images in the file are re-scored. In the present embodiment, all photographed images stored in the file (recording medium 221) are scored based on learning results, and automatic editing and automatic file deletion are performed according to the scored scores. It is a structure. Therefore, when re-learning or setting of learning parameters from an external device is performed, it is also necessary to update the scores of captured images. Therefore, in step 1507, recalculation is performed to add a new score to the photographed image stored in the file, and when the process ends, the learning mode process ends.

  In this embodiment, a method of proposing a user's favorite video by extracting a scene that the user seems to like, learning its features, and reflecting the feature on operations such as automatic shooting and automatic editing has been described. However, the invention is not limited to this application. For example, it can also be used for an application that proposes an image different from the user's own preference. As an example of the realization method, it is as follows.

(1) Method using neural network in which preference is learned For learning, the user preference learning is carried out as described above. Then, in S908 of "automatic imaging", automatic imaging is performed when the output value of the neural network is a value indicating that it is different from the user's preference which is teacher data. For example, if an image preferred by the user is used as a teacher image and learning is performed so that a high value is output when showing a feature similar to the teacher image, it is automatically performed on the condition that the output value is lower than a predetermined value. Take a picture. Similarly, also in the subject searching process and the automatic editing process, a process is performed in which the output value of the neural network becomes a value indicating that it is different from the user's preference as teacher data.

(2) Method of using a neural network in which a situation different from preference is learned In this method, at the time of the learning process, learning is performed by using a situation different from the user's preference as teacher data. For example, assuming that the image photographed manually is a scene photographed by the user, the learning method using the image as teacher data has been described above. On the other hand, in the present embodiment, an image captured manually is not used as teacher data, and a scene for which manual imaging has not been performed for a predetermined time or more is added as teacher data. Alternatively, if there is a scene whose feature is similar to the manually captured image in the teacher data, it may be deleted from the teacher data. Further, an image having a feature different from the image acquired by the external communication device may be added to the teacher data, or an image having a feature similar to the image acquired may be deleted from the teacher data. By doing this, data different from the user's preference is collected in the teacher data, and as a result of learning, the neural network can determine the situation different from the user's preference. Then, in the automatic shooting, a scene different from the user's preference can be shot by shooting according to the output value of the neural network. In addition, in automatic editing, it is possible to propose an edited image that is different from the preference of the user.

  As described above, by intentionally presenting a video different from the user's preference, it is possible to reduce missed shots by shooting in a scene in which the user manually takes a picture. In addition, by proposing a shooting and editing effect in a scene that is not in the user's own idea, it is possible to expect the user to be aware of it and to broaden the range of preference.

  In addition, by combining the above-described means, it is also easy to adjust the degree of conformity to the user's preference, such as a proposal of a situation that is somewhat similar to, but partially different from, the user's preference. The degree of conformity to the preference of the user may be changed according to the mode setting, the state of the various sensors, and the state of the detection information.

  In the present embodiment, although the explanation has been made based on the configuration for learning in the imaging apparatus 101, the external device 301 has a learning process, data necessary for learning is communicated to the external device 301, and only the external device The same learning effect can be realized even with a configuration that executes learning. In that case, as described above in <Learning by changing parameters in the communication device>, learning is performed by setting parameters such as weights of the neural network learned on the external device side in the imaging apparatus 101 by communication. It may be configured.

  Further, both the inside of the imaging apparatus 101 and the inside of the external device 301 may have a learning process. For example, learning information possessed by the external device 301 may be communicated to the imaging apparatus 101 at the timing when the learning mode processing 716 is performed in the imaging apparatus 101, and learning may be performed by merging learning parameters.

Further, in S907 of the above-described automatic shooting mode processing (FIG. 9), it is determined whether or not the user (manual) has issued a shooting instruction, and in the case where the shooting instruction is issued, the process proceeds to S910. Here, the photographing instruction by the user (manually) may be made possible by the user rotating the direction in which the imaging device is facing (hereinafter referred to as the photographing direction) by hand. FIG. 17 is a block diagram showing the configuration of the lens barrel rotation drive unit 205. As shown in FIG. Reference numerals 1701 to 1707 in FIG. 17 denote configurations related to the drive of the pan axis. Reference numerals 1708 to 1714 in FIG. 17 denote configurations related to drive control of the tilt axis. Since the basic configuration related to the drive of the pan axis and the drive of the tilt are the same, only the configuration related to the drive of the pan axis will be described, and the description of the configuration related to the drive of the tilt axis will be omitted. Reference numeral 1701 denotes an image position-pan position conversion unit for calculating a target position when driving the pan shaft 1706 from the difference between the target position on the image of the subject and the current position. FIG. 18 is a diagram showing the relationship between the current position of an object and the target position in an image captured by an imaging device. Reference numeral 1801 denotes a momentary image obtained by the image processing unit 207 while the imaging device is searching for a subject. Reference numeral 1802 denotes the current position (x1, y1) of the subject. Reference numeral 1803 denotes the target position (x0, y0) of the subject. When calculating the pan and tilt target positions from the difference between the target position 1803 of the subject on the image and the current position 1802, the following equation is used.
kp (f) x (x1-x0) (Expression 1)
kt (f) x (y1-y0) (Equation 2)

  kp (f) is a conversion coefficient for calculating the pan target position from the difference between the target position and the current position on the image of the subject that changes according to the focal length f of the imaging device. kt (f) is a conversion coefficient for calculating the target position of the tilt from the difference between the target position and the current position on the image of the subject which changes according to the focal length f of the imaging device.

  Reference numeral 1702 in FIG. 17 denotes a compensator. The compensator 1702 calculates a control output by performing a PID control operation so as to eliminate the difference between the current pan position and the pan target position calculated by the image position-pan position converter 1701. An imaging direction change operation detection unit 1703 detects an imaging direction change operation from the difference between the pan target position and the current position (hereinafter referred to as position deviation), the control output, and the moving speed of the pan. When a change in the shooting direction is detected by the shooting direction change operation detection unit 1703, the control output is turned off to stop the pan drive. On the other hand, when the change in the shooting direction is not detected by the shooting direction change operation detection unit 1703, pan drive control is performed according to the control output calculated by the compensator 1702. Reference numeral 1704 denotes a driver for generating a drive signal according to the control output calculated by the compensator 1702. An ultrasonic motor (USM) 1705 is an actuator for driving the pan shaft 1706. Reference numeral 1707 denotes a moving speed detection unit for calculating the moving speed of the pan from the time change of the pan position. The moving speed detection unit 1707 calculates the moving speed of the pan from the amount of change in the pan position for each control sampling. FIG. 19 is a flowchart showing a flow of detecting a shooting direction change operation by a user operation and updating learning information with the shooting area after the shooting direction change operation as an important area.

  In step S1901, it is determined whether the user has performed a shooting direction change operation of the imaging apparatus. In the detection of the shooting direction change operation by the user, the shooting direction change operation detection unit 1703 determines that the shooting direction change is present when a control output and a positional deviation described later satisfy predetermined conditions. If the photographing direction change operation is detected in S1901, the process advances to S1902 to stop the position control operation. If the subject is being tracked or searched, the position control operation is stopped after being interrupted. On the other hand, when the shooting direction change operation is not detected in S1901, the detection of the shooting direction change operation is continued. After the position control is stopped in S1902, the process advances to S1903 to determine the end of the shooting direction change operation by the user.

  In the determination as to the end of the shooting direction change operation, the shooting direction change operation detection unit 1703 determines continuation or end of the shooting direction change operation based on the moving speed of pan. If it is determined that the shooting direction change operation has ended, the process advances to step S1904 to store shooting area information after it is determined that the shooting direction change operation has ended. The area to be stored stores the closest area by comparing the angle of view determined from the position of the imaging apparatus, the pan position, the tilt position, and the focal length with each divided area. If it is determined in S1903 that the shooting direction change operation is in progress, detection of the end of the shooting direction change operation is continued. In S1905, the learning information is updated with the area stored in S1904 as an area more important than the other divided areas. In step S1906, after subject tracking and position control are enabled, the process advances to step S1901 to resume detection of the shooting direction change operation. As an example in which the user performs a shooting direction change operation, the user rotates the lens barrel 102 while shooting a flower with the imaging device 101 so that the optical axis of the imaging device 101 faces a specific person outside the angle of view. An example when the shooting direction change operation is performed will be described. FIG. 20 illustrates an example in which the learning information is updated with the area where the person 2003 exists as the important area after rotating the lens barrel 102 with the user's hand in the direction of the person 2003 while photographing the flower 2001 with the imaging device 101. It is a schematic diagram for doing. Reference numeral 2002 in FIG. 20 denotes an optical axis of the imaging device 101 during imaging of the flower 2001. Reference numeral 2004 denotes an optical axis after the user manually changes the shooting direction. Reference numeral 2005 indicates the rotation direction of the lens barrel 102 when the user changes the shooting direction. Using FIG. 21 and FIG. 22, after changing the photographing direction in the direction of the person 2003 by the user operation while photographing the flower 2001, the operation from updating the learning information to the area where the person 2003 exists as an important area is described. Do. 21 (a), (b), (c), and (d), from the time of shooting a flower to the time of changing the shooting direction to the direction of the specific person 2003 and changing the shooting direction to update learning information. Is a diagram showing an image taken at a certain moment. FIG. 22 shows a pan control output 2201 until the user changes the shooting direction to the direction of the specific person 2003 while shooting a flower, and the area of the angle of view after the change is regarded as an important area and the learning information is updated. It is the figure which showed the time change of the position deviation 2202 and the moving speed 2203. FIG. Ta, tb, tc, and td in FIG. 22 are times at which the images shown in FIGS. 21 (a), (b), (c), and (d) were captured, respectively. ThC in FIG. 22 is a threshold of control output used to determine that the user has manually rotated the lens barrel 102. ThDiff is a threshold of positional deviation used to determine that the user has manually rotated the lens barrel 102. The control output of the compensator 1702 is turned off assuming that the user changes the imaging direction when the control output is ThC or more and the position deviation is ThDiff or more for a predetermined time (t2-t1 in FIG. 22). . ThV is a threshold value of the moving speed of the pan axis used to determine that the user has finished the shooting direction operation. CMax is the maximum value of the control output of the compensator 1702. In position control, the drive of the pan axis is controlled so that the object is positioned at the target position on the image by changing the control output in the range of -CMax to CMax. T1 in FIG. 22 indicates the time when the control output 2201 is ThC or more and the positional deviation is ThDiff or more after the user starts the shooting direction operation. t2 indicates the time when the control output 2201 is ThC or more and the positional deviation 2202 is ThDiff or more and the imaging direction change determination time (t2-t1) has elapsed. t3 indicates the time when the moving speed of the pan axis is less than ThV for the first time after time t2. t4 indicates a time when an elapsed time from when the moving speed becomes ThV or less at time t3 becomes the photographing direction change end determination time (t4-t3).

  FIG. 21A shows an image captured at the timing of time ta while the flower 2001 is captured. Reference numeral 2101 in FIG. 21A denotes a subject frame indicating a subject to be tracked, searched, or photographed. Reference numeral 2102 denotes a target point as a target position on the image at the center of the subject frame 2101. A point at which two lines 2102 intersect is a target position on the image of the subject. During normal shooting operation (state in which shooting direction change operation is not performed), alignment is performed by drive control of the pan axis or tilt axis such that the center of the subject frame 2101 and the target point 2102 overlap. FIG. 21B is an image captured when the user rotates the lens barrel 102 in the right direction with respect to the fixed unit 103 at the timing of time tb in the state of FIG. 21A. The black arrows in FIG. 21B indicate the pan driving direction of position control, and the white arrows indicate the rotational direction of the lens barrel 102 due to the user's shooting direction changing operation. Looking at the control output 2201 and the position deviation 2202 at time tb, the position deviation 2202 tends to increase even though the control output is at the maximum value CMax. From this, it can be determined that the user intentionally rotates the pan axis. In this embodiment, after detecting that the user is rotating the lens barrel 102 by hand, the camera waits for a predetermined time (t2-t1) before the control output of the compensator 1702 is turned off. Judgment of direction change is performed. This is because the shooting direction has been changed if the user has not performed a direction change operation because the user has unintentionally touched the lens barrel, or if the user has not performed a direction change operation due to a load change of the pan axis or tilt axis during search driving. It is a measure for not judging it as The time from the start of the user's shooting direction change operation to the quick determination of the shooting direction change determination may be shortened or eliminated.

  In FIG. 21C, when the control output of the compensator 1702 is turned off at time tc, the pan axis is rotated to near the new subject by the user's shooting direction change operation, and the target subject enters the angle of view. Of the As described above, the user needs to continue the shooting direction change operation until the subject to be newly photographed enters the angle of view. As in the case of the imaging apparatus 101 shown in the present embodiment, when it is not possible to directly check the image being changed in shooting direction, the smart device is used to confirm the image being changed and the operation is performed. Make sure you are in the corner. As a means for the user to know that other subjects to be photographed have entered the angle of view, if a new subject enters the angle of view while changing the shooting direction, the LED control unit 224 may cause the LED to emit light The user may be notified by making the output unit 218 output voice.

  FIG. 21 (d) shows an image being captured while tracking a new subject after changing the shooting direction with the control output of the compensator 1702 turned ON at the timing of time t4. A time t4 is a timing after the time when the moving speed 2203 of the pan at the time td becomes ThV or less elapses for the photographing direction change operation end determination time (t4-t3) or more. If it is determined that the shooting direction change operation by the user is completed at time t4, the learning information is updated after setting the shooting area at time t4 as a user's favorite area with higher importance than other areas. . Also, the subject existing in this area may be tracked as an important subject, and one or more operations of photographing and authentication registration may be performed. For example, as shown in FIG. 21D, when a person 2003 is present within the angle of view at the timing when it is detected that the user has finished the shooting direction change operation by the user, the person 2003 is tracked as an important subject and shooting is performed. , Perform any operation of certification registration. The learning information update process may be performed only when there is a learning instruction from the user, not automatically. For example, after the imaging apparatus notifies the user that the subject is in the angle of view, the learning information is updated only when a specific voice command for learning instruction registered in advance is input. The learning information may be updated only when the user gives a learning instruction.

  In this embodiment, the detection of the start and the end of the shooting direction change operation of the imaging apparatus by the user is detected by the control output of the compensator, the position deviation, and the moving speed of the drive axis. If detectable, it may be detected by other methods. For example, the presence or absence of the imaging direction change by the user may be detected based on the time change of the signal of the gyro sensor or the acceleration sensor from the device shake detection unit 209. FIG. 23 shows a change in output of the acceleration sensor of the device shake detection unit 209 when the imaging direction of the imaging device is changed by a user operation. 2301 indicates time change of acceleration. ThA1 is a threshold value of acceleration used when it is determined that the user has started the shooting direction change operation. ThA2 is an acceleration threshold value that determines that the user has finished the shooting direction change operation. The threshold and the acceleration may be compared to detect the start and end of the shooting direction change operation. At this time, in order to prevent false detection of the shooting direction change operation, the time change pattern of the acceleration at the time of the shooting direction change operation is learned in advance, and the similarity to the time change pattern at which the time change of the detected acceleration is learned is It may be determined that the imaging direction has been changed if the value is equal to or more than a predetermined value. Similarly, the presence or absence of the shooting direction operation may be detected according to the change of the motion vector of the image captured by the imaging device.

  Further, in the above description, the processing for learning the imaging area that is within the angle of view after the imaging direction change operation has been described as an important area. However, the present invention is not limited to this, and when there is a change in the shooting area due to a zoom change or a user operation on an external device, processing may be performed to learn the shooting area after the change operation as an important area.

Other Embodiments
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  The present invention is not limited to photographing with digital cameras and digital video cameras, and can also be incorporated into imaging devices such as surveillance cameras, web cameras, and mobile phones.

DESCRIPTION OF SYMBOLS 101 Imaging device 301 Smart device 501 Wearable device 104 Tilt rotation unit 105 Pan rotation unit

Claims (10)

Search means for searching for a subject,
First detection means for detecting the state of the device;
Second detecting means for detecting a state related to the searched subject;
A determination unit that determines a shooting process based on the detection result of the state of the device and the detection result of the state of the subject;
According to the result of detection of the state of the device, the result of detection of the state of the subject, and a plurality of data obtained in at least one of the determined photographing processing, regularity is extracted, and according to the extraction result Has learning means to learn the user's preferences,
An imaging apparatus, wherein the determination of the photographing process of the determination unit is changed by learning by the learning unit.   The image pickup apparatus according to claim 1, wherein the first detection means detects at least one of an angle of the apparatus, an amount of vibration of the apparatus, and an amount of movement of the apparatus.   The imaging apparatus according to claim 1, wherein the second detection unit detects at least one of a distance to an object and a distance to an object in a peripheral region of the object.   2. The image pickup apparatus according to claim 1, wherein at least one of the information detected by the first detection unit and the information detected by the second detection unit is recorded.   At least one of still image shooting, still image continuous shooting, moving image shooting, horizontal panoramic shooting, vertical panoramic shooting, and 360-degree panoramic shooting are determined as the photographing processing. An imaging device according to Item 1.   The imaging apparatus according to claim 1, wherein the imaging apparatus performs imaging by two or more types of imaging processing.   The imaging device according to any one of claims 1 to 6, wherein the imaging device changes an operation based on a state of the device.   The imaging device according to any one of claims 1 to 7, wherein the state of the device is based on a change in acceleration within a predetermined time. It is a control method of an imaging device, and
A search step of searching for a subject;
A first detection step of detecting the state of the device;
A second detection step of detecting a state related to the searched subject;
A determination step of determining a photographing process based on a detection result of the state of the device and a detection result of the state of the subject;
According to the result of detection of the state of the device, the result of detection of the state of the subject, and a plurality of data obtained in at least one of the determined photographing processing, regularity is extracted, and according to the extraction result Have a learning process to learn user's preferences,
A control method characterized in that the determination of the photographing process of the determination step is changed by learning in the learning step.   A computer readable storage medium storing a program for executing the control method according to claim 9 by a computer.

Images