JP2019110525A - Imaging device and method of controlling the same, program, and storage medium - Google Patents

Abstract

To suppress that recording of a video which a user wants to photograph is missed as much as possible, in an imaging device that performs automatic photographing.SOLUTION: An imaging device comprises: an imaging unit that images a subject image and outputs image data; a controller that controls whether or not a photographing operation of recording the image data outputted by the imaging unit is executed; and an acquisition unit that acquires information on a frequency of the photographing operation. The controller changes a threshold for deciding whether or not the photographing operation is executed depending on the information on the frequency.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an automatic imaging technique in an imaging device.

  In still image / moving image shooting with an imaging device such as a camera, it is common for the user to determine the shooting target through the finder etc., check the shooting situation by himself, adjust the framing of the shot image, and shoot the image. is there. Conventionally, such an imaging apparatus detects a user operation error or an external environment, notifies the user when it is not suitable for photographing, or controls a camera so as to be in a state suitable for photographing. It is equipped.

  A life log camera (Patent Document 1) is known which periodically and continuously captures an image without performing a shooting instruction by the user for an imaging device that performs shooting by such a user operation. The life log camera is used in a state of being attached to the user's body by a strap or the like, and records a scene that the user sees in daily life as an image at regular time intervals. In life log camera shooting, shooting is performed at a fixed time interval, instead of shooting at the intended timing such as when the user releases the shutter, so unexpected moments that are not normally shot can be left as images. .

Japanese Patent Publication No. 2016-536868 Publication

  However, when the user wears the life log camera, if the automatic shooting is performed periodically, the following problems occur.

  One is that shooting is performed at a constant time interval regardless of the user's intention, so that there is a possibility that the user may miss a picture of the moment when he really wants to shoot. The other is that if the shooting interval is shortened to avoid missed shooting, the power consumption for shooting increases and the available shooting time decreases.

  The present invention has been made in view of the above-mentioned problems, and an object thereof is to make it possible to minimize missed shooting of a video that a user desires to shoot in an imaging device that performs automatic shooting.

  An image pickup apparatus according to the present invention comprises an image pickup means for picking up an object image and outputting image data, a control means for controlling whether or not to carry out a photographing operation for recording image data outputted by the image pickup means, Acquisition means for acquiring information on the frequency of the photographing operation, wherein the control means changes a threshold for determining whether to perform the photographing operation according to the information on the frequency I assume.

  According to the present invention, in an imaging apparatus that performs automatic imaging, in an imaging apparatus that performs automatic imaging, it is possible to minimize missed shooting of a video that the user wants to capture.

BRIEF DESCRIPTION OF THE DRAWINGS The figure which shows typically the external appearance of the camera which is one Embodiment of the imaging device of this invention. FIG. 1 is a block diagram showing an overall configuration of a camera of an embodiment. FIG. 2 is a diagram showing an example of the configuration of a wireless communication system of a camera and an external device. FIG. 2 shows the configuration of an external device. FIG. 2 is a diagram showing the configuration of a camera and an external device. FIG. 2 shows the configuration of an external device. 6 is a flowchart illustrating the operation of a first control unit. 6 is a flowchart illustrating the operation of a second control unit. 6 is a flowchart for explaining the operation of shooting mode processing. The figure for demonstrating area division in a photography picture. The figure for demonstrating control of imaging frequency. The figure explaining a neural network. FIG. 6 is a diagram showing a state in which an image is browsed by an external device. 6 is a flowchart illustrating learning mode determination. 6 is a flowchart illustrating learning processing.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings.

<Configuration of camera>
FIG. 1 is a view schematically showing the appearance of a camera which is an embodiment of the imaging apparatus of the present invention. The camera 101 shown in FIG. 1A is provided with a power switch, an operation member capable of performing a camera operation, and the like. A lens barrel 102 integrally including an imaging lens group and an imaging element as an imaging optical system for imaging a subject image is movably attached to a fixed unit 103 of the camera 101. Specifically, the lens barrel 102 is attached to the fixing unit 103 via the tilt rotation unit 104 and the pan rotation unit 105, which are mechanisms capable of rotationally driving the fixing unit 103.

  The tilt rotation unit 104 has a motor drive mechanism capable of rotationally driving the lens barrel 102 in the pitch direction shown in FIG. 1B, and the pan rotation unit 105 has a yaw shown in FIG. 1B. It has a motor drive mechanism that can be rotationally driven in a direction. That is, the camera 101 has a mechanism for rotationally driving the lens barrel 102 in two axial directions. Each axis shown in FIG. 1B is defined with respect to the position of the fixing unit 103. The angular velocity meter 106 and the accelerometer 107 are disposed at the fixed portion 103 of the camera 101. The vibration of the camera 101 is detected based on the output signals of the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit 104 and the pan rotation unit 105 are rotationally driven to correct the shake of the lens barrel 102, The inclination can be corrected. The angular velocity meter 106 and the accelerometer 107 also detect the movement of the camera based on the measurement result of a fixed period.

  FIG. 2 is a block diagram showing the overall configuration of the camera 101 of the present embodiment. In FIG. 2, the first control unit 223 includes, for example, a CPU (MPU), a memory (DRAM, SRAM), and the like. Then, according to the program stored in the non-volatile memory (EEPROM) 216, various processes are executed to control each block of the camera 101 or to control data transfer between each block. The non-volatile memory 216 is an electrically erasable / recordable memory, and as described above, stores constants, programs and the like for the operation of the first control unit 223.

  In FIG. 2, the zoom unit 201 includes a zoom lens that performs magnification change (enlargement / reduction of the formed object image). The zoom drive control unit 202 drives and controls the zoom unit 201 and detects the focal length at that time. The focus unit 203 includes a focus lens that performs focus adjustment (focus adjustment). The focus drive control unit 204 drives and controls the focus unit 203. The imaging unit 206 includes an imaging element, receives light incident through each lens group, and outputs information of charge corresponding to the light amount to the image processing unit 207 as an analog image signal. The zoom unit 201, the focus unit 203, and the imaging unit 206 are disposed in the lens barrel 102.

  An image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to digital image data obtained by A / D converting an analog image signal, and applies the digital image data after application. Output. The digital image data output from the image processing unit 207 is converted into a recording format such as JPEG format by the image recording unit 208, stored in the memory 215, and transmitted to the video output unit 217 described later.

  The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to rotate the lens barrel 102 in the tilt direction and the pan direction. The apparatus shake detection unit 209 includes an angular velocity meter (gyro sensor) 106 that detects angular velocities in the three axis directions of the camera 101, and an accelerometer (acceleration sensor) 107 that detects acceleration in the three axis directions of the camera 101. Then, based on the signals detected by those sensors, the rotation angle of the device, the shift amount of the device, and the like are calculated.

  The audio input unit 213 acquires an audio signal around the camera 101 by a microphone provided in the camera 101, converts the audio signal into a digital audio signal, and transmits the digital audio signal to the audio processing unit 214. The audio processing unit 214 performs processing related to audio such as optimization processing of the input digital audio signal. Then, the first control unit 223 transmits the audio signal processed by the audio processing unit 214 to the memory 215. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing unit 207 and the audio processing unit 214.

  An image processing unit 207 and an audio processing unit 214 read out an image signal and an audio signal temporarily stored in the memory 215 and perform encoding of the image signal, encoding of the audio signal, and the like, and a compressed image signal and a compressed audio signal. Generate The first control unit 223 transmits the compressed image signal and the compressed audio signal to the recording and reproducing unit 220.

  The recording and reproducing unit 220 records, on the recording medium 221, the compressed image signal and the compressed sound signal generated by the image processing unit 207 and the sound processing unit 214, and other control data related to photographing. When the audio signal is not compressed and encoded, the first control unit 223 causes the recording / reproducing unit 220 to transmit the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207. It transmits and makes the recording medium 221 record.

  The recording medium 221 may be a recording medium built in the camera 101 or a removable recording medium, and can record various data such as a compressed image signal generated by the camera 101, a compressed audio signal, and an audio signal. In general, a medium having a larger capacity than the non-volatile memory 216 is used as the recording medium 221. For example, the recording medium 221 includes a recording medium of any type such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a non-volatile semiconductor memory, and a flash memory.

  The recording and reproducing unit 220 reads (reproduces) the compressed image signal, the compressed audio signal, the audio signal, various data, and the program recorded on the recording medium 221. Then, the first control unit 223 transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing unit 207 and the audio processing unit 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode the signal according to a predetermined procedure, and transmit the decoded signal to the video output unit 217.

  A plurality of microphones are disposed in the voice input unit 213, and the voice processing unit 214 can detect the direction of sound with respect to the plane on which the plurality of microphones are installed, and is used for searching for subjects and automatic photographing described later. . Further, the voice processing unit 214 detects a specific voice command. The voice command may be configured so that the user can register a specific voice to the camera in addition to some commands registered in advance. It also performs sound scene recognition. In sound scene recognition, a sound scene is determined by a network learned by machine learning in advance based on a large amount of audio data. For example, a network for detecting a specific scene, such as “Cheering,” “Applause,” “Speaking,” is set in the audio processing unit 214, and a specific sound scene or a specific sound is specified. Detect voice commands. When the voice processing unit 214 detects a specific sound scene or a specific voice command, it outputs a detection trigger signal to the first control unit 223 or the second control unit 211.

  In addition to the first control unit 223 that controls the entire main system of the camera 101, a second control unit 211 that controls the power supply of the first control unit 223 is provided. The first power supply unit 210 and the second power supply unit 212 respectively supply power for operating the first control unit 223 and the second control unit 211. By pressing the power button provided on the camera 101, power is first supplied to both the first control unit 223 and the second control unit 211. However, as described later, the first control unit 223 controls the first power supply unit. It also performs control to turn off its own power supply to 210. Even while the first control unit 223 is not in operation, the second control unit 211 is in operation, and information from the device shake detection unit 209 and the audio processing unit 214 is input. The second control unit 211 determines whether or not to start the first control unit 223 based on various input information, and when it is determined that the first control unit 223 is to be started, the first control unit 223 is used as the first control unit 223. To supply power to

  The audio output unit 218 outputs an audio pattern set in advance from a speaker incorporated in the camera 101, for example, at the time of shooting. The LED control unit 224 turns on an LED provided in the camera 101 based on a preset lighting pattern or blinking pattern, for example, at the time of photographing or the like. The video output unit 217 includes, for example, a video output terminal, and outputs an image signal to display a video on a connected external display or the like. Further, the audio output unit 218 and the video output unit 217 may be one combined terminal, for example, a terminal such as an HDMI (registered trademark: High-Definition Multimedia Interface) terminal.

  The communication unit 222 is a unit that performs communication between the camera 101 and an external device, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. Also, the camera 101 is driven based on an instruction from the external apparatus by receiving a command for start and end of shooting, and control signals related to shooting such as pan / tilt and zoom driving. Further, information such as various parameters related to learning processed by a learning processing unit 219 described later is transmitted and received between the camera 101 and an external device. The communication unit 222 includes, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a wireless USB (registered trademark), or a GPS receiver.

  The environment sensor 226 detects the state of the environment around the camera 101 at a predetermined cycle. The environment sensor 226 includes a temperature sensor that detects a temperature around the camera 101, an air pressure sensor that detects a change in air pressure around the camera 101, and an illuminance sensor that detects brightness around the camera 101. Furthermore, it has a humidity sensor that detects the humidity around the camera 101, a UV sensor that detects the amount of ultraviolet light around the camera 101, and the like. In addition to the detected temperature information, barometric pressure information, brightness information, humidity information, and UV information, the temperature change amount, pressure change amount, brightness change amount, humidity change, etc. are calculated by calculating the change rate at predetermined time intervals from the detected various information. It is used for determination of automatic photographing etc. which mention the amount and the amount of change of ultraviolet rays later.

<Communication with external device>
FIG. 3 is a view showing a configuration example of a wireless communication system of the camera 101 and the external device 301. As shown in FIG. The camera 101 is a digital camera having a photographing function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

  The camera 101 and the external device 301 are, for example, a first communication 302 based on a wireless LAN conforming to the IEEE 802.11 standard series, and a control station and a dependent station such as Bluetooth Low Energy (hereinafter, referred to as “BLE”). Communication is possible with a second communication 303 having a master-slave relationship. The wireless LAN and BLE are an example of a communication method, and each communication apparatus has two or more communication functions, for example, one communication function that performs communication in the relationship between the control station and the dependent station. Other communication methods may be used as long as control of the other communication function can be performed. However, the first communication 302 such as a wireless LAN can perform higher-speed communication than the second communication 303 such as BLE, and the second communication 303 consumes less power than the first communication 302. And / or the communicable distance is short.

  The configuration of the external device 301 will be described with reference to FIG. The external apparatus 301 includes, for example, a public wireless control unit 406 for public wireless communication in addition to the wireless LAN control unit 401 for wireless LAN and the BLE control unit 402 for BLE. In addition, the external device 301 further includes a packet transmission / reception unit 403. The wireless LAN control unit 401 performs RF control of wireless LAN, communication processing, driver processing for performing various control of communication by wireless LAN conforming to the IEEE 802.11 standard series, and protocol processing regarding communication by wireless LAN. The BLE control unit 402 performs RF control of BLE, communication processing, driver processing for performing various controls of communication by BLE, and protocol processing regarding communication by BLE. The public wireless control unit 406 performs RF control of public wireless communication, communication processing, driver processing for performing various controls of public wireless communication, and protocol processing related to public wireless communication. The public wireless communication conforms to, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmission / reception unit 403 performs processing for performing transmission and / or reception of packets related to communication by wireless LAN and BLE and public wireless communication. In the present embodiment, the external device 301 is described as performing at least one of transmission and reception of packets in communication, but other communication formats such as circuit switching may be used other than packet switching. It is also good.

  The external device 301 further includes, for example, a control unit 411, a storage unit 404, a GPS reception unit 405, a display unit 407, an operation unit 408, a voice input voice processing unit 409, and a power supply unit 410. The control unit 411 controls the entire external device 301, for example, by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program executed by the control unit 411 and various information such as parameters required for communication. Various operations described later are realized by the control unit 411 executing a control program stored in the storage unit 404.

  The power supply unit 410 supplies power to the external device 301. The display unit 407 has a function capable of outputting visually recognizable information, such as an LCD or an LED, or a sound output such as a speaker, and displays various information. The operation unit 408 includes, for example, a button for receiving an operation of the external device 301 by the user. The display unit 407 and the operation unit 408 may be configured by a common member such as a touch panel, for example.

  The voice input voice processing unit 409 may be configured to obtain a voice uttered by the user using, for example, a general-purpose microphone built in the external device 301, and identify a user's operation command by voice recognition processing. Also, using a dedicated application in the external device 301, a voice command is acquired by the user's pronunciation, and a specific voice for causing the voice processing unit 214 of the camera 101 to recognize via the first communication 302 by wireless LAN It can also be registered as a command.

  A GPS (Global Positioning System) reception unit 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude / latitude information) of the external device 301. Alternatively, the current position of the external device 301 may be estimated based on information of a wireless network existing in the vicinity using a Wi-Fi Positioning System (WPS) or the like. When the acquired current GPS position information is located in the position range (within the range of the predetermined radius centering on the detection position) which is set in advance in advance, or the GPS position information changes in position more than the predetermined position In this case, the movement information is notified to the camera 101 via the BLE control unit 402. Then, it is used as a parameter for automatic shooting and automatic editing described later.

  As described above, the camera 101 and the external device 301 exchange data by communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, it transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. In addition, transmission of a photographing instruction from the external device 301 to the camera 101, transmission of voice command registration data, transmission of a predetermined position detection notification based on GPS position information, transmission of a location movement notification, and the like are performed. Also, it performs transmission and reception of learning data using a dedicated application in the external device 301.

<Configuration of accessories>
FIG. 5 is a view showing an example of the arrangement of an external apparatus 501 capable of communicating with the camera 101. As shown in FIG. The camera 101 is a digital camera having a photographing function, and the external apparatus 501 is a wearable device including various sensing units capable of communicating with the camera 101 by, for example, a Bluetooth communication module.

  The external device 501 is configured to be worn on, for example, the arm of the user, and is a sensor that detects biological information such as the pulse, heart rate, blood flow, etc. of the user at a predetermined cycle, and an acceleration sensor that can detect the motion state of the user. Etc. are mounted.

  The biological information detection unit 602 is, for example, a pulse sensor that detects a pulse, a heart rate sensor that detects a heart rate, a blood flow sensor that detects a blood flow, a sensor that detects a change in potential by contact of skin using a conductive polymer. including. The present embodiment will be described using a heart rate sensor as the living body information detection unit 602. The heart rate sensor irradiates the skin with infrared light using, for example, an LED or the like, and detects the heart rate of the user by detecting the infrared light transmitted through the body tissue with a light receiving sensor and processing the signal. The biological information detection unit 602 outputs the detected biological information as a signal to the control unit 607 (see FIG. 6).

  The shake detection unit 603 that detects the user's motion state includes, for example, an acceleration sensor or a gyro sensor, and performs motion such as whether the user is moving or swings an arm based on acceleration information. It can be detected. In addition, an operation unit 605 for receiving an operation of the external apparatus 501 by the user, and a display unit 604 such as a monitor for outputting visually recognizable information such as an LCD or LED are mounted.

  FIG. 6 is a diagram showing the configuration of the external device 501. As shown in FIG. As described above, the external device 501 includes, for example, the control unit 607, the communication unit 601, the biological information detection unit 602, the shake detection unit 603, the display unit 604, the operation unit 605, the power supply unit 606, and the storage unit 608.

  The control unit 607 controls the entire external device 501, for example, by executing a control program stored in the storage unit 608. The storage unit 608 stores, for example, a control program executed by the control unit 607 and various information such as parameters required for communication. Various operations described later are realized, for example, by the control unit 607 executing a control program stored in the storage unit 608.

  The power supply unit 606 supplies power to the external device 501. The display unit 604 includes, for example, an output unit of visually recognizable information such as an LCD or an LED, or an output unit capable of sound output such as a speaker, and displays various information. The operation unit 605 includes, for example, a button for receiving an operation of the external device 501 by the user. The display unit 604 and the operation unit 605 may be configured by a common member such as a touch panel, for example. In addition, the operation unit 605 may be configured to acquire, for example, voice uttered by the user using a general-purpose microphone built in the external device 501, and to identify a user's manipulation instruction by voice recognition processing.

  The communication unit 601 transmits the various detection information acquired by the biological information detection unit 602 and the shaking detection unit 603 and processed by the control unit 607 to the camera 101. For example, detection information can be transmitted to the camera 101 at the timing at which a change in the user's heartbeat is detected, or detection information can be transmitted at the timing of a change in moving state such as walking movement / running movement / stop. Further, detection information can be transmitted at a timing at which a pre-set arm-like motion is detected, or detection information can be transmitted at a timing at which movement of a preset distance is detected.

<Operation sequence of camera>
FIG. 7 is a flow chart for explaining an example of the operation of the first control unit 223 of the camera 101 in the present embodiment.

  When the user operates a power button provided on the camera 101, power is supplied from the first power supply unit 210 to each block of the first control unit 223 and the camera 101. Similarly, power is supplied from the second power supply unit 212 to the second control unit 211. The details of the operation of the second control unit 211 will be described later using the flowchart of FIG.

When power is supplied, the process of FIG. 7 starts. In step S701, the start condition is read. In the present embodiment, there are the following three cases where the power is turned on.
(1) The power button is manually pressed to activate the power (2) An external device (for example, the external device 301) sends an activation instruction by external communication (for example, BLE communication) to activate the power (3) The power is activated by the instruction of the second control unit 211. Here, when the power is activated by the instruction of the second control unit 211 of (3), the activation condition calculated in the second control unit 211 is Although it will be read, the details will be described later with reference to FIG. Further, although the start condition read in here is used as one parameter element at the time of subject search and automatic photographing, this will also be described later. When the start condition reading is completed, the process proceeds to step S702.

  In step S702, the detection signals of various sensors are read. One of the sensor signals read in here is a sensor signal for detecting vibration of a gyro sensor, an acceleration sensor, or the like in the device shake detection unit 209. In addition, it is a signal of the rotational position of the tilt rotation unit 104 or the pan rotation unit 105. Further, it includes an audio signal detected by the audio processing unit 214, a detection trigger signal of specific speech recognition, a sound direction detection signal, a detection signal of environmental information detected by the environmental sensor 226, and the like. When the detection signals of various sensors are read in step S702, the process proceeds to step S703.

  In step S703, it is detected whether a communication instruction has been transmitted from the external device, and when there is a communication instruction, communication with the external device is performed. For example, remote operation via wireless LAN or BLE from external device 301, transmission / reception of audio signal, image signal, compressed audio signal, compressed image signal, etc., operation instruction for imaging from external device 301, voice command registration data , Notification of predetermined position detection based on GPS position information, notification of location movement, transmission / reception of learning data, etc. Also, when there is an update of biological information such as user's exercise information, arm action information, and heartbeat from the external device 501, information is read via BLE. The environment sensor 226 described above may be mounted on the camera 101, but may be mounted on the external device 301 or the external device 501. In that case, in step S703, environmental information is also read via BLE. When communication reading from the external device is performed in step S703, the process proceeds to step S704.

  In step S704, mode setting determination is performed, and the process proceeds to step S705. In step S705, it is determined in step S704 whether the operation mode is set to the low power consumption mode. If the mode is not one of "automatic shooting mode", "automatic editing mode", "automatic image transfer mode", "learning mode", and "automatic file deletion mode" described later, the low power consumption mode is set. It is judged. If it is determined in step S705 that the low power consumption mode is set, the process proceeds to step S706.

  In step S706, the second control unit 211 (SubCPU) is notified of various parameters (the shake detection determination parameter, the sound detection parameter, and the time lapse detection parameter) related to the activation factor determined in the second control unit 211. The values of the various parameters change as they are learned in a learning process described later. When the process of step S706 is finished, the process proceeds to step S707, the power supply of the first control unit 223 (Main CPU) is turned off, and the process is finished.

  If it is determined in step S705 that the mode is not the low power consumption mode, it is determined whether the mode setting in step S704 is the automatic shooting mode. Here, the process of mode setting determination in step S704 will be described. The mode to be determined is selected from the following.

(1) Automatic shooting mode <mode judgment condition>
From each detection information (image, sound, time, vibration, place, physical change, environmental change) that has been set for learning, elapsed time since transition to automatic shooting mode, information such as past shooting information and number of shots, When it is determined that the automatic shooting should be performed, the automatic shooting mode is set.

<In-mode processing>
In the automatic photographing mode processing (step S710), pan / tilt and zoom are driven to automatically search for a subject based on each detection information (image, sound, time, vibration, place, body change, environment change). Then, when it is determined that it is the timing at which the user's favorite shooting can be performed, shooting is automatically performed.

(2) Automatic editing mode <mode judgment condition>
If it is determined that the automatic editing should be performed from the elapsed time since the previous automatic editing and the past photographed image information, the automatic editing mode is set.

<In-mode processing>
In the automatic editing mode process (step S 712), a still image or moving image selection process is performed based on learning, and based on the learning, a highlight moving image summarized into one moving image by an image effect, a time of edited moving image, etc. An automatic editing process is performed to create

(3) Image transfer mode <mode determination condition>
When an automatic image transfer mode is set by an instruction using a dedicated application in the external device 301, it is determined that automatic transfer should be performed based on the elapsed time since the previous image transfer and the past captured image information Then, the automatic image transfer mode is set.

<In-mode processing>
In the image automatic transfer mode process (step S714), the camera 101 automatically extracts an image that will suit the user's preference, and automatically transfers an image that seems to be the user's preference to the external apparatus 301. The image extraction of the user's preference is performed based on the score that determines the user's preference added to each image described later.

(4) Learning mode <mode judgment condition>
If it is determined that automatic learning should be performed from the elapsed time since the previous learning process and information integrated with the image that can be used for learning, the number of learning data, etc. It is set. Alternatively, when there is an instruction to set the learning mode via the communication from the external device 301, this mode is also set.

<In-mode processing>
In the learning mode process (step S 716), each operation information in the external device 301 (image acquisition information from the camera, information manually edited through a dedicated application, determination value information input by the user with respect to the image in the camera) Based on a notification of learning information from the external device 301, etc., learning is performed according to the user's preference using a neural network. Further, learning regarding detection such as registration of personal authentication, voice registration, sound scene registration, general object recognition registration, etc., and learning of the conditions of the above-mentioned low power consumption mode, etc. are simultaneously performed.

(5) File automatic deletion mode <mode judgment condition>
If it is determined that automatic file deletion should be performed based on the elapsed time since the previous automatic file deletion and the remaining capacity of the non-volatile memory 216 storing the image, the automatic file deletion mode is set. Ru.

<In-mode processing>
In the file automatic deletion mode process (step S 718), a file to be automatically deleted is specified and deleted from the images in the non-volatile memory 216 based on the tag information of each image and the photographed date and time.

  Details of processing in each of the above modes will be described later.

  Referring back to FIG. 7, when it is determined in step S705 that the mode is not the low power consumption mode, the process proceeds to step S709, and it is determined whether the mode setting is the automatic shooting mode. If it is determined that the mode is the automatic shooting mode, the process proceeds to step S710, and the automatic shooting mode process is performed. When the process ends, the process returns to step S702 to repeat the process. If it is determined in step S709 that the mode is not the automatic shooting mode, the process proceeds to step S711.

  In step S711, it is determined whether the mode setting is the automatic editing mode. If the mode is the automatic editing mode, the process advances to step S712 to perform automatic editing mode processing. When the process ends, the process returns to step S702 to repeat the process. If it is determined in step S711 that the automatic editing mode is not set, the process proceeds to step S713. The automatic editing mode is not directly related to the subject matter of the present invention, and thus the detailed description is omitted.

  In step S713, it is determined whether the mode setting is the automatic image transfer mode. If the automatic image transfer mode is selected, the process proceeds to step S714, and automatic image transfer mode processing is performed. When the process ends, the process returns to step S702 to repeat the process. If it is determined in step S713 that the mode is not the automatic image transfer mode, the process proceeds to step S715. The automatic image transfer mode is not directly related to the subject matter of the present invention, and thus the detailed description is omitted.

  In step S715, it is determined whether the mode setting is the learning mode. If the mode setting is the learning mode, the process proceeds to step S716, and learning mode processing is performed. When the process ends, the process returns to step S702 to repeat the process. If it is determined in step S715 that the learning mode is not set, the process proceeds to step S717.

  In step S717, it is determined whether the mode setting is the automatic file deletion mode. If the automatic file deletion mode is selected, the process proceeds to step S718, and the automatic file deletion mode processing is performed. When the process ends, the process returns to step S702 to repeat the process. If it is determined in step S717 that the mode is not the file automatic deletion mode, the process returns to step S702 to repeat the processing. The automatic file deletion mode is not directly related to the subject matter of the present invention, and thus the detailed description is omitted.

  FIG. 8 is a flow chart for explaining an example of the operation of the second control unit 211 of the camera 101 in the present embodiment.

  When the user operates a power button provided on the camera 101, power is supplied from the first power supply unit 210 to each block of the first control unit 223 and the camera 101. Similarly, power is supplied from the second power supply unit 212 to the second control unit 211.

  When power is supplied, the second control unit (SubCPU) 211 is activated, and the process of FIG. 8 starts. In step S801, it is determined whether a predetermined sampling period has elapsed. The predetermined sampling cycle is set to, for example, 10 msec, and the process proceeds to step S802 in a cycle of 10 msec. If it is determined that the predetermined sampling cycle has not elapsed, the second control unit 211 waits as it is.

In step S802, reading of learning information is performed. The learning information is information transferred when communicating information to the second control unit 211 in step S706 of FIG. 7, and includes, for example, the following information.
(1) Determination of specific swing detection (used in step S804 described later)
(2) Determination of specific sound detection (used in step S805 described later)
(3) Determination of passage of time (used in step S 807 described later)
When the learning information is read in step S802, the process proceeds to step S803, and a shake detection value is acquired. The shake detection value is an output value of a gyro sensor, an acceleration sensor, or the like in the device shake detection unit 209.

  When the shake detection value is acquired in step S803, the process proceeds to step S804, and detection processing of a specific shake state set in advance is performed. Here, the determination process is changed according to the learning information read in step S802. Some examples are described.

<Tap detection>
It is possible to detect a state (tap state) in which the user taps the camera 101 with, for example, a fingertip from the output value of the acceleration sensor 107 attached to the camera 101. By passing the output of the three-axis acceleration sensor 107 through a band pass filter (BPF) set in a specific frequency region at a predetermined sampling cycle, it is possible to extract a signal region of acceleration change due to a tap. Tap detection is performed depending on whether the number of times the acceleration signal after passing through the BPF exceeds the predetermined threshold ThreshA is the predetermined number CountA during the predetermined time TimeA. For double taps, CountA is set to 2, and for triple taps, CountA is set to 3. Further, Time A and Thresh A can also be changed by learning information.

<Detection of shaking condition>
The shaking state of the camera 101 can be detected from the output values of the gyro sensor 106 and the acceleration sensor 107 attached to the camera 101. High frequency components of the outputs of the gyro sensor 106 and the acceleration sensor 107 are cut by a high pass filter (HPF), low frequency components are cut by a low pass filter (LPF), and absolute value conversion is performed. The vibration detection is performed based on whether or not the number of times the calculated absolute value exceeds the predetermined threshold value ThreshB is equal to or more than the predetermined number of times CountB during the predetermined time TimeB. Thus, it is possible to determine, for example, whether the shaking is small when the camera 101 is placed on a desk or the like, or whether the shaking is large when walking while wearing the camera 101 as a wearable camera. Further, by providing a plurality of conditions for the determination threshold value and the determination count number, it is possible to detect a fine swing state according to the swing level. Time B, Thresh B, and Count B can also be changed by learning information.

  In the above, the method of detecting a specific shaking condition by determining the detection value of the shaking detection sensor has been described. However, from the data of the shake detection sensor sampled within a predetermined time, a specific shake state registered in advance by the learned neural network can be input by inputting it to the shake state determiner using a neural network. It is also possible to detect. In that case, the learning information reading in step S802 is a weight parameter of the neural network.

  When the detection process of the specific swing state is performed in step S804, the process proceeds to step S805, and the detection process of the specific sound set in advance is performed. Here, the detection determination process is changed according to the learning information read in step S802. Some examples are described.

<Specific voice command detection>
Detect specific voice commands. In addition to several pre-registered voice commands, the user can register a specific voice to the camera.

<Specific sound scene recognition>
Based on a large amount of voice data, a sound scene is determined by a network learned by machine learning. For example, it detects a specific scene such as "Cheers up", "Applause", "Speaking" or the like. The scene to detect changes with learning.

<Sound level judgment>
Sound level detection is performed by determining whether the magnitude of the sound level exceeds a predetermined magnitude for a predetermined time. The predetermined time, the predetermined size, and the like change by learning.

<Sound direction judgment>
With a plurality of microphones arranged on a plane, the direction of sound is detected for sound of a predetermined size.

  The above determination process is performed in the voice processing unit 214, and it is determined in step S805 whether or not a specific sound has been detected by each setting learned in advance.

  After the specific sound detection process is performed in step S805, the process proceeds to step S806, in which it is determined whether the power of the first control unit 223 is in the OFF state. If the first control unit 223 (Main CPU) is in the OFF state, the process proceeds to step S 807, and the process of detecting the passage of time set in advance is performed. Here, the detection determination process is changed according to the learning information read in step S802. The learning information is information transferred when communicating information to the second control unit 211 in step S706 described with reference to FIG. The elapsed time from when the first control unit 223 transitions from ON to OFF is measured, and if the elapsed time is equal to or longer than a predetermined time TimeC, it is determined that the time has elapsed, and if shorter than TimeC, the time has elapsed It is judged that it has not done. TimeC is a parameter that changes according to learning information.

When the time lapse detection process is performed in step S807, the process proceeds to step S808, and it is determined whether a condition for canceling the low power consumption mode is satisfied. The low power consumption mode cancellation is determined by the following conditions.
(1) that a specific vibration has been detected (2) that a specific sound has been detected (3) that a predetermined time has elapsed (1) For the specific vibration state detection processing in step S804, It is determined whether or not the movement of the vehicle has been detected. As for (2), it is determined by the specific sound detection process in step S805 whether or not a specific sound is detected. For (3), it is determined by the time lapse detection process in step S 807 whether a predetermined time has elapsed. If at least one of (1) to (3) is established, it is determined that low power consumption mode cancellation is performed.

  If it is determined in step S808 that the release of the low power consumption mode is determined, the process proceeds to step S809, the power of the first control unit 223 is turned on, and the conditions under which the release of the low power consumption mode is determined in step S810 The first control unit 223 is notified of any of the time). Then, the process returns to step S801 to loop the process. If it is determined in step S808 that any of the release conditions is not satisfied and it is determined that the low power consumption mode release condition is not satisfied, the process returns to step S801 to loop the process.

  On the other hand, if it is determined in step S806 that the first control unit 223 is in the ON state, the process advances to step S811 to notify the first control unit 223 of the information acquired in steps S803 to S805, and returns to step S801. Loop the process.

  In the present embodiment, even when the first control unit 223 is in the ON state, the second control unit 211 performs the shake detection and the detection of the specific sound and notifies the first control unit 223 of the detection result. . However, when the first control unit 223 is ON, the processing in steps S 803 to S 805 is not performed, and the configuration in the first control unit 223 (in step S 702 in FIG. 7) performs vibration detection and specific sound detection. May be

  As described above, the conditions for transitioning to the low power consumption mode and the conditions for canceling the low power consumption mode are learned based on the user's operation by performing steps S704 to S707 in FIG. 7 and the processing in FIG. Ru. Then, the camera operation can be performed in accordance with the convenience of the user who owns the camera 101. The method of learning will be described later.

  In the above, although the method of canceling the low power consumption mode by vibration detection, sound detection, or time lapse has been described in detail, the low power consumption mode may be canceled by environmental information. Environmental information can be determined based on whether the absolute amount or change amount of temperature, barometric pressure, brightness, humidity, or ultraviolet light amount exceeds a predetermined threshold, and the threshold can be changed by learning described later.

  Further, it is also possible to judge the cancellation of the low power consumption mode by judging the vibration detection, the sound detection, the detection information of time lapse, the absolute value or the change amount of each environment information based on the neural network. In this determination process, the determination conditions can be changed by learning described later.

<Automatic shooting mode processing>
The automatic photographing mode process will be described with reference to FIG. First, in step S901, the image processing unit 207 performs image processing on a signal acquired by the imaging unit 206 to generate an image for object detection. An object detection process for detecting a person or an object is performed on the generated image.

  When detecting a person, the face or the human body of the subject is detected. In the face detection process, a pattern for determining the face of a person is determined in advance, and a portion matching the pattern in the captured image can be detected as a face region of the person. In addition, the degree of reliability indicating the likelihood of the subject's face is also calculated at the same time. The reliability is calculated from, for example, the size of the face area in the image, the degree of matching with the face pattern, and the like. Similarly for object recognition, it is possible to recognize an object that matches the pattern registered in advance.

  In addition, there is also a method of extracting a characteristic subject using a histogram such as hue or saturation in a captured image. With regard to the image of the subject captured within the shooting angle of view, processing of dividing the distribution derived from the histogram of the hue, saturation, etc. into a plurality of sections and classifying the images captured for each section is executed. For example, a histogram of a plurality of color components is created for the captured image, divided by the mountain-shaped distribution range, the captured image is classified in the region belonging to the combination of the same sections, and the image region of the subject is recognized Be done. By calculating the evaluation value for each image area of the recognized subject, the image area of the subject having the highest evaluation value can be determined as the main subject area. Each subject information can be obtained from the imaging information by the above method.

  In step S902, the image blur correction amount is calculated. Specifically, first, the absolute angle of camera shake is calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the tilt rotation unit 104 and the pan rotation unit 105 are moved in the angular direction that cancels the absolute angle to obtain an angle for correcting the image blur, and this is used as the image blur correction amount. In the image blur correction amount calculation process here, the calculation method can be changed by a learning process described later.

  In step S903, the state of the camera is determined. It is determined what kind of vibration / motion state of the camera is at present based on the camera angle detected by angular velocity information, acceleration information, GPS position information, and the like. For example, when the camera 101 is attached to a car and shooting is performed, subject information such as a surrounding landscape changes largely depending on the distance moved. Therefore, it is determined whether or not it is a "vehicle moving state" which is attached to a car or the like and is moving at a high speed, and is used for automatic subject search described later. In addition, it is determined whether or not the change in the angle of the camera is large, and it is determined whether or not the camera 101 is in the "posting state" with almost no shaking. In the case of the “post-taking state”, it may be considered that there is no change in the position of the camera 101 itself, so that it is possible to perform a subject search for the post-taking. In addition, when the angle change of the camera is relatively large, it is determined that the "hand-held state", and it is possible to perform a hand-held subject search.

In step S904, a subject search process is performed. The subject search is configured by the following processing.
(1) Area division (2) Calculation of importance level for each area (3) Determination of search target area Hereinafter, each processing will be sequentially described.

(1) Area Division Area division will be described using FIG. 10A. As shown in FIG. 10A (a), area division is performed all around the position of the camera (the origin O is the camera position). In the example of FIG. 10A (a), the tilt direction and the pan direction are divided at 22.5 degrees each. When division is performed as shown in FIG. 10A (a), the horizontal circumference decreases and the area area decreases as the angle in the tilt direction deviates from 0 degrees. Therefore, as shown in FIG. 10A (b), when the tilt angle is 45 degrees or more, the area range in the horizontal direction is set larger than 22.5 degrees.

  FIGS. 10A (c) and 10A (d) show examples of the area divided region within the shooting angle of view. An axis 1301 is the direction of the camera 101 at the time of initialization, and area division is performed with this direction as a reference position. Reference numeral 1302 denotes the angle of view area of the image being captured, and an example of the image at that time is shown in FIG. 10A (d). In the image of the angle of view being imaged, the image is divided as shown by reference numerals 1303 to 1318 in FIG. 10A (d) based on the area division.

(2) Calculation of Importance Level for Each Area For each area divided as described above, the importance level indicating the priority to be searched is calculated according to the situation of the subject present in the area and the situation of the scene. . The importance level based on the condition of the subject is, for example, the number of persons present in the area, the size of the face of the person, the direction of the face, the certainty of face detection, the facial expression of the person, the personal authentication result of the person, etc. Calculated based on In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene discrimination result (blue sky, back light, sunset scene etc.), sound level from the direction of area and voice recognition result, motion detection in area Calculated based on the information etc.

  In addition, when camera vibration is detected in the camera state determination (step S 903) in FIG. 9, the importance level can also be changed according to the vibration state. For example, when it is determined that the “lay-in state” is set, it is determined that a subject search is performed centering on a high priority subject (for example, the owner of the camera) registered in the face authentication. In addition, automatic photographing described later is also performed with priority given to the face of the owner of the camera, for example. As a result, even if the camera owner wears the camera and takes a long time to take a picture, removing the camera and placing it on a desk can leave many images captured by the owner. it can. At this time, it is possible to search for a face by pan / tilt, so even if you don't think about the camera's placement angle, etc., you just need to install it properly and you will see an image of the owner and a group photo with many faces. You can leave it.

  It should be noted that under the above conditions alone, as long as there is no change in each area, the area with the highest importance level will be the same, and as a result, the area to be searched will remain unchanged. Therefore, the importance level is changed according to the past shooting information. Specifically, the area continuously designated as the search area for a predetermined time may be lowered in importance level, or may be lowered for a predetermined time in an area photographed in step S 910 described later. .

(3) Determination of Search Target Area When the importance level of each area is calculated as described above, an area having a high importance level is determined as a search target area. Then, the pan / tilt search target angle required to capture the search target area at the angle of view is calculated.

  Returning to the explanation of FIG. 9, in step S905, pan / tilt driving is performed. Specifically, the pan / tilt driving amount is calculated by adding the image blur correction amount at the control sampling frequency and the driving angle based on the pan / tilt search target angle. Then, the lens barrel rotation drive unit 205 controls driving of the tilt rotation unit 104 and the pan rotation unit 105 respectively.

  In step S906, the zoom unit 201 is controlled to perform zoom driving. Specifically, the zoom is driven according to the state of the search target subject determined in step S904. For example, in the case where the search target subject is the face of a person, if the face on the image is too small, detection is not possible because the size is smaller than the minimum detectable size, and there is a risk of losing sight. In such a case, control is performed to increase the size of the face on the image by zooming to the telephoto side. On the other hand, when the face on the image is too large, the subject or the camera itself easily moves the subject out of the angle of view. In such a case, control is performed to reduce the size of the face on the screen by zooming to the wide angle side. By performing zoom control as described above, it is possible to maintain a state suitable for tracking an object.

  In step S 907, it is determined whether there is a manual imaging instruction. If there is a manual imaging instruction, the process proceeds to step S 910. At this time, the manual photographing instruction may be either by pressing the shutter button, by tapping the camera housing with a finger or the like (tap), by an audio command input, or by an instruction from an external device. A shooting instruction triggered by a tap operation is determined by the device shake detection unit 209 detecting, in a short period, continuous high-frequency acceleration when the user taps the camera housing. The voice command input is a shooting instruction method of recognizing a voice by the voice processing unit 214 and using it as a trigger for shooting when the user utters a combination (for example, "take a photo" or the like) instructing a predetermined shooting. The instruction from the external device is, for example, a photographing instruction method using as a trigger a shutter instruction signal transmitted using a dedicated application from a smartphone or the like connected to the camera and BlueTooth.

  If it is determined in step S 907 that a manual imaging instruction has not been issued, the process advances to step S 908 to perform automatic imaging determination. In the automatic shooting determination, it is determined whether or not automatic shooting is to be performed, and determination of a shooting method (whether still image shooting, moving image shooting, continuous shooting, panoramic shooting, or the like is to be performed).

<Determining Whether to Perform Automatic Shooting>
The determination as to whether or not to perform automatic shooting (shooting operation for recording image data output by the imaging unit) is performed as follows. Specifically, it is determined to execute automatic shooting in the following two cases. First, based on the importance level of each area obtained in step S904, when the importance level exceeds a predetermined value, it is determined that automatic shooting is to be performed. The second is determination based on a neural network, which will be described later. Note that the recording here may be recording of image data in the memory 215 or recording of image data in the non-volatile memory 216. Also, it includes one that automatically transfers an image to the external device 301 and records image data on the external device 301 side.

  In the present embodiment, as described above, control is performed such that shooting is automatically performed by an automatic shooting determination parameter such as the importance level. The following problems occur in an imaging apparatus that performs automatic shooting when a predetermined condition is satisfied.

  One is when the frequency of automatic shooting is high. Even if you want to shoot evenly within the determined time, shooting will be performed if the predetermined conditions are met, so the shooting frequency will be very high in the first half, and the battery will remain in the second half. It may happen that the quantity / card remaining amount is insufficient and shooting becomes impossible.

  The other is the case where the frequency of automatic shooting is low. Even when a vendor or the like wants to shoot a predetermined number of photos, the predetermined conditions for automatic shooting may not be satisfied easily, and the number of captured photos may run short.

  Therefore, in order to control the shooting frequency, it may be preferable to change the determination parameter of the automatic shooting depending on the situation of the spot or the situation of the camera.

For example, in events such as weddings where time is limited, the following automatic photographing control tends to be preferred.
(1) I want to shoot a large number of images including people and things (2) I want to shoot without worrying about the remaining amount of battery and recording media because it is short time shooting (3) Pan, tilt On the other hand, when it is desired to record an event during the entire day for shooting in such a limited time, it is likely that automatic shooting control as described below is preferred.
(1) I want to sort the subject to some extent because it is shooting for a long time (2) I want to shoot with energy saving considering the remaining amount of battery and recording media (3) Pan, tilt control consumes the battery than normal control As the above control example is described with reference to FIG. 10B (a). First, imaging conditions (for example, wedding: 2 hours) such as imaging time T (total imaging time) are input according to a user's instruction (for example, the external device 301 or voice input). Based on the input information, the remaining amount of the battery (such as the first power supply unit 210), and the remaining amount of the recording medium (such as the recording medium 221), the target number of captured images S is determined.

  The monitoring is performed at a constant time interval, and the determination threshold value for automatic shooting and the camera control parameter are updated as needed based on the monitoring result. Note that the determination threshold value for automatic shooting and the initial value of the camera control parameter are determined by the learning result up to that point or the determination using a neural network.

  In the example shown in FIG. 10B (a), the parameters are updated so that the number of captured images falls within a region R indicated by a broken line in the drawing. In FIG. 10B (a), the horizontal axis indicates the elapsed time, and the vertical axis indicates the cumulative number of captured images. The number of shots is controlled to be within the range R. If shooting is performed so that the cumulative number of shots increases approximately linearly (linearly) with the passage of time, shooting is performed almost uniformly over the entire shooting time. Because it is considered possible to

  In the example of FIG. 10B (a), it is determined that the number of shots is insufficient at monitoring time A, the determination threshold for automatic shooting is lowered, the pan and tilt movable range is expanded, and the main subject is actively searched. Increase the frequency of shooting. On the other hand, at monitoring time B, it is determined that the number of captured images is large, the determination threshold for automatic imaging is increased, the pan and tilt movable range is narrowed, and the imaging frequency is decreased. At the monitoring time C, it is determined that the appropriate number of captured images is obtained, and the control threshold of the automatic imaging determination threshold and the control parameter of the movable range of pan and tilt are maintained. As described above, the number of photographed images is monitored at fixed intervals and the imaging frequency is controlled as needed toward the target number S in the imaging time T which is a predetermined period.

  For example, a CPU that controls the camera 101 detects a face of a subject based on image information, recognizes an expression of the face, and detects an expression that is a specific expression (for example, a state of joy, sadness, The apparatus further includes a determination unit that determines whether the feature value exceeds a threshold), and a control unit that performs a subject recording operation (automatic imaging) according to the determination result of the determination unit. In this case, the determination threshold for automatic imaging is adjusted in accordance with the imaging frequency. By this adjustment, even if the facial expression of the subject determined by the determination unit is the same, the imaging operation is performed when the imaging frequency is the first frequency, and the imaging frequency is the second frequency. Is controlled so as not to carry out the photographing operation. As a result, it is possible to obtain a desired number of photographed images and reduce the shortage of the recording memory.

  In addition, for example, the CPU that controls the camera 101 recognizes the face of the subject based on the image information and the direction of the face, and the face is directed in a specific direction, particularly, the front. The apparatus further includes a determination unit that determines the presence or absence, and a control unit that performs automatic photographing according to the determination result of the determination unit. In this case, the determination threshold for automatic imaging (the threshold for determining whether or not to face the front) is adjusted according to the imaging frequency. By this adjustment, even if the direction of the face of the subject determined by the determination unit is the same, the imaging operation is performed when the imaging frequency is the first frequency, and the imaging frequency is the second frequency. Is controlled so as not to carry out the photographing operation. As a result, it is possible to obtain a desired number of photographed images and reduce the shortage of the recording memory.

  In addition, the same applies to the case where the state of the subject's eyes is recognized, and the automatic photographing is performed when the eyes are in a predetermined state, particularly when the eyes are firmly opened and the eyes are facing the camera. The same applies to the case where the posture of the subject is recognized and automatic shooting is performed when the subject is in a predetermined posture. The same applies to the case where the motion of the subject is recognized and the automatic photographing is performed when the subject performs a predetermined motion.

  As described above, when the condition of the subject is recognized and it is determined whether the condition of the subject is in the specific condition and the automatic photographing is performed according to the determination result, the photographing frequency is the first frequency. The control unit performs control so as not to perform the photographing operation when the photographing frequency is the second frequency. As a result, in the photographing time T, a desired number of photographed images can be obtained, and the shortage of the recording memory in the middle of the photographing time can be reduced.

  As another control example, an example shown in FIG. 10B (b) will be described. As in the control example of FIG. 10B (a), first, the user inputs an imaging time T according to an instruction from the user, and the target number S is determined. Monitor at fixed time intervals and hold the result. The content held here holds an evaluation value, which is used for determination of automatic shooting within the monitoring period, indicating whether the image is worth photographing for the user. In FIG. 10B (b), the result is that the image (only one image) of the evaluation value equal to or larger than the threshold Th is automatically photographed. Therefore, it is determined that the number of captured images is insufficient at time T1, and control to lower the evaluation threshold is performed. On the other hand, as a result of lowering the evaluation value threshold at time T2, five images are automatically taken and it is judged that the number of images is excessive with respect to the target number, and the evaluation threshold is raised to lower the imaging frequency. It has become. As described above, the evaluation value used in the determination of the past automatic shooting is monitored, and the determination threshold is updated as needed so as to obtain an appropriate number of shootings. As a result, excessive photographing processing in a short time is suppressed. In addition, while obtaining the desired number of shots in the shooting time T, it is possible to reduce the shortage of the recording memory in the middle of the shooting time.

  Further, as another control example, a configuration will be described in which captured image data is stored only when the evaluation value of the image exceeds a threshold. During automatic photographing, the number of photographed images is monitored at fixed intervals (for example, time T1, T2, T3, T4), and the threshold of the evaluation value of the image is changed toward the target number. For example, when the number of captured images is insufficient at time T1, the image evaluation value threshold is lowered to facilitate storage. Also, for example, when it is determined that the number of captured images is the one at time T2, the threshold value of the evaluation value of the image is increased to make it difficult to save. As a result, in the photographing time T, a desired number of photographed images can be obtained, and the shortage of the recording memory in the middle of the photographing time can be reduced.

  By controlling (changing) the imaging frequency in accordance with the imaging situation as described above, it is possible to perform automatic imaging in which an appropriate number of imaging images can be obtained. As a result, in the imaging device that performs automatic shooting, it is possible to minimize missed shooting of a video that the user wants to shoot.

  In the above description, control is performed to change the shooting frequency according to the shooting conditions, but in addition, control is performed to change the shooting frequency taking into consideration the performance of communication between the camera 101 and the external device. May be

  Next, a second determination, that is, determination based on a neural network will be described. An example of a multi-layer perceptron network is shown in FIG. 11 as an example of a neural network. A neural network is used to predict an output value from an input value, and learns in advance an input value and an output value that is an exemplar for the input to obtain a new input value. On the other hand, it is possible to estimate an output value that follows the learned model. The method of learning will be described later. In FIG. 11, 1201 and its vertical circles indicate neurons in the input layer, 1203 and vertical circles indicate neurons in the middle layer, and 1204 indicate neurons in the output layer. Arrows as shown by 1202 indicate connections connecting each neuron. In the judgment based on the neural network, the input layer neuron is given as input the subject appearing in the current angle of view and the feature value based on the state of the scene or the camera, and the operation based on the forward propagation rule of the multilayer perceptron Get the output value from the output layer. Then, if the value of the output is equal to or greater than the threshold value, it is determined to perform automatic photographing. The characteristics of the subject include the current zoom magnification, the result of general object recognition at the current angle of view, the result of face detection, the number of faces in the current angle of view, the degree of smile on the face, the degree of eye closing, the face angle, and face recognition ID number, gaze angle of the subject person, scene discrimination result, elapsed time since last shooting, current time, GPS position information and change amount from last shooting position, current voice level, voice emitting person, applause, Use of vibration information (acceleration information, camera status), environment information (temperature, air pressure, illuminance, humidity, amount of ultraviolet rays), etc. whether or not cheers are rising. Furthermore, when there is information notification from the external device 501, notification information (user's exercise information, arm action information, biological information such as heart beat, etc.) is also used as a feature. This feature is converted into a numerical value of a predetermined range, and given to each neuron of the input layer as a feature amount. Therefore, each neuron of the input layer is required as many as the number of feature quantities to be used.

  In this judgment based on this neural network, the output value can be changed by changing the connection weight between each neuron in learning processing described later, and the result of the judgment can be adapted to the learning result.

  Further, the determination of automatic photographing also changes according to the start condition of the first control unit 223 read in step S702 in FIG. 7. For example, in the case of activation by tap detection or activation by a specific voice command, there is a very high possibility that the operation is for the user to want to take a picture at present. Therefore, the shooting frequency is set to be high.

<Judgment of shooting method>
In the determination of the shooting method, it is determined which of the still image shooting, the moving image shooting, the continuous shooting, and the panoramic shooting to execute based on the state of the camera and the surrounding objects detected in steps S901 to S904. Do. For example, when the subject (person) is stationary, still image shooting is performed, and when the subject is moving, moving image shooting or continuous shooting is performed. In addition, when a plurality of subjects exist so as to surround the camera, or when it is determined that the subject is a scenic site based on the above-described GPS information, images captured sequentially while operating the pan and tilt May be combined to generate a panoramic image. Similar to the determination method of <determination of whether or not to perform automatic imaging>, various information detected before imaging may be determined based on a neural network to determine the imaging method. Further, in this determination process, the determination conditions can also be changed by a learning process described later.

  Returning to the explanation of FIG. 9, in step S909, if it is determined that automatic imaging is determined by the automatic imaging determination in step S908, the process proceeds to step S910, and if it is not determined that automatic imaging is performed, automatic imaging mode processing is performed. finish.

  In step S910, automatic imaging is started. At this time, shooting according to the shooting method determined in step S 908 is started. At this time, auto focus control is performed by the focus drive control unit 204. Also, exposure control is performed such that the subject has appropriate brightness by using an aperture control unit, a sensor gain control unit, and a shutter control unit (not shown). Furthermore, after photographing, the image processing unit 207 performs various known image processing such as auto white balance processing, noise reduction processing, gamma correction processing, and the like to generate an image.

  In addition, when a predetermined condition is satisfied at the time of photographing, the camera may notify that the person to be photographed is to be photographed before photographing. As a method of notification, for example, sound generation from the audio output unit 218 or LED lighting by the LED control unit 224 may be used. The predetermined conditions include, for example, the number of faces within the angle of view, the degree of smile of the face, the degree of closing of the eyes, the gaze angle and face angle of the subject person, the face authentication ID number, the number of persons registered for personal authentication, As a result of general object recognition, scene discrimination result, elapsed time since last shooting, shooting time, whether the current position based on GPS information is a scenic site, voice level at the time of shooting, presence or absence of person making voice , Applause, whether cheers are rising, vibration information (acceleration information, camera state), environment information (temperature, pressure, illuminance, humidity, amount of ultraviolet light) and the like. By performing informing shooting on the basis of these conditions, it is possible to leave an image of a preferred camera view in a scene of high importance.

  With regard to such notification before shooting, it is also possible to determine the method and timing of the notification by judging information of the shot image or various information detected before shooting based on the neural network. Further, in this determination process, the determination conditions can also be changed by a learning process described later.

  In step S 911, editing processing such as processing the image generated in step S 910 or adding it to a moving image is performed. Specifically, image processing includes trimming processing based on a person's face and in-focus position, image rotation processing, HDR (high dynamic range) effect processing, blur effect processing, color conversion filter effect processing, and the like. In image processing, a plurality of processed images may be generated based on the image generated in step S910 by a combination of the above-described processing, and may be stored separately from the image generated in step S910. In addition, in the moving image processing, processing may be performed such that the captured moving image or still image is added to the generated edited moving image while performing special effect processing of slide, zoom, and fade. Also in the editing in step S911, it is possible to determine the method of image processing by judging the information of the photographed image or various information detected before photographing based on the neural network. Further, in this determination process, the determination conditions can also be changed by a learning process described later.

  In step S912, learning information generation processing of the photographed image is performed. Here, the information used for the learning process described later is generated and recorded. Specifically, the zoom magnification at the time of shooting, the general object recognition result at the time of shooting, the face detection result, the number of faces included in the shot image, the degree of smile of the face, the degree of eye closing, the face angle, the face Authentication ID number, gaze angle of subject person, scene discrimination result, elapsed time since last shooting, shooting time, GPS position information and change amount from last shooting position, voice level at shooting, person speaking voice, Applause, whether cheers are rising or not, vibration information (acceleration information, camera status), environment information (temperature, air pressure, illuminance, humidity, amount of ultraviolet light), movie shooting time, manual shooting instruction, etc. is there. Furthermore, a score, which is the output of a neural network that quantifies the preferences of the user's image, is also calculated. These pieces of information are generated and recorded as tag information in the photographed image file. Alternatively, the information may be written to the non-volatile memory 216 or may be stored in the recording medium 221 as so-called catalog data in the form of a list of information of each photographed image.

  In step S913, the past shooting information is updated. Specifically, the number of shots for each area described in step S908, the number of shots for each person registered for personal identification, the number of shots for each object recognized by general object recognition, and the number of shots for each scene of scene determination The count of the number of images corresponding to the image taken this time is incremented by one.

<Learning process>
Next, learning in accordance with the preference of the user in the present embodiment will be described. In the present embodiment, using a neural network as shown in FIG. 11 and a machine learning algorithm, the learning processing unit 219 performs learning in accordance with the user's preference. The neural network is used to predict the output value from the input value, and learns in advance the actual value of the input value and the actual value of the output value to output an output for the new input value. The value can be estimated. By using the neural network, learning is performed according to the user's preference for the above-mentioned automatic photographing, automatic editing, and subject search. In addition, registration of subject information (results of face recognition, general object recognition, etc.) as feature data to be input to the neural network, and operation of changing photographing notification control, low power consumption mode control, and automatic file deletion by learning are also performed.

In the present embodiment, the operation to which the learning process is applied is the following operation.
(1) Automatic shooting (2) Automatic editing (3) Subject search (4) Subject registration (5) Shooting notification control (6) Low power consumption mode control (7) File automatic deletion (8) Image blur correction (9) image Automatic Transfer Of the operations to which the above-described learning process is applied, automatic editing, file automatic deletion, and automatic image transfer are not directly related to the subject matter of the present invention, and thus the description thereof is omitted.

<Auto shoot>
Describe learning for automatic shooting. In automatic shooting, learning is performed to automatically take an image that suits the user's preference. As described with reference to the flowchart of FIG. 9, the learning information generation process (step S912) is performed after shooting (after step S910). An image to be learned is selected by a method to be described later, and learning is performed by changing the weight of the neural network based on the learning information included in the image.

  The learning is performed by changing the neural network that determines the automatic shooting timing and changing the neural network that determines the shooting method (still image shooting, moving image shooting, continuous shooting, panoramic shooting, etc.).

<Object Search>
The learning for the subject search will be described. In the subject search, learning is performed to automatically search for a subject matching the user's preference. As described using the flowchart of FIG. 9, in the subject search process (step S904), the importance level of each area is calculated, pan / tilt and zoom are driven, and the subject search is performed. The learning is performed based on the photographed image and the detection information during the search, and is reflected as a learning result by changing the weight of the neural network. Various detection information during the search operation is input to the neural network, and the importance level is determined to perform the object search reflecting the learning. In addition to the calculation of the importance level, for example, control of a pan / tilt search method (speed, frequency of movement) is also performed.

<Subject registration>
The learning for subject registration will be described. In subject registration, learning is performed to automatically register and rank subjects according to the user's preference. As learning, for example, registration of face recognition registration and registration of general object recognition, registration of gesture and voice recognition, and scene recognition by sound are performed. Authentication registration is performed for a person and an object, and setting of ranking is performed based on the number and frequency of image acquisition, the number and frequency of manual photographing, and the frequency of appearance of a subject under search. The registered information is registered as an input for determination using each neural network.

<Recording notification control>
The learning for the imaging notification will be described. As described in step S 910 in FIG. 9, when the predetermined condition is satisfied immediately before shooting, shooting is performed after notifying that the person to be shot is to be shot by the camera. For example, the line of sight of the subject is visually guided by driving the pan / tilt, the speaker sound emitted from the audio output unit 218, or the LED lighting by the LED control unit 224 is used to guide the attention of the subject Do. Immediately after the above notification, it is determined whether to use the detection information for learning based on whether or not the detection information of the subject (for example, smile level, eye line detection, gesture) is obtained, and the weight of the neural network is changed. Learn by letting them do it.

  Each detection information immediately before shooting is input to the neural network, and it is judged whether to notify or not, each operation (sound (sound level / sound type / timing), light (lighting time, speed), camera direction ( Perform pan / tilt motion) determination.

<Low power consumption mode control>
As described above with reference to FIGS. 7 and 8, control is performed to turn on / off the power supply to the Main CPU (the first control unit 223). However, to return to the low power consumption mode or to the low power consumption state Also learn transition conditions of The learning of the condition for canceling the low power consumption mode will be described.

<Sound detection>
For example, it is possible to learn by manually setting a specific voice, a specific sound scene that the user wants to detect, and a specific sound level by communication using a dedicated application of the external device 301, for example. In addition, a plurality of detection methods are set in advance in the voice processing unit, an image to be learned is selected by a method to be described later, learning front and back sound information included in the image is performed, and sound determination (specific sound command You can also learn by setting sound scenes such as "Cheers" and "Applause".

<Environmental information detection>
For example, the environment information change that the user wants to set as the start condition can be learned by setting manually by communication using a dedicated application of the external device 301, for example. For example, activation can be performed according to specific conditions such as temperature, air pressure, brightness, humidity, absolute amount of ultraviolet light amount, and change amount. It is also possible to learn a determination threshold based on each piece of environmental information. If it is determined from the camera detection information after activation by the environmental information that the activation factor is not determined, the parameter of each determination threshold is set so as to make it difficult to detect environmental change.

  Each of the above-mentioned parameters also changes depending on the remaining capacity of the battery. For example, when the battery remaining amount is low, it becomes difficult to enter various determinations, and when the battery remaining amount is large, it becomes easy to enter various determinations. Specifically, even when the battery remaining amount is large even in the shaking state detection result or the sound scene detection result that is not a factor that the user necessarily wants to start the camera, it may be determined to start the camera.

  In addition, the determination of the low power consumption mode cancellation condition can also be performed based on a neural network from vibration detection, sound detection, information on time lapse detection, environment information, battery remaining amount, and the like. In this case, learning is performed by selecting an image to be learned by a method to be described later, and changing weights of the neural network based on learning information included in the image.

  Next, learning of the transition condition to the low power consumption state will be described. As shown in FIG. 7, when it is determined that the mode setting determination in step S704 is neither “automatic shooting mode” “automatic editing mode” “image automatic transfer mode” “learning mode” “automatic file deletion mode” , Enter low power mode. The determination conditions of each mode are as described above, but the conditions under which each mode is determined also change by learning.

<Automatic shooting mode>
As described above, the importance level for each area is determined, and automatic shooting is performed while searching for a subject with pan and tilt. However, when it is determined that a subject to be shot does not exist, the automatic shooting mode is canceled. For example, when the importance level of all the areas or the value obtained by adding the importance levels of the respective areas becomes equal to or less than a predetermined threshold, the automatic photographing mode is canceled. At this time, the predetermined threshold may be lowered according to the elapsed time after the transition to the automatic photographing mode. The transition to the low power consumption mode is facilitated as the elapsed time from the transition to the automatic shooting mode becomes longer.

  In addition, by changing the predetermined threshold according to the remaining capacity of the battery, it is possible to perform the low power consumption mode control in consideration of the battery durability. For example, when the battery remaining amount is low, the threshold is increased to facilitate transition to the low power consumption mode, and when the battery remaining amount is large, the threshold is reduced to transition to the low power consumption mode. Here, the parameter (elapsed time threshold TimeC) of the next low power consumption mode cancellation condition is set for the second control unit 211 (SubCPU) based on the elapsed time since the transition to the automatic shooting mode last time and the number of shots. Do. Each threshold mentioned above changes by learning. The learning is performed by, for example, manually setting the imaging frequency, the activation frequency, and the like by communication using a dedicated application of the external apparatus 301.

  In addition, an average value of elapsed time from when the power button of the camera 101 is turned on to when the power button is turned off or distribution data for each time zone may be accumulated to learn each parameter. In this case, for users with short time from power ON to OFF, the time interval for returning from the low power consumption mode and transition to the low power consumption state becomes short, and the time from power ON to OFF is long Is learned so that the interval is long.

  In addition, learning is also performed by the detection information under search. While it is determined that the number of important subjects set by learning is large, the time interval for returning from the low power consumption mode and the transition to the low power consumption state is short, and the number of important subjects is small. , Will be learned to increase the interval.

Image blur correction
The learning for the image blur correction will be described. Image blur correction is performed by calculating a correction amount in step S902 in FIG. 9 and driving pan and tilt in step S905 based on the correction amount. In image blur correction, learning is performed to perform correction in accordance with the characteristics of the user's shaking. By using, for example, a PSF (Point Spread Function) on a captured image, it is possible to estimate the direction and size of blur. In the generation of the learning information in step S912 in FIG. 9, the estimated direction and size of blur are added to the image as information.

  In the learning mode processing in step S 716 of FIG. 7, the detected direction and magnitude of the estimated blur are output, and each detected information at the time of shooting (motion vector information of the image at a predetermined time before shooting, detected subject (person or object Learning the weights of the neural network for image blur correction using the motion information and vibration information (gyro output, acceleration output, camera status) as input, as well as environmental information (temperature, pressure, illuminance, humidity), Sound information (sound scene determination, specific voice detection, sound level change), time information (elapsed time from start, elapsed time from last shooting), location information (GPS position information, position movement change amount) etc. are also input In addition, it may be determined.

  At the time of calculation of the image blur correction amount in step S902, by inputting each of the above detection information to the neural network, it is possible to estimate the size of the blur at the moment of the photographing. Then, when the size of the estimated blur is large, control such as increasing the shutter speed becomes possible. In addition, when the size of the estimated blur is large, a blurred image is generated. Therefore, it is possible to take a method such as prohibiting photographing.

  In addition, since there is a limit to the pan / tilt drive angle, no further correction can be made if it reaches the drive end, but the image during exposure is estimated by estimating the magnitude and direction of blur at the time of shooting. It is possible to estimate the range required for pan / tilt driving to correct blurring. When there is no margin of the movable range during exposure, it is possible to suppress large blurring by setting the cutoff frequency of the filter for calculating the image blur correction amount so as not to exceed the movable range. If the movable range is likely to be exceeded, the movable range is secured by rotating the pan / tilt angle in the direction opposite to the direction likely to exceed the movable range immediately before exposure to ensure the movable range. You can also take pictures without As a result, image blur correction can be learned in accordance with the characteristics and usage of the user at the time of photographing, so that it is possible to prevent the photographed image from being blurred.

  In the above-described <Determination of photographing method>, it may be determined whether or not the moving subject has no blur, and the non-moving background is photographed and the follow shot photographing is performed. In that case, subject shake correction may be performed by estimating a pan / tilt driving speed for shooting a subject without blurring from detection information before shooting. At this time, the driving speed can be estimated by inputting the above detection information to the neural network which has already been learned. The learning is performed based on the information by estimating the direction and size of blurring in the block where the main subject is located by dividing the image into blocks and estimating the PSF of each block.

  In addition, it is possible to learn the background flow amount from the information of the image selected by the user. In that case, it is possible to estimate the magnitude of blur in a block where the main subject is not located, and to learn the user's preference based on the information. By setting the shutter speed at the time of shooting based on the learned background flow amount of preference, it is possible to automatically perform shooting that can obtain a shooting effect that meets the user's preference.

  Next, the learning method will be described. As learning methods, there are “learning in the camera” and “learning in cooperation with the communication device”.

The in-camera learning method will be described below. There are the following methods for in-camera learning in the present embodiment.
(1) Learning by detection information at manual shooting (2) Learning by detection information at object search <Learning by detection information at manual shooting>
As described in step S 907 to step S 913 in FIG. 9, in the present embodiment, the camera 101 can perform two shootings, manual shooting and automatic shooting. If a manual imaging instruction has been issued in step S 907, information indicating that the captured image is an image captured manually is added in step S 912. If it is determined in step S909 that automatic imaging is ON and imaging is performed, information is added in step S912 that the captured image is an image captured automatically.

  Here, in the case of manual shooting, the possibility of shooting based on the user's favorite subject, favorite scene, favorite place and time interval is very high. Therefore, learning is performed based on each feature data obtained at the time of manual shooting and learning information of the shot image. Further, from detection information at the time of manual imaging, learning is performed regarding extraction of feature amounts in the captured image, registration of personal authentication, registration of individual facial expressions, and registration of human combinations. In addition, from the detection information at the time of subject search, for example, learning is performed to change the importance of a nearby person or object from the expression of the subject registered individually.

<Learning by detection information at the time of subject search>
During the subject search operation, it is determined what kind of person, an object, and a scene are captured at the same time as a subject registered for personal identification registration, and a time ratio at which the subject is captured at the same time is calculated. For example, the time ratio in which the person A of the personal identification registration subject is shown simultaneously with the person B of the personal identification registration subject is calculated. Then, when the person A and the person B fall within the angle of view, various types of detection information are stored as learning data so as to increase the score of the automatic photographing determination, and learning is performed in the learning mode process (step S716).

  In another example, a time ratio is calculated in which the person A of the personal identification registration subject is shown simultaneously with the subject "cat" determined by the general object recognition. Then, when the person A and the "cat" fall within the angle of view, various types of detection information are stored as learning data so as to increase the score of the automatic photographing determination, and learning is performed in the learning mode process (step S716).

  Also, when a high degree of smile of the person A of the personal identification registration subject is detected, or when an expression such as "joy" or "surprise" is detected, the subject shown at the same time is learned to be important. Alternatively, when an expression such as "anger" or "true face" is detected, processing such as not learning is performed because the subject appearing at the same time is unlikely to be important.

Next, learning in cooperation with an external device in the present embodiment will be described. There are the following methods for learning in cooperation with an external device in the present embodiment.
(1) Learning by obtaining an image by an external device (2) Learning by inputting a determination value to an image through an external device (3) Learning by analyzing an image stored in the external device ( 4) Learning from the information uploaded to the SNS server in the external device (5) Learning by changing the camera parameters in the external device (6) Learning from the information in which the image was manually edited in the external device <With the external device Learning by having acquired an image>
As described in FIG. 3, the camera 101 and the external apparatus 301 have communication means for performing the first and second communications 302 and 303. Then, transmission and reception of an image is mainly performed by the first communication 302, and the image in the camera 101 can be transmitted to the external device 301 via a dedicated application in the external device 301. Also, thumbnail images of image data stored in the camera 101 can be browsed using a dedicated application in the external device 301. The user can select an image that he / she likes from the thumbnail images, check the image, and transmit the image to the external device 301 by operating the image acquisition instruction.

  At this time, since the user selects and acquires an image, the acquired image is very likely to be an image preferred by the user. Therefore, it is possible to perform various types of learning of the user's preference by determining that the acquired image is an image to be learned and performing learning based on the learning information of the acquired image.

  Here, an operation example will be described. An example in which an image in the camera 101 is browsed using an application dedicated to the external device 301 is shown in FIG. The display unit 407 displays thumbnail images (1604 to 1609) of the image data stored in the camera, and the user can select and acquire an image that he / she likes. At this time, buttons 1601, 1602, and 1603 constituting a display method change unit for changing the display method are provided.

  When the button 1601 is pressed, the display mode is changed to the date-and-time priority display mode, and the images are displayed on the display unit 407 in the order of the shooting date and time of the image in the camera 101. For example, an image with a new date and time is displayed at a position indicated by 1604, and an image with an old date and time is displayed at a position indicated by 1609.

  When the button 1602 is pressed, the recommended image priority display mode is changed. The images in the camera 101 are displayed on the display unit 407 in descending order of score, based on the scores obtained by determining the preference of the user for each image calculated in step S912 in FIG. For example, an image with a high score is displayed at a position indicated by 1604, and an image with a low score is displayed at a position indicated by 1609.

  When the button 1603 is pressed, a person or an object / subject can be designated, and when a specific person or an object / subject is subsequently designated, only a specific subject can be displayed. The buttons 1601 to 1603 can simultaneously turn on the setting. For example, when all the settings are turned on, only a designated subject is displayed, an image with a new shooting date and time is prioritized, and an image with a high score is prioritized and displayed. As described above, since the user's preference is learned also for the photographed image, it is possible to extract only the user's favorite image from a large number of photographed images by a simple confirmation operation.

<Learning by inputting a judgment value to an image through an external device>
As described above, the camera 101 and the external device 301 have communication means, and can browse the image stored in the camera 101 using a dedicated application in the external device 301. Here, the user may be configured to score each image. A high score (for example, 5 points) can be given to an image that the user thinks like, or a low score (for example, 1 point) can be given to images that the user thinks is not like. Make it a structure to learn. The score of each image is used for relearning along with the learning information in the camera. Learning is performed so that the output of the neural network, which has feature data from specified image information as an input, approaches a point specified by the user.

  In the present embodiment, the user inputs the determination value to the photographed image via the external apparatus 301. However, the determination value may be directly input to the image by operating the camera 101. In that case, for example, the camera 101 is provided with a touch panel display, and the user presses a GUI button displayed on the screen display unit of the touch panel display to set a mode for displaying a photographed image. Then, the user can perform similar learning by a method of inputting a determination value to each image while confirming the photographed image.

<Learning by analyzing stored image in external device>
The external apparatus 301 includes a storage unit 404, and the storage unit 404 is configured to record an image other than the image captured by the camera 101. At this time, it is easy for the user to view the image stored in the external device 301 and to easily upload the image to the shared server via the public wireless control unit 406. Very likely to be included.

  The control unit 411 of the external device 301 is configured to be able to process an image stored in the storage unit 404 with the same capability as the learning processing unit 219 in the camera 101 using a dedicated application. Then, learning is performed by communicating the processed data for learning to the camera 101. Alternatively, an image or data to be learned by the camera 101 may be transmitted, and learning may be performed in the camera 101. Alternatively, the user may select an image to be learned from among the images stored in the recording unit 404 using a dedicated application, and may learn.

<Learning from the information uploaded to the SNS server in the external device>
Next, a method of using information in a social networking service (SNS), which is a service or website that can build a social network focusing on the connection between people, is used for learning. When uploading an image to the SNS, there is a technique of inputting a tag relating to the image from the external device 301 and transmitting it along with the image. In addition, there is also a technology for inputting liking or dislike to images uploaded by other users, and it is possible to determine whether the images uploaded by the other users are photos of the user of the external apparatus 301 who prefers.

  The dedicated SNS application downloaded into the external device 301 can acquire the image and information about the image uploaded by the user as described above. Also, by inputting whether or not the user likes the image uploaded by another user, it is possible to acquire the user's favorite image and tag information. The images and tag information thereof are analyzed, and can be learned in the camera 101.

  The control unit 411 of the external device 301 is configured to obtain an image uploaded by the user as described above and an image determined to be liked by the user, and can be processed with the same capability as the learning processing unit 219 in the camera 101. . Then, learning is performed by communicating the processed data for learning to the camera 101. Alternatively, an image to be learned may be transmitted to the camera 101 and learning may be performed in the camera 101.

  In addition, subject information which the user may prefer is estimated from subject information (for example, subject object information such as a dog or a cat, scene information such as a beach, expression information such as a smile, etc.) set as tag information. Then, learning is performed by registering as a subject to be detected to be input to the neural network.

  In addition, image information currently in circulation in the world can be estimated from statistical values of tag information (image filter information and subject information) in the SNS, and can be learned in the camera 101.

<Learning by changing camera parameters in external device>
As described above, the camera 101 and the external device 301 have communication means. Then, the learning parameter (weight of neural network, selection of subject to be input to neural network, etc.) currently set in the camera 101 is communicated to the external device 301 and stored in the storage unit 404 of the external device 301. it can. In addition, using a dedicated application in the external device 301, learning parameters set in a dedicated server can be acquired via the public wireless control unit 406 and set as learning parameters in the camera 101. As a result, by storing the parameters at a certain point in the external device 301 and setting them in the camera 101, it is possible to return the learning parameters. In addition, learning parameters possessed by other users can be acquired via a dedicated server and set in their own cameras 101.

  Further, voice commands, authentication registrations, gestures registered by the user may be registered using an application dedicated to the external device 301, or important places may be registered. These pieces of information are treated as the imaging trigger described in the automatic imaging mode process (FIG. 9) and input data for automatic imaging determination. In addition, it is possible to set the shooting frequency, the start interval, the ratio of still images and moving pictures, a desired image, and the like, and the start interval described in <Low Power Consumption Mode Control> may be set.

<Learning from the information on which the image was manually edited by an external device>
A dedicated application of the external device 301 can be provided with a function that can be manually edited by a user operation, and the contents of editing work can be fed back to learning. For example, editing of image effect application (trimming process, rotation process, slide, zoom, fade, color conversion filter effect, time, still image / moving image ratio, BGM) is possible. Then, a neural network for automatic editing is trained so that the manually added image effect is determined for the learning information of the image.

  Next, the learning processing sequence will be described. In the mode setting determination in step S704 in FIG. 7, it is determined whether or not learning processing should be performed. If it is determined that the learning processing should be performed, the learning mode processing in step S716 is performed.

  The determination conditions of the learning mode will be described. Whether or not to shift to the learning mode is determined based on the elapsed time since the previous learning process, the number of information that can be used for learning, and whether there is a learning process instruction via the communication device. FIG. 13 shows a flow of processing for determining whether or not to shift to the learning mode, which is determined in the mode setting determination processing of step S704.

  When the start of the learning mode determination is instructed in the mode setting determination process of step S704, the process of FIG. 13 starts. In step S1401, it is determined whether there is a registration instruction from the external apparatus 301. The registration here may be <learning by obtaining an image by an external device>, <learning by inputting a determination value to an image through an external device>, or <storage in an external device. It is determined whether or not there is a registration instruction for learning, such as learning by analyzing an image.

  If there is a registration instruction from the external apparatus 301 in step S1401, the process advances to step S1408 to set the learning mode determination to TRUE, perform processing of step S716, and end the learning mode determination process. If there is no registration instruction from the external device in step S1401, the process advances to step S1402.

  In step S1402, it is determined whether there is a learning instruction from an external device. The learning instruction here is a determination as to whether or not an instruction to set the learning parameter has been given, as in <Learning by changing camera parameters in external device>. If it is determined in step S1402 that there is a learning instruction from the external apparatus, the process advances to step S1408 to set the learning mode determination to TRUE, perform processing of step S716, and end the learning mode determination process. If it is determined in step S1402 that there is no learning instruction from the external apparatus, the process advances to step S1403.

  In step S1403, an elapsed time Time N after the previous learning process (recalculation of weights of the neural network) is performed, and the process proceeds to step S1404. In step S 1404, the number of new data to be learned DN (the number of images designated to be learned during the elapsed time Time N since the previous learning process was performed) is acquired, and the process proceeds to step S 1405. In step S1405, a threshold value DT for determining whether to enter the learning mode is calculated from the elapsed time TimeN. The smaller the threshold value DT, the easier it is to enter the learning mode. For example, DTa, which is the value of the threshold DT when TimeN is smaller than a predetermined value, is set larger than DTb, which is the value of the threshold DT when TimeN is larger than the predetermined value. Is set to be small. As a result, even when the amount of learning data is small, if the time lapse is large, it is easy to enter the learning mode and learning is performed again so that the camera can easily change learning according to the time of use.

  After calculating the threshold value DT in step S1405, the process proceeds to step S1406, and it is determined whether the number of data to be learned DN is larger than the threshold value DT. If the number of data DN is larger than the threshold value DT, the process advances to step S1407 to set DN to 0. Thereafter, the process proceeds to step S1408, the learning mode determination is set to TRUE, and the process of step S716 (FIG. 7) is set to be performed, and the learning mode determination process is ended.

  If it is determined in step S1406 that the DN is less than or equal to the threshold value DT, the process advances to step S1409. Since there is neither a registration instruction from an external device nor a learning instruction from an external device, and the number of learning data is less than a predetermined value, the learning mode determination is set to FALSE and the process of step S716 is set not to be performed. The determination process ends.

  Next, the process in the learning mode process (step S716) will be described. A detailed flowchart showing the operation of the learning mode process is shown in FIG.

  When it is determined in step S715 in FIG. 7 that the learning mode is set, and the process proceeds to step S716, the processing in FIG. 14 starts. In step S1501, it is determined whether there is a registration instruction from the external apparatus 301. If it is determined in step S1501 that a registration instruction is issued from the external apparatus 301, the process advances to step S1502. In step S1502, various registration processing is performed.

  The various registrations are registrations of features to be input to the neural network, such as registration of face recognition, registration of general object recognition, registration of sound information, registration of location information, and the like. When the registration process is completed, the process proceeds to step S1503, and the element to be input to the neural network is changed from the information registered in step S1502. When the process of step S1503 ends, the process proceeds to step S1507.

  If there is no registration instruction from the external device 301 in step S1501, the process advances to step S1504, and it is determined whether there is a learning instruction from the external device 301. If there is a learning instruction from the external device 301, the process proceeds to step S1505, the learning parameter communicated from the external device 301 is set in each determiner (weight of neural network, etc.), and the process proceeds to step S1507.

  If there is no learning instruction from the external device 301 in step S1504, learning (recalculation of weights of neural networks) is performed in step S1506. The process of step S1506 is performed as in the case where the number of data to be learned DN exceeds the threshold value DT and the respective determinators are relearned, as described with reference to FIG. Retraining is performed using a method such as error back propagation method or gradient descent method, and weights of the neural network are recalculated to change parameters of each decision unit. When the learning parameter is set, the process proceeds to step S1507.

  In step S1507, the images in the file are re-scored. In the present embodiment, all photographed images stored in the file (recording medium 221) are scored based on learning results, and automatic editing and automatic file deletion are performed according to the scored scores. It is a structure. Therefore, when re-learning or setting of learning parameters from an external device is performed, it is necessary to update the score of the photographed image. Therefore, in step S1507, recalculation is performed to add a new score to the captured image stored in the file, and when the process ends, the learning mode process ends.

  In the present embodiment, a method has been described in which a user's favorite video is proposed by extracting a scene that the user seems to like, learning its features, and reflecting it on camera operations such as automatic shooting and automatic editing. The invention is not limited to this application. For example, it can also be used for an application that proposes an image different from the user's own preference. As an example of the realization method, it is as follows.

<Method using neural network with learning preference>
For learning, the user's preference is learned by the method described above. Then, in S908 of <automatic imaging>, automatic imaging is performed when the output value of the neural network is a value indicating that it is different from the user's preference which is teacher data. For example, if an image preferred by the user is used as a teacher image, and learning is performed so that a high value is output when showing a feature similar to the teacher image, conversely, the condition is that the output value is lower than a predetermined value. Perform automatic shooting. Similarly, also in the subject searching process and the automatic editing process, a process is performed in which the output value of the neural network becomes a value indicating that it is different from the user's preference as teacher data.

<How to use a neural network trained on a situation different from preference>
In this method, at the time of learning processing, a situation different from the user's preference is learned as teacher data. For example, in the above description, it is assumed that the image photographed manually is a scene photographed preferably by the user, and the learning method in which this is used as teacher data has been described. However, here, the image taken manually is not used as teacher data, and a scene for which manual shooting has not been performed for a predetermined time or more is added as teacher data. Alternatively, if there is a scene whose feature is similar to the manually captured image in the teacher data, it may be deleted from the teacher data. Further, an image having a feature different from the image acquired by the external device may be added to the teacher data, or an image having a feature similar to the acquired image may be deleted from the teacher data. By doing this, data different from the user's preference is collected in the teacher data, and as a result of learning, the neural network can determine a situation different from the user's preference. Then, in the automatic shooting, a scene different from the user's preference can be shot by shooting according to the output value of the neural network.

  As described above, by intentionally proposing a video different from the user's own preference, it is possible to capture a scene where the user will not manually shoot and reduce missed shots. Also, by proposing shooting in a scene that is not in the user's own idea, it is possible to expect the effect of making the user aware or broadening the range of preference.

  Further, by combining the above methods, it is possible to propose a situation that is somewhat similar to, but partially different from, the user's preference, and it is easy to adjust the degree of conformity to the user's preference. The degree of conformity to the preference of the user may be changed according to the mode setting, the state of various sensors, and the state of detection information.

  In the present embodiment, the configuration for learning in the camera 101 has been described. However, the same learning effect can be realized with a configuration in which the external device 301 has a learning function, data necessary for learning is communicated to the external device 301, and learning is performed only on the external device. In that case, as described above in <Learning by changing the camera parameters in the external device>, learning is performed by setting parameters such as weights of the neural network learned on the external device side to the camera 101 by communication. It may be configured.

  In addition, both the camera 101 and the external device 301 have a learning function, and for example, the camera 101 can learn the learning information possessed by the external device 301 at the timing when the learning mode processing (step S716) is performed in the camera 101. , And may be configured to perform learning by merging learning parameters.

(Other embodiments)
Furthermore, the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read the program. It can also be realized by the process to be executed. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  Examples of embodiments of the present invention are listed below.

(Embodiment 1) Imaging means for imaging a subject image and outputting image data
Control means for controlling whether or not to carry out an imaging operation for recording the image data output by the imaging means;
Acquisition means for acquiring information on the frequency of the photographing operation;
Equipped with
The image pickup apparatus, wherein the control means changes a threshold for determining whether to perform the photographing operation according to the information related to the frequency.

(Embodiment 2)
It further comprises detection means for detecting information of the subject,
2. The image pickup apparatus according to claim 1, wherein the control means determines whether or not to carry out the photographing operation by comparing information of the subject with the threshold value.

(Embodiment 3)
3. The imaging apparatus according to claim 2, wherein the detection unit detects information of a subject based on at least one of a detected sound and image data captured by the imaging unit.

(Embodiment 4)
4. The imaging apparatus according to any one of the embodiments 1 to 3, wherein the initial value of the threshold is determined based on a result of past learning.

(Embodiment 5)
4. The image pickup apparatus according to any one of the embodiments 1 to 4, wherein the information related to the frequency of the photographing operation is the number of photographed images for every predetermined period.

(Embodiment 6)
6. The imaging apparatus according to embodiment 5, wherein the control means determines the threshold for a next fixed period based on the number of times of imaging in the past.

(Embodiment 7)
The apparatus further comprises determination means for determining a target number of shots in a predetermined period,
7. The control method according to any one of the embodiments 1 to 6, wherein the control means changes a threshold value for determining whether or not to perform the imaging operation based on the target number of imaging and the information on the frequency. The imaging device according to any one of the items.

(Embodiment 8)
9. The imaging apparatus according to claim 7, wherein the control unit changes the threshold value so that the number of captured images increases linearly toward the target number of captured images as the imaging time elapses.

(Embodiment 9)
9. The imaging apparatus according to claim 7, wherein the determination unit determines the target number of shot images based on a shooting condition set based on a manual input or a voice input by a user.

(Embodiment 10)
10. The imaging apparatus according to embodiment 9, wherein the user's manual input or voice input is performed using a smart device.

(Embodiment 11)
11. The imaging device according to embodiment 9 or 10, wherein the imaging condition includes information of a total imaging time.

(Embodiment 12)
12. The image pickup apparatus according to claim 11, wherein the photographing condition further includes information of a recording medium and a remaining amount of a battery.

(Embodiment 13)
Imaging means for imaging a subject image and outputting image data;
Control means for controlling whether or not to carry out an imaging operation for recording the image data output by the imaging means;
Detection means for detecting the face of the subject;
A determination unit that determines the state of the face of the subject detected by the detection unit;
Acquisition means for acquiring information on the frequency of the photographing operation;
Equipped with
The control means performs the photographing operation when the frequency is the first frequency even if the condition of the face of the subject determined by the determination means is the same, and the case where the frequency is the second frequency And an imaging device characterized by performing control so as not to carry out a photographing operation.

(Embodiment 14)
The state of the subject's face includes the expression of the face of the subject, the direction of the face of the subject, the degree of opening of the subject's eyes, the line of sight of the subject, the attitude of the subject, and the state of the action of the subject. 13. The imaging device according to 13.

(Fifteenth Embodiment)
It further comprises changing means for changing the direction of the imaging means in order to direct the direction of the imaging means to the subject,
15. The image pickup apparatus according to any one of the embodiments 1 to 14, wherein the change unit changes a movable range in which the orientation of the image pickup unit is changed according to the frequency.

(Sixteenth Embodiment)
The imaging device according to the fifteenth embodiment, wherein the change unit rotates the imaging unit in a pan direction and a tilt direction.

(Seventeenth Embodiment)
It further comprises zoom means for enlarging or reducing the image of the subject on the imaging means,
17. The image pickup apparatus according to any one of the embodiments 1 to 16, wherein the zoom unit changes the control of the enlargement or reduction according to the frequency.

(Embodiment 18)
A method of controlling an image pickup apparatus comprising an image pickup means for picking up an image of a subject and outputting image data, comprising:
A control step of controlling whether or not to perform a photographing operation of recording the image data output by the imaging unit;
Obtaining an information related to the frequency of the photographing operation;
In the control step, a threshold value for determining whether to perform the photographing operation is changed according to the information related to the frequency.

(Embodiment 19)
A method of controlling an image pickup apparatus comprising an image pickup means for picking up an image of a subject and outputting image data, comprising:
Detecting the face of the subject;
A determination step of determining the state of the face of the subject detected by the detection step;
A control step of controlling whether or not to perform a photographing operation of recording the image data output by the imaging unit;
Obtaining an information related to the frequency of the photographing operation;
In the control step, even if the condition of the face of the subject determined in the determination step is the same, the photographing operation is performed if the frequency is the first frequency, and the frequency is the second frequency A control method of an image pickup apparatus, wherein control is performed so as not to carry out a photographing operation.

(Embodiment 20)
The program for making a computer perform each process of the control method of embodiment 18 or 19.

(Embodiment 21)
20. A computer-readable storage medium storing a program for causing a computer to execute the steps of the control method according to embodiment 18 or 19.

101: camera, 301: smart device, 501: wearable device, 104: tilt rotation unit, 105: pan rotation unit

Claims (21)

Imaging means for imaging a subject image and outputting image data;
Control means for controlling whether or not to carry out an imaging operation for recording the image data output by the imaging means;
Acquisition means for acquiring information on the frequency of the photographing operation;
Equipped with
The image pickup apparatus, wherein the control means changes a threshold for determining whether to perform the photographing operation according to the information related to the frequency. It further comprises detection means for detecting information of the subject,
The imaging apparatus according to claim 1, wherein the control unit determines whether to perform the imaging operation by comparing information of the subject with the threshold.   3. The image pickup apparatus according to claim 2, wherein the detection unit detects information of a subject based on at least one of the detected sound and the image data captured by the imaging unit.   The imaging apparatus according to any one of claims 1 to 3, wherein the initial value of the threshold is determined based on a result of past learning.   The image pickup apparatus according to any one of claims 1 to 4, wherein the information on the frequency of the photographing operation is the number of photographed images at predetermined intervals.   The imaging apparatus according to claim 5, wherein the control unit determines the threshold for a next fixed period based on the number of captured images in the past. The apparatus further comprises determination means for determining a target number of shots in a predetermined period,
The controller according to any one of claims 1 to 6, wherein the control means changes a threshold value for determining whether or not to perform the imaging operation based on the target number of imaging and the information on the frequency. An imaging device according to claim 1.   8. The image pickup apparatus according to claim 7, wherein the control means changes the threshold so that the number of photographed images increases linearly toward the target number of photographed images as the photographing time passes.   9. The image pickup apparatus according to claim 7, wherein the determination unit determines the target number of shot images based on a shooting condition set based on a manual input or a voice input by a user.   The imaging apparatus according to claim 9, wherein the user's manual input or voice input is performed using a smart device.   The imaging apparatus according to claim 9, wherein the imaging condition includes information of a total imaging time.   The image pickup apparatus according to claim 11, wherein the photographing condition further includes information of a recording medium and a remaining amount of a battery. Imaging means for imaging a subject image and outputting image data;
Control means for controlling whether or not to carry out an imaging operation for recording the image data output by the imaging means;
Detection means for detecting the face of the subject;
A determination unit that determines the state of the face of the subject detected by the detection unit;
Acquisition means for acquiring information on the frequency of the photographing operation;
Equipped with
The control means performs the photographing operation when the frequency is the first frequency even if the condition of the face of the subject determined by the determination means is the same, and the case where the frequency is the second frequency And an imaging device characterized by performing control so as not to carry out a photographing operation.   The state of the subject's face includes the expression of the face of the subject, the direction of the face of the subject, the degree of opening of the subject's eyes, the line of sight of the subject, the attitude of the subject, and the state of the action of the subject. 13. The imaging device according to 13. It further comprises changing means for changing the direction of the imaging means in order to direct the direction of the imaging means to the subject,
The imaging apparatus according to any one of claims 1 to 14, wherein the changing unit changes a movable range in which the direction of the imaging unit is changed according to the frequency.   16. The image pickup apparatus according to claim 15, wherein the change unit rotates the image pickup unit in a pan direction and a tilt direction. It further comprises zoom means for enlarging or reducing the image of the subject on the imaging means,
The image pickup apparatus according to any one of claims 1 to 16, wherein the zoom unit changes the control of the enlargement or reduction according to the frequency. A method of controlling an image pickup apparatus comprising an image pickup means for picking up an image of a subject and outputting image data, comprising:
A control step of controlling whether or not to perform a photographing operation of recording the image data output by the imaging unit;
Obtaining an information related to the frequency of the photographing operation;
In the control step, a threshold value for determining whether to perform the photographing operation is changed according to the information related to the frequency. A method of controlling an image pickup apparatus comprising an image pickup means for picking up an image of a subject and outputting image data, comprising:
Detecting the face of the subject;
A determination step of determining the state of the face of the subject detected by the detection step;
A control step of controlling whether or not to perform a photographing operation of recording the image data output by the imaging unit;
Obtaining an information related to the frequency of the photographing operation;
In the control step, even if the condition of the face of the subject determined in the determination step is the same, the photographing operation is performed if the frequency is the first frequency, and the frequency is the second frequency A control method of an image pickup apparatus, wherein control is performed so as not to carry out a photographing operation.   A program for causing a computer to execute each step of the control method according to claim 18 or 19.   A computer readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 18 or 19.

Images