JP7403218B2 - Imaging device, its control method, program, storage medium - Google Patents

Description

The present invention relates to automatic photographing technology in an imaging device.

When shooting still images and videos using an imaging device such as a camera, it is common for the user to determine the subject to be shot through a finder, etc., check the shooting conditions themselves, adjust the framing of the shot image, and then shoot the image. be. Traditionally, such imaging devices detect user operational errors and the external environment, notify the user if the camera is not suitable for shooting, and control the camera to bring the camera into a state suitable for shooting. It is equipped.

A life log camera (Patent Document 1) is known that performs photography periodically and continuously without the user giving a photography instruction to an imaging device that performs photography by such a user's operation. A life log camera is used while attached to a user's body with a strap or the like, and records scenes that the user sees in daily life as images at regular time intervals. When taking pictures with a life log camera, pictures are taken at regular intervals, rather than at the user's intended timing, such as when the user releases the shutter, so it is possible to capture unexpected moments that would not normally be taken. .

Special table 2016-536868 publication

However, when a user wears a life log camera and periodically performs automatic photography, the following problems occur.

One is that since images are taken at fixed time intervals regardless of the user's intention, there is a possibility that the user may miss taking the image at the moment he or she really wants to take. Another problem is that if the shooting interval is shortened in order to avoid missed shots, the power consumption for shooting will increase and the available shooting time will be shortened.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to enable an image capturing apparatus that performs automatic shooting to minimize the possibility of missing a video that a user wants to shoot.

An imaging device according to the present invention includes: an imaging means for imaging a subject image and outputting image data; a control means for controlling whether to perform a photographing operation for recording image data outputted by the imaging means; 1 ; and a determining means for determining a target number of shots in a predetermined period . The present invention is characterized in that a threshold value for determining whether or not to perform the photographing operation is changed based on information indicating the photographing operation.

Advantageous Effects of Invention According to the present invention, in an imaging device that performs automatic imaging, it is possible to minimize the possibility of a user missing a video that he or she wants to capture.

1 is a diagram schematically showing the appearance of a camera that is an embodiment of an imaging device of the present invention. FIG. 1 is a block diagram showing the overall configuration of a camera according to an embodiment. FIG. 1 is a diagram showing an example of a configuration of a wireless communication system between a camera and an external device. FIG. 3 is a diagram showing the configuration of an external device. The figure which shows the structure of a camera and an external device. FIG. 3 is a diagram showing the configuration of an external device. 5 is a flowchart illustrating the operation of the first control section. 5 is a flowchart illustrating the operation of the second control section. 5 is a flowchart illustrating the operation of shooting mode processing. FIG. 3 is a diagram for explaining area division within a photographed image. FIG. 3 is a diagram for explaining control of shooting frequency. A diagram explaining a neural network. A diagram showing how images are viewed on an external device. Flowchart illustrating learning mode determination. Flowchart illustrating learning processing.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the accompanying drawings.

<Camera configuration>
FIG. 1 is a diagram schematically showing the appearance of a camera that is an embodiment of the imaging device of the present invention. The camera 101 shown in FIG. 1(a) is provided with a power switch, an operation member for operating the camera, and the like. A lens barrel 102 that integrally includes a photographic lens group and an image sensor as an imaging optical system for imaging a subject image is movably attached to a fixed portion 103 of the camera 101. Specifically, the lens barrel 102 is attached to the fixed part 103 via a tilt rotation unit 104 and a pan rotation unit 105, which are mechanisms that can be rotated relative to the fixed part 103.

The tilt rotation unit 104 includes a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in FIG. 1(b), and the pan rotation unit 105 rotates the lens barrel 102 in the yaw direction shown in FIG. Equipped with a motor drive mechanism that can rotate in the direction. That is, the camera 101 has a mechanism that rotates the lens barrel 102 in two axial directions. Each axis shown in FIG. 1(b) is defined with respect to the position of the fixed part 103. The angular velocity meter 106 and the accelerometer 107 are arranged on the fixed part 103 of the camera 101. Then, vibration of the camera 101 is detected based on the output signals of the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit 104 and the pan rotation unit 105 are rotationally driven, thereby correcting the shake of the lens barrel 102. You can also correct the tilt. Furthermore, the angular velocity meter 106 and the accelerometer 107 also detect movement of the camera based on measurement results over a certain period.

FIG. 2 is a block diagram showing the overall configuration of the camera 101 of this embodiment. In FIG. 2, the first control unit 223 includes, for example, a CPU (MPU), memory (DRAM, SRAM), and the like. Then, according to the program stored in the nonvolatile memory (EEPROM) 216, various processes are executed to control each block of the camera 101 and data transfer between the blocks. The nonvolatile memory 216 is an electrically erasable/recordable memory, and stores constants, programs, etc. for the operation of the first control section 223 as described above.

In FIG. 2, a zoom unit 201 includes a zoom lens that performs magnification (enlargement/reduction of a formed subject image). The zoom drive control section 202 drives and controls the zoom unit 201 and detects the focal length at that time. The focus unit 203 includes a focus lens that performs focus adjustment. A focus drive control section 204 drives and controls the focus unit 203. The imaging unit 206 includes an imaging element, receives light incident through each lens group, and outputs charge information corresponding to the amount of light to the image processing unit 207 as an analog image signal. Note that the zoom unit 201, focus unit 203, and imaging section 206 are arranged inside the lens barrel 102.

The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data obtained by A/D converting the analog image signal, and converts the applied digital image data. Output. The digital image data output from the image processing section 207 is converted into a recording format such as JPEG format by the image recording section 208, and is stored in the memory 215 or transmitted to the video output section 217, which will be described later.

The lens barrel rotation drive section 205 drives the tilt rotation unit 104 and the pan rotation unit 105, and rotates the lens barrel 102 in the tilt direction and the pan direction. The device shake detection unit 209 includes an angular velocity meter (gyro sensor) 106 that detects the angular velocity of the camera 101 in three axial directions, and an accelerometer (acceleration sensor) 107 that detects the acceleration of the camera 101 in the three axial directions. Then, based on the signals detected by these sensors, the rotation angle of the device, the amount of shift of the device, etc. are calculated.

The audio input unit 213 acquires an audio signal around the camera 101 using a microphone provided in the camera 101, converts it into a digital audio signal, and transmits the digital audio signal to the audio processing unit 214. The audio processing unit 214 performs audio-related processing such as optimization processing of the input digital audio signal. The audio signal processed by the audio processing section 214 is then transmitted to the memory 215 by the first control section 223. The memory 215 temporarily stores image signals and audio signals obtained by the image processing section 207 and the audio processing section 214.

The image processing unit 207 and the audio processing unit 214 read the image signal and audio signal temporarily stored in the memory 215, encode the image signal, encode the audio signal, etc., and convert the image signal and the audio signal into compressed image signals and compressed audio signals. generate. The first control section 223 transmits these compressed image signals and compressed audio signals to the recording/reproducing section 220.

The recording and reproducing unit 220 records compressed image signals and compressed audio signals generated by the image processing unit 207 and the audio processing unit 214, and other control data related to photographing on the recording medium 221. Furthermore, when the audio signal is not compressed and encoded, the first control unit 223 sends the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording/reproducing unit 220. It is transmitted and recorded on the recording medium 221.

The recording medium 221 may be a recording medium built into the camera 101 or a removable recording medium, and can record various data such as compressed image signals, compressed audio signals, and audio signals generated by the camera 101. Generally, a medium having a larger capacity than the nonvolatile memory 216 is used as the recording medium 221. For example, the recording medium 221 includes any type of recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

The recording and reproducing unit 220 reads (reproduces) compressed image signals, compressed audio signals, audio signals, various data, and programs recorded on the recording medium 221. The first control unit 223 then transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing section 207 and the audio processing section 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode them according to a predetermined procedure, and transmit the decoded signals to the video output section 217.

A plurality of microphones are arranged in the audio input section 213, and an audio processing section 214 can detect the direction of sound with respect to the plane on which the plurality of microphones are installed, and is used for searching for a subject and automatic shooting, which will be described later. . Furthermore, the voice processing unit 214 detects a specific voice command. In addition to some commands registered in advance, the voice commands may be configured such that the user can register specific voices in the camera. It also performs sound scene recognition. In sound scene recognition, sound scenes are determined using a network trained in advance through machine learning based on a large amount of audio data. For example, a network is set up in the audio processing unit 214 to detect specific scenes such as "cheers", "claps", "sounds", etc. Detect voice commands. When the audio processing unit 214 detects a specific sound scene or specific audio command, it outputs a detection trigger signal to the first control unit 223 and the second control unit 211.

Separately from the first control section 223 that controls the entire main system of the camera 101, a second control section 211 that controls the power supply to the first control section 223 is provided. The first power supply section 210 and the second power supply section 212 supply power for operating the first control section 223 and the second control section 211, respectively. When the power button provided on the camera 101 is pressed, power is first supplied to both the first control unit 223 and the second control unit 211, but as described later, the first control unit 223 is connected to the first power supply unit. It also performs control to turn off its own power supply to 210. Even while the first control section 223 is not operating, the second control section 211 is operating, and information from the device shaking detection section 209 and the audio processing section 214 is input. The second control unit 211 determines whether or not to start the first control unit 223 based on various input information, and when it is determined that the first control unit 223 is to be started, the second control unit 211 causes the first control unit 223 to be activated. Instructs to supply power to.

The audio output unit 218 outputs a preset audio pattern from a speaker built into the camera 101, for example, when photographing. The LED control unit 224 lights up the LED provided in the camera 101 based on a preset lighting pattern or blinking pattern, for example, when photographing. The video output unit 217 includes, for example, a video output terminal, and outputs an image signal to display a video on a connected external display or the like. Further, the audio output section 218 and the video output section 217 may be a single terminal connected to each other, for example, a terminal such as an HDMI (registered trademark: High-Definition Multimedia Interface) terminal.

The communication unit 222 is a part that communicates between the camera 101 and an external device, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. It also receives control signals related to photography, such as commands to start and end photography, pan/tilt, and zoom drive, and drives the camera 101 based on instructions from an external device. Furthermore, information such as various parameters related to learning to be processed by a learning processing unit 219, which will be described later, is transmitted and received between the camera 101 and an external device. The communication unit 222 includes, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a Wireless USB (registered trademark), and a GPS receiver.

The environment sensor 226 detects the state of the environment around the camera 101 at a predetermined cycle. The environment sensor 226 includes a temperature sensor that detects the temperature around the camera 101, an atmospheric pressure sensor that detects changes in the air pressure around the camera 101, and an illuminance sensor that detects the brightness around the camera 101. Furthermore, it also includes a humidity sensor that detects the humidity around the camera 101, a UV sensor that detects the amount of ultraviolet rays around the camera 101, and the like. In addition to the detected temperature information, atmospheric pressure information, brightness information, humidity information, and UV information, the amount of temperature change, amount of atmospheric pressure change, amount of brightness change, and humidity change is calculated by calculating the rate of change at a predetermined time interval from various detected information. It is used to judge the amount of ultraviolet rays and the amount of change in ultraviolet rays for automatic shooting, etc., which will be described later.

<Communication with external devices>
FIG. 3 is a diagram showing a configuration example of a wireless communication system between the camera 101 and the external device 301. The camera 101 is a digital camera with a photographing function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

The camera 101 and the external device 301 have a first communication 302 using a wireless LAN based on the IEEE802.11 standard series, and a control station and a dependent station such as Bluetooth Low Energy (hereinafter referred to as "BLE"). It is possible to communicate with the second communication 303 having a master-slave relationship. Note that wireless LAN and BLE are examples of communication methods, and each communication device has two or more communication functions. For example, one communication function that performs communication in the relationship between a control station and a dependent station, Other communication methods may be used as long as it is possible to control the other communication function. However, the first communication 302 such as wireless LAN is capable of faster communication than the second communication 303 such as BLE, and the second communication 303 consumes less power than the first communication 302. or the communicable distance is short.

The configuration of the external device 301 will be explained using FIG. 4. The external device 301 includes, for example, a wireless LAN control unit 401 for wireless LAN and a BLE control unit 402 for BLE, as well as a public wireless control unit 406 for public wireless communication. Furthermore, the external device 301 further includes a packet transmitting/receiving section 403. The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, driver processing for controlling various types of wireless LAN communication based on the IEEE 802.11 standard series, and protocol processing regarding wireless LAN communication. The BLE control unit 402 performs BLE RF control, communication processing, driver processing for controlling various types of BLE communication, and protocol processing regarding BLE communication. The public radio control unit 406 performs RF control of public radio communication, communication processing, driver processing for performing various controls of public radio communication, and protocol processing related to public radio communication. Public wireless communication is based on, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmitting/receiving unit 403 performs processing for transmitting and/or receiving packets related to wireless LAN, BLE communication, and public wireless communication. Note that in this embodiment, the external device 301 will be described as one that performs at least one of sending and receiving packets in communication; however, other communication formats other than packet exchange, such as circuit switching, may be used. Good too.

The external device 301 further includes, for example, a control section 411, a storage section 404, a GPS reception section 405, a display section 407, an operation section 408, a voice input voice processing section 409, and a power supply section 410. The control unit 411 controls the entire external device 301, for example, by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program executed by the control unit 411 and various information such as parameters necessary for communication. Various operations described below are realized by the control unit 411 executing a control program stored in the storage unit 404.

The power supply unit 410 supplies power to the external device 301. The display unit 407 has a function capable of outputting visually recognizable information such as an LCD or LED, or outputting sound such as a speaker, and displays various information. The operation unit 408 includes, for example, buttons that accept operations on the external device 301 by the user. Note that the display section 407 and the operation section 408 may be configured by a common member such as a touch panel, for example.

The voice input voice processing unit 409 may be configured to acquire the voice uttered by the user using, for example, a general-purpose microphone built into the external device 301, and identify the user's operation command through voice recognition processing. Additionally, a dedicated application in the external device 301 is used to acquire a voice command by the user's pronunciation, and a specific voice is sent to be recognized by the voice processing unit 214 of the camera 101 via the first communication 302 using the wireless LAN. It can also be registered as a command.

A GPS (Global Positioning System) receiving unit 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude/latitude information) of the external device 301. Alternatively, the current location of the external device 301 may be estimated using WPS (Wi-Fi Positioning System) or the like based on information on surrounding wireless networks. If the acquired current GPS location information is located within a preset location range (within a predetermined radius around the detected location), or if the GPS location information has changed more than a predetermined amount. In this case, the movement information is notified to the camera 101 via the BLE control unit 402. Then, it is used as a parameter for automatic shooting and automatic editing, which will be described later.

As described above, the camera 101 and the external device 301 exchange data through communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, it transmits and receives data such as audio signals, image signals, compressed audio signals, and compressed image signals. Further, the external device 301 transmits a shooting instruction to the camera 101, transmits voice command registration data, transmits a predetermined position detection notification based on GPS position information, transmits a location movement notification, and the like. It also transmits and receives learning data using a dedicated application within the external device 301.

<Configuration of accessories>
FIG. 5 is a diagram showing a configuration example of an external device 501 that can communicate with the camera 101. The camera 101 is a digital camera with a photographing function, and the external device 501 is a wearable device that includes various sensing units that can communicate with the camera 101 using, for example, a Bluetooth communication module.

The external device 501 is configured to be worn on the user's arm, for example, and includes a sensor that detects biological information such as the user's pulse, heartbeat, and blood flow at a predetermined cycle, and an acceleration sensor that can detect the user's exercise state. etc. are installed.

The biological information detection unit 602 includes, for example, a pulse sensor that detects pulses, a heartbeat sensor that detects heartbeats, a blood flow sensor that detects blood flow, and a sensor that uses conductive polymer to detect changes in potential due to skin contact. including. In this embodiment, a heartbeat sensor will be used as the biological information detection unit 602. The heartbeat sensor detects the user's heartbeat by emitting infrared light onto the skin using, for example, an LED, and detecting the infrared light that has passed through the body tissue with a light receiving sensor and processing the signal. The biological information detection unit 602 outputs the detected biological information as a signal to the control unit 607 (see FIG. 6).

The sway detection unit 603 that detects the user's movement state is equipped with, for example, an acceleration sensor or a gyro sensor, and detects motions such as whether the user is moving or waving his arms around based on acceleration information. can be detected. Furthermore, an operation unit 605 that accepts operations on the external device 501 by a user, and a display unit 604 such as a monitor that outputs visually recognizable information such as an LCD or LED are installed.

FIG. 6 is a diagram showing the configuration of external device 501. As described above, the external device 501 includes, for example, a control section 607, a communication section 601, a biological information detection section 602, a shaking detection section 603, a display section 604, an operation section 605, a power supply section 606, and a storage section 608.

The control unit 607 controls the entire external device 501, for example, by executing a control program stored in the storage unit 608. The storage unit 608 stores, for example, a control program executed by the control unit 607 and various information such as parameters necessary for communication. Various operations described below are realized, for example, by the control unit 607 executing a control program stored in the storage unit 608.

The power supply unit 606 supplies power to the external device 501. The display unit 604 has, for example, an output unit for visually perceivable information such as an LCD or LED, or an output unit for outputting sound such as a speaker, and displays various information. The operation unit 605 includes, for example, buttons that accept operations on the external device 501 by the user. Note that the display unit 604 and the operation unit 605 may be configured by a common member such as a touch panel, for example. Further, the operation unit 605 may be configured to acquire the voice emitted by the user using, for example, a general-purpose microphone built into the external device 501, and identify the user's operation command through voice recognition processing.

Various detection information acquired by the biological information detection unit 602 and the shaking detection unit 603 and processed by the control unit 607 is transmitted to the camera 101 by the communication unit 601. For example, the detection information can be transmitted to the camera 101 at the timing when a change in the user's heartbeat is detected, or the detection information can be transmitted at the timing when the movement state changes such as walking/running/stopping. Further, the detection information may be transmitted at the timing when a preset arm swing motion is detected, or the detection information may be transmitted at the timing when movement of a preset distance is detected.

<Camera operation sequence>
FIG. 7 is a flowchart illustrating an example of the operation handled by the first control unit 223 of the camera 101 in this embodiment.

When the user operates the power button provided on the camera 101, power is supplied from the first power supply section 210 to the first control section 223 and each block of the camera 101. Similarly, power is supplied from the second power supply section 212 to the second control section 211. Details of the operation of the second control section 211 will be described later using the flowchart of FIG.

When power is supplied, the process shown in FIG. 7 starts. In step S701, startup conditions are read. In this embodiment, there are the following three conditions for starting the power supply.
(1) The power button is manually pressed to start the power. (2) A startup instruction is sent from an external device (for example, the external device 301) through external communication (for example, BLE communication), and the power is started (3) The power supply is started according to the instruction from the second control unit 211. Here, when the power supply is started according to the instruction from the second control unit 211 in (3), the activation condition calculated in the second control unit 211 is The details will be described later using FIG. 8. Further, the activation condition read here is used as one parameter element during subject search and automatic shooting, and this will also be described later. When the reading of the startup conditions is completed, the process advances to step S702.

In step S702, detection signals from various sensors are read. One of the sensor signals read here is a signal from a sensor that detects vibrations, such as a gyro sensor or an acceleration sensor in the device shake detection section 209. It is also a signal of the rotational position of the tilt rotation unit 104 and the pan rotation unit 105. Furthermore, the signals include a voice signal detected by the voice processing unit 214, a detection trigger signal for specific voice recognition, a sound direction detection signal, a detection signal of environmental information detected by the environment sensor 226, and the like. When the detection signals of various sensors are read in step S702, the process advances to step S703.

In step S703, it is detected whether a communication instruction has been sent from an external device, and if there is a communication instruction, communication with the external device is performed. For example, remote operation via wireless LAN or BLE from the external device 301, transmission and reception of audio signals, image signals, compressed audio signals, compressed image signals, etc., operation instructions such as shooting from the external device 301, voice command registration data transmission, predetermined position detection notification based on GPS position information, location movement notification, transmission and reception of learning data, etc. Further, when there is an update of user's exercise information, arm action information, heartbeat, and other biological information from the external device 501, the information is read via BLE. Note that the environment sensor 226 described above may be mounted on the camera 101, but may also be mounted on the external device 301 or the external device 501. In that case, in step S703, environment information is also read via BLE. When the communication from the external device is read in step S703, the process advances to step S704.

In step S704, a mode setting determination is made, and the process advances to step S705. In step S705, it is determined whether the operation mode is set to low power consumption mode in step S704. If you are not in any of the following modes: "Automatic shooting mode", "Automatic editing mode", "Automatic image transfer mode", "Learning mode", and "Automatic file deletion mode", the mode will be set to low power consumption mode. It will be judged. If it is determined in step S705 that the mode is low power consumption mode, the process advances to step S706.

In step S706, the second control unit 211 (SubCPU) is notified of various parameters related to the activation factor determined within the second control unit 211 (shake detection determination parameters, sound detection parameters, and time elapsed detection parameters). The values of various parameters change as they are learned in a learning process described later. When the process in step S706 is completed, the process proceeds to step S707, where the power to the first control unit 223 (Main CPU) is turned off, and the process ends.

If it is determined in step S705 that it is not the low power consumption mode, it is determined whether the mode setting in step S704 is automatic shooting mode. Here, the mode setting determination process in step S704 will be explained. The mode to be determined is selected from the following.

(1) Automatic shooting mode <Mode judgment conditions>
Based on information such as each detection information set for learning (image, sound, time, vibration, location, physical change, environmental change), elapsed time since switching to automatic shooting mode, past shooting information/number of shots, etc. When it is determined that automatic photography should be performed, automatic photography mode is set.

<In-mode processing>
In automatic shooting mode processing (step S710), the subject is automatically searched for by driving panning, tilting, and zooming based on each detection information (image, sound, time, vibration, location, change in body, and change in environment). Then, when it is determined that it is time to take a photograph of the user's preference, photographing is automatically performed.

(2) Automatic editing mode <Mode judgment conditions>
If it is determined that automatic editing should be performed based on the elapsed time since the last automatic editing and information on past captured images, automatic editing mode is set.

<In-mode processing>
In automatic editing mode processing (step S712), still images and moving images are selected based on learning, and based on learning, highlight videos are combined into one video based on image effects, edited video time, etc. An automatic editing process is performed to create the .

(3) Image transfer mode <Mode judgment conditions>
If the automatic image transfer mode is set by an instruction using a dedicated application in the external device 301, it is determined that automatic transfer should be performed based on the elapsed time since the last image transfer and information on previously captured images. When this happens, automatic image transfer mode is set.

<In-mode processing>
In automatic image transfer mode processing (step S714), the camera 101 automatically extracts images that are likely to match the user's preferences, and automatically transfers the images that are likely to be the user's preferences to the external device 301. Extraction of the user's favorite image is performed using a score that is added to each image and determined by the user's preference, which will be described later.

(4) Learning mode <Mode judgment conditions>
If it is determined that automatic learning should be performed based on the elapsed time since the last learning process, the information integrated into the image that can be used for learning, the number of learning data, etc., automatic learning sending mode will be activated. Set. Alternatively, this mode is also set when there is an instruction to set the learning mode via communication from the external device 301.

<In-mode processing>
In the learning mode process (step S716), each operation information on the external device 301 (image acquisition information from the camera, information manually edited via a dedicated application, judgment value information input by the user for the image in the camera) , based on notification of learning information from the external device 301, etc., uses a neural network to perform learning tailored to the user's preferences. Further, learning related to detection such as registration of personal authentication, voice registration, sound scene registration, general object recognition registration, etc., and learning of the above-mentioned low power consumption mode conditions, etc. are also performed at the same time.

(5) File automatic deletion mode <Mode judgment conditions>
When it is determined that automatic file deletion should be performed based on the elapsed time since the previous automatic file deletion and the remaining capacity of the non-volatile memory 216 in which images are recorded, automatic file deletion mode is set. Ru.

<In-mode processing>
In the automatic file deletion mode process (step S718), files to be automatically deleted are designated from among the images in the nonvolatile memory 216 based on the tag information of each image, the date and time of photographing, and the like.

Details of the processing in each of the above modes will be described later.

Returning to the explanation of FIG. 7, if it is determined in step S705 that the low power consumption mode is not set, the process proceeds to step S709, and it is determined whether the mode setting is automatic shooting mode. As a result of the determination, if it is the automatic photographing mode, the process advances to step S710, and automatic photographing mode processing is performed. When the process ends, the process returns to step S702 and repeats the process. If it is determined in step S709 that the mode is not automatic shooting mode, the process advances to step S711.

In step S711, it is determined whether the mode setting is automatic editing mode, and if it is automatic editing mode, the process advances to step S712, where automatic editing mode processing is performed. When the process ends, the process returns to step S702 and repeats the process. If it is determined in step S711 that the mode is not automatic editing mode, the process advances to step S713. Note that the automatic editing mode is not directly related to the gist of the present invention, so a detailed explanation will be omitted.

In step S713, it is determined whether the mode setting is automatic image transfer mode, and if it is automatic image transfer mode, the process advances to step S714, where automatic image transfer mode processing is performed. When the process ends, the process returns to step S702 and repeats the process. If it is determined in step S713 that the mode is not automatic image transfer mode, the process advances to step S715. Note that the automatic image transfer mode is not directly related to the gist of the present invention, so a detailed explanation will be omitted.

In step S715, it is determined whether the mode setting is learning mode, and if it is learning mode, the process advances to step S716, where learning mode processing is performed. When the process ends, the process returns to step S702 and repeats the process. If it is determined in step S715 that the mode is not learning mode, the process advances to step S717.

In step S717, it is determined whether the mode setting is automatic file deletion mode, and if it is automatic file deletion mode, the process advances to step S718, where automatic file deletion mode processing is performed. When the process ends, the process returns to step S702 and repeats the process. If it is determined in step S717 that the file automatic deletion mode is not set, the process returns to step S702 and repeats the process. Note that the automatic file deletion mode is not directly related to the gist of the present invention, so detailed explanation will be omitted.

FIG. 8 is a flowchart illustrating an example of the operation handled by the second control unit 211 of the camera 101 in this embodiment.

When the user operates the power button provided on the camera 101, power is supplied from the first power supply section 210 to the first control section 223 and each block of the camera 101. Similarly, power is supplied from the second power supply section 212 to the second control section 211.

When power is supplied, the second control unit (SubCPU) 211 is activated and the process shown in FIG. 8 starts. In step S801, it is determined whether a predetermined sampling period has elapsed. The predetermined sampling period is set to, for example, 10 msec, and the process proceeds to step S802 at a 10 msec period. If it is determined that the predetermined sampling period has not elapsed, the second control unit 211 remains on standby.

In step S802, learning information is read. The learning information is information transferred when communicating information to the second control unit 211 in step S706 of FIG. 7, and includes, for example, the following information.
(1) Determination of specific shaking detection (used in step S804 described later)
(2) Determination of specific sound detection (used in step S805 described later)
(3) Determining the passage of time (used in step S807 described later)
When the learning information is read in step S802, the process advances to step S803, and a shaking detection value is acquired. The shaking detection value is an output value of a gyro sensor, an acceleration sensor, or the like in the device shaking detection section 209.

When the shaking detection value is acquired in step S803, the process advances to step S804, and a preset specific shaking state detection process is performed. Here, the determination process is changed based on the learning information read in step S802. Some examples will be explained.

<Tap detection>
A state in which the user taps the camera 101 with, for example, a fingertip (tap state) can be detected from the output value of the acceleration sensor 107 attached to the camera 101. By passing the output of the triaxial acceleration sensor 107 through a band pass filter (BPF) set in a specific frequency range at a predetermined sampling period, it is possible to extract the signal range of acceleration changes due to taps. Tap detection is performed depending on whether the number of times the acceleration signal passed through the BPF exceeds a predetermined threshold value ThreshA during a predetermined time TimeA is a predetermined number of times CountA. For double taps, CountA is set to 2, and for triple taps, CountA is set to 3. Furthermore, TimeA and ThreshA can also be changed depending on the learning information.

<Detection of shaking condition>
It is possible to detect the shaking state of the camera 101 from the output values of the gyro sensor 106 and acceleration sensor 107 attached to the camera 101. After cutting high frequency components of the outputs of the gyro sensor 106 and the acceleration sensor 107 with a high pass filter (HPF), and cutting low frequency components with a low pass filter (LPF), absolute value conversion is performed. Vibration detection is performed depending on whether the number of times the calculated absolute value exceeds a predetermined threshold value ThreshB during a predetermined time TimeB is equal to or greater than a predetermined number CountB. This makes it possible to determine whether the shaking is small, such as when the camera 101 is placed on a desk or the like, or the shaking is large, such as when the camera 101 is worn as a wearable camera while walking. Further, by setting a plurality of conditions for determination threshold values and determination count numbers, it is also possible to detect fine shaking states according to the shaking level. TimeB, ThreshB, and CountB can also be changed depending on learning information.

The method for detecting a specific shaking state by determining the detection value of the shaking detection sensor has been described above. However, by inputting data from the shaking detection sensor sampled within a predetermined period of time into a shaking state determiner using a neural network, the trained neural network can detect a specific shaking state that has been registered in advance. It is also possible to detect. In that case, the learning information read in step S802 becomes the weight parameter of the neural network.

When a specific shaking state detection process is performed in step S804, the process advances to step S805, and a preset specific sound detection process is performed. Here, the detection determination process is changed based on the learning information read in step S802. Some examples will be explained.

<Specific voice command detection>
Detect specific voice commands. In addition to several pre-registered voice commands, the user can also register specific voices to the camera.

<Specific sound scene recognition>
Sound scenes are determined using a network trained through machine learning based on a large amount of audio data in advance. For example, specific scenes such as ``cheers'', ``claps'', and ``sounds'' are detected. The scenes to be detected change through learning.

<Sound level judgment>
The sound level is detected by determining whether the sound level exceeds a predetermined level for a predetermined period of time. The predetermined time, predetermined size, etc. change depending on learning.

<Sound direction determination>
A plurality of microphones arranged on a plane detect the direction of sound of a predetermined loudness.

The above-mentioned determination process is performed within the audio processing unit 214, and it is determined in step S805 whether a specific sound has been detected based on each setting learned in advance.

When the specific sound detection processing is performed in step S805, the process advances to step S806, and it is determined whether the power of the first control unit 223 is in the OFF state. If the first control unit 223 (Main CPU) is in the OFF state, the process advances to step S807, and a preset time elapse detection process is performed. Here, the detection determination process is changed based on the learning information read in step S802. The learning information is information transferred when communicating information to the second control unit 211 in step S706 described in FIG. 7. The elapsed time from when the first control unit 223 transitions from ON to OFF is measured, and if the elapsed time is greater than or equal to a predetermined time TimeC, it is determined that the time has elapsed, and if it is shorter than TimeC, the time has elapsed. It is determined that this has not been done. TimeC is a parameter that changes depending on learning information.

When the time elapse detection process is performed in step S807, the process advances to step S808, and it is determined whether the conditions for canceling the low power consumption mode are satisfied. Canceling the low power consumption mode is determined based on the following conditions.
(1) A specific shaking has been detected. (2) A specific sound has been detected. It is determined whether or not shaking has been detected. Regarding (2), it is determined whether a specific sound has been detected by the specific sound detection process in step S805. Regarding (3), it is determined whether a predetermined time has elapsed by the time elapse detection process in step S807. If at least one of (1) to (3) is satisfied, it is determined to cancel the low power consumption mode.

When it is determined in step S808 that the low power consumption mode is to be canceled, the process proceeds to step S809, where the first control unit 223 is powered on, and in step S810, the conditions (shaking, sound, time) to the first control unit 223. Then, the process returns to step S801 and loops. If it is determined in step S808 that none of the cancellation conditions apply and the condition is not for canceling the low power consumption mode, the process returns to step S801 and loops.

On the other hand, if it is determined in step S806 that the first control unit 223 is in the ON state, the process advances to step S811, and the information acquired in steps S803 to S805 is notified to the first control unit 223, and the process returns to step S801. Loop the process.

In this embodiment, even when the first control section 223 is in the ON state, the second control section 211 detects shaking and detects specific sounds, and the detection results are notified to the first control section 223. . However, if the first control unit 223 is ON, the process in steps S803 to S805 is not performed, and the configuration is such that vibration detection and specific sound detection are performed in the process within the first control unit 223 (step S702 in FIG. 7). You can.

As described above, by performing steps S704 to S707 in FIG. 7 and the processing in FIG. 8, the conditions for transitioning to the low power consumption mode and the conditions for canceling the low power consumption mode are learned based on the user's operation. Ru. Then, it becomes possible to perform camera operations tailored to the usability of the user who owns the camera 101. The learning method will be described later.

Note that although the method for canceling the low power consumption mode based on shaking detection, sound detection, or the passage of time has been described in detail above, the low power consumption mode may also be canceled based on environmental information. The environmental information can be determined based on whether the absolute amount or amount of change in temperature, atmospheric pressure, brightness, humidity, or amount of ultraviolet rays exceeds a predetermined threshold, and the threshold can also be changed by learning described later.

Further, the determination to cancel the low power consumption mode may be made by determining the absolute value and amount of change of each environmental information based on the vibration detection, sound detection, detection information of the passage of time, and the amount of change of each environmental information. In this judgment process, the judgment conditions can be changed through learning, which will be described later.

<Automatic shooting mode processing>
Automatic photographing mode processing will be explained using FIG. 9. First, in step S901, the image processing unit 207 performs image processing on a signal captured by the imaging unit 206 to generate an image for subject detection. Subject detection processing for detecting people, objects, etc. is performed on the generated image.

When detecting a person, the face or human body of the subject is detected. In the face detection process, a pattern for determining a person's face is determined in advance, and a portion of the captured image that matches the pattern can be detected as a person's face area. In addition, the reliability level indicating the certainty that the subject's face is true is also calculated at the same time. The reliability is calculated from, for example, the size of the face area in the image, the degree of matching with the face pattern, and the like. Similarly, for object recognition, it is possible to recognize objects that match a pre-registered pattern.

There is also a method of extracting a characteristic object using a histogram of hue, saturation, etc. in a captured image. Regarding the image of the subject captured within the photographic field of view, a process is performed in which the distribution derived from the histogram of hue, saturation, etc. is divided into a plurality of sections, and the captured image is classified for each section. For example, a histogram of multiple color components is created for a captured image, divided by its mountain-shaped distribution range, images captured in areas that belong to the same combination of sections are classified, and the image area of the subject is recognized. be done. By calculating an evaluation value for each image area of the recognized object, it is possible to determine the image area of the object with the highest evaluation value as the main object area. With the above method, each subject information can be obtained from the imaging information.

In step S902, an image blur correction amount is calculated. Specifically, first, the absolute angle of the camera shake is calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the tilt rotation unit 104 and the pan rotation unit 105 are moved in an angular direction that cancels out the absolute angle, and an angle for correcting the image blur is determined, and is used as an image blur correction amount. Note that the calculation method of the image blur correction amount calculation process here can be changed by a learning process described later.

In step S903, the status of the camera is determined. The current vibration/motion state of the camera is determined based on the camera angle, camera movement amount, etc. detected using angular velocity information, acceleration information, GPS position information, etc. For example, when the camera 101 is attached to a car to take a picture, subject information such as surrounding scenery changes greatly depending on the distance traveled. Therefore, it is determined whether the device is in a ``vehicle moving state'' where it is attached to a car or the like and is moving at a high speed, and is used for automatic subject search, which will be explained later. Further, it is determined whether the change in the angle of the camera is large or not, and whether the camera 101 is in a "stationary shooting state" in which there is almost no shaking is determined. When the camera 101 is in the "stationary shooting state", it can be considered that there is no change in the position of the camera 101 itself, so it is possible to search for a subject for stationary shooting. Furthermore, if the angle change of the camera is relatively large, it is determined that the camera is in a "hand-held state", and a hand-held object search can be performed.

In step S904, subject search processing is performed. The object search consists of the following processing.
(1) Area division (2) Calculation of importance level for each area (3) Determination of search target area Below, each process will be explained in sequence.

(1) Area division Area division will be explained using FIG. 10A. As shown in FIG. 10A(a), area division is performed all around the camera position (origin O is the camera position) as the center. In the example of FIG. 10A(a), the tilt direction and the pan direction are each divided into 22.5 degrees. When divided as shown in FIG. 10A(a), as the angle in the tilt direction moves away from 0 degrees, the circumference in the horizontal direction becomes smaller and the area becomes smaller. Therefore, as shown in FIG. 10A(b), when the tilt angle is 45 degrees or more, the horizontal area range is set to be larger than 22.5 degrees.

FIGS. 10A(c) and 10A(d) show examples of areas divided into areas within the shooting angle of view. An axis 1301 is the direction of the camera 101 at the time of initialization, and area division is performed using this direction as a reference position. Reference numeral 1302 indicates the viewing angle area of the image being captured, and an example of the image at that time is shown in FIG. 10A(d). Within the image of the angle of view being captured, the image is divided as indicated by numerals 1303 to 1318 in FIG. 10A(d) based on area division.

(2) Calculating the importance level for each area For each area divided as above, calculate the importance level that indicates the priority for searching, depending on the situation of objects existing in the area and the situation of the scene. . The importance level based on the situation of the subject is determined based on, for example, the number of people in the area, the size of the faces of the people, the orientation of the faces, the certainty of face detection, the facial expressions of the people, the results of personal identification of the people, etc. Calculated based on In addition, the importance level depending on the situation of the scene includes, for example, general object recognition results, scene discrimination results (blue sky, backlight, sunset view, etc.), sound levels and voice recognition results from the direction of the area, and motion detection within the area. Calculated based on information etc.

Further, if camera vibration is detected in the camera state determination (step S903) in FIG. 9, the importance level can also be changed depending on the vibration state. For example, when it is determined that the camera is in a "stationary shooting state", it is determined that a subject search is performed centering on a subject with a high priority (for example, the owner of the camera) among those registered by face authentication. Further, automatic photographing, which will be described later, is also performed with priority given to the face of the owner of the camera, for example. As a result, even if the owner of the camera spends a lot of time carrying the camera around and taking pictures, by removing the camera and placing it on a desk or other surface, many images of the owner can be left. can. At this time, it is possible to search for faces by panning and tilting, so you can create images of the owner or group photos of many faces by simply setting up the camera, without having to think about the angle of the camera. You can leave it behind.

Note that if only the above conditions are met, as long as there is no change in each area, the area with the highest level of importance will remain the same, and as a result, the area to be searched will remain the same forever. Therefore, the importance level is changed depending on past shooting information. Specifically, the importance level may be lowered for an area that has been designated as a search area for a predetermined period of time, or the importance level may be lowered for a predetermined period of time for an area photographed in step S910, which will be described later. .

(3) Determining the search target area Once the importance level of each area has been calculated as described above, the area with the higher importance level is determined as the search target area. Then, the pan/tilt search target angle required to capture the search target area at the angle of view is calculated.

Returning to the explanation of FIG. 9, in step S905, pan/tilt driving is performed. Specifically, the pan/tilt drive amount is calculated by adding the image blur correction amount at the control sampling frequency and the drive angle based on the pan/tilt search target angle. The lens barrel rotation drive section 205 drives and controls the tilt rotation unit 104 and the pan rotation unit 105, respectively.

In step S906, the zoom unit 201 is controlled to drive the zoom. Specifically, the zoom is driven according to the state of the search target object determined in step S904. For example, if the object to be searched is a person's face, if the face on the image is too small, it will be smaller than the minimum detectable size and will not be detected and may be lost. In such a case, control is performed so that the size of the face on the image becomes larger by zooming toward the telephoto side. On the other hand, if the face in the image is too large, the subject is likely to move out of the field of view due to movement of the subject or the camera itself. In such a case, zoom to the wide-angle side to reduce the size of the face on the screen. By performing zoom control in this manner, it is possible to maintain a state suitable for tracking the subject.

In step S907, it is determined whether there is a manual photographing instruction, and if there is a manual photographing instruction, the process advances to step S910. At this time, the manual photographing instruction may be by pressing the shutter button, by lightly tapping (tapping) the camera housing with a finger, by inputting a voice command, by an instruction from an external device, or the like. A shooting instruction triggered by a tap operation is determined by detecting continuous high-frequency acceleration in a short period of time by the device shake detection unit 209 when the user taps the camera housing. Voice command input is a photography instruction method in which when a user utters a command word (for example, "take a photo") instructing a predetermined photography, the voice processing unit 214 recognizes the voice and uses it as a trigger for photography. The instruction from the external device is, for example, a shooting instruction method in which a shutter instruction signal transmitted from a smartphone or the like connected to the camera using a dedicated application is used as a trigger.

If there is no manual photographing instruction in step S907, the process advances to step S908 and automatic photographing determination is performed. In the automatic shooting determination, it is determined whether or not automatic shooting is to be performed, and the shooting method is determined (determining which of still image shooting, video shooting, continuous shooting, panoramic shooting, etc. to be performed).

<Determining whether to perform automatic shooting>
A determination as to whether or not to perform automatic photography (a photography operation that records image data output by the imaging unit) is performed as follows. Specifically, in the following two cases, it is determined that automatic photographing is to be performed. First, based on the importance level for each area obtained in step S904, if the importance level exceeds a predetermined value, it is determined that automatic photographing is to be performed. The second method is determination based on a neural network, which will be described later. Note that the recording here may be recording the image data to the memory 215 or recording the image data to the nonvolatile memory 216. It also includes one that automatically transfers images to the external device 301 and records the image data on the external device 301 side.

In this embodiment, as described above, the automatic imaging determination parameters such as the importance level are used to control the automatic imaging. The following problems occur in an imaging device that automatically takes pictures when a predetermined condition is met.

One is when the frequency of automatic shooting is high. Even if you want to take pictures evenly within a set period of time, the camera will take pictures only if certain conditions are met, so the frequency of shooting will be very high during the first half of the time, and the remaining battery will run out during the second half of the time. There is a possibility that you will not be able to take pictures due to insufficient capacity/card remaining capacity.

Another case is when the frequency of automatic photographing is low. Even if a trader or the like wants to take a predetermined number of images, the predetermined conditions for automatic photography may not be easily met, resulting in a shortage of the number of images to be taken.

Therefore, in order to control the frequency of shooting, it may be better to change the automatic shooting determination parameters depending on the situation at the scene and the situation of the camera.

For example, in events such as weddings where time is limited, the following automatic shooting control tends to be preferred.
(1) I want to take a large number of images, including people and things. (2) I want to shoot for a short time without worrying about the remaining battery power or recording media. (3) I want to be able to pan and tilt. On the other hand, if you want to record the events of a whole day during such limited shooting time, automatic shooting control such as the one described below tends to be preferred.
(1) I want to select subjects to a certain extent because I am shooting for a long time. (2) I want to shoot in an energy-efficient way, taking into account the remaining battery power and recording media. (3) Pan and tilt control consume more battery than normal control. An example of the above-mentioned control will be explained using FIG. 10B(a). First, photographing conditions (for example, wedding ceremony: 2 hours) such as photographing time T (total photographing time) are input by user's instructions (for example, from the external device 301 or voice input). The target number of shots S is determined based on this input information, the remaining amount of the battery (such as the first power supply section 210), and the remaining amount of the recording medium (such as the recording medium 221).

Monitoring is performed at regular time intervals, and automatic shooting determination thresholds and camera control parameters are updated as needed based on the monitoring results. Note that the determination threshold for automatic photography and the initial value of the camera control parameters are determined by the results of previous learning or determination using a neural network.

In the example shown in FIG. 10B(a), the parameters are updated so that the number of captured images falls within the region R indicated by the broken line in the figure. In FIG. 10B(a), the horizontal axis shows elapsed time, and the vertical axis shows the cumulative number of captured images. Controlling the number of shots so that it falls within region R means that if you shoot so that the cumulative number of shots increases approximately linearly over time, shots will be taken almost evenly over the entire shooting time. This is because it is thought that it is possible to do the following.

In the example shown in FIG. 10B(a), it is determined that the number of shots is insufficient at monitoring time A, the automatic shooting judgment threshold is lowered, the pan and tilt movable ranges are expanded, the main subject is actively searched, and the automatic Increase the frequency of shooting. On the other hand, at monitoring time B, it is determined that the number of shots is large, and the automatic shooting determination threshold is raised, the pan and tilt movable ranges are narrowed, and the shooting frequency is reduced. At monitoring time C, it is determined that an appropriate number of shots has been obtained, and the automatic shooting determination threshold and the control parameters for the pan and tilt movable ranges are maintained. In this way, the number of images to be photographed for each fixed period is monitored at regular intervals, and the photographing frequency is controlled as needed toward the target number of images S for the photographing time T, which is a predetermined period.

For example, the CPU that controls the camera 101 includes a detection unit that detects a subject's face based on image information, and a detection unit that recognizes facial expressions to determine whether the expression is in a specific state (for example, joy, sadness, anger, or surprise). The image forming apparatus includes a determination section that determines whether the characteristic value exceeds a threshold value), and a control section that performs a subject recording operation (automatic photographing) in accordance with the determination result of the determination section. In this case, the automatic imaging determination threshold is adjusted according to the imaging frequency. With this adjustment, even if the facial expressions of the subject judged by the judgment unit are the same, the shooting operation is performed when the shooting frequency is the first frequency, and the shooting operation is performed when the shooting frequency is the second frequency. is controlled so that the photographing operation is not performed. This makes it possible to obtain the desired number of images and to reduce the shortage of recording memory.

For example, the CPU that controls the camera 101 may also include a detection unit that detects the face of the subject based on image information, and a detection unit that recognizes the direction of the subject's face and detects whether the face is facing in a specific direction, especially the front. It has a determination section that determines whether there is a bird present, and a control section that performs automatic photographing according to the determination result of the determination section. In this case, the determination threshold value for automatic photographing (threshold value for determining whether or not the subject is facing forward) is adjusted according to the photographing frequency. With this adjustment, even if the direction of the subject's face determined by the determination unit is the same, the photographing operation is performed when the photographing frequency is the first frequency, and the photographing operation is performed when the photographing frequency is the second frequency. is controlled so that the photographing operation is not performed. This makes it possible to obtain the desired number of images and to reduce the shortage of recording memory.

The same applies to the case where the subject's eye condition is recognized and automatic photographing is performed when the subject's eyes are in a predetermined state, particularly when the eyes are fully open and the eyes are facing the camera. The same applies when the posture of the subject is recognized and automatic photographing is performed when the subject assumes a predetermined posture. Further, the same applies when the motion of the subject is recognized and automatic photographing is performed when the subject performs a predetermined motion.

In this way, when the state of the subject is recognized, it is determined whether the subject is in a specific state, and automatic shooting is performed according to the judgment result, if the shooting frequency is the first frequency, performs the photographing operation, and when the photographing frequency is the second frequency, control is performed so that the photographing operation is not performed. This makes it possible to obtain the desired number of images during the photographing time T and to reduce the shortage of recording memory during the photographing time.

As another control example, an example shown in FIG. 10B(b) will be described. As in the control example shown in FIG. 10B(a), first, the shooting time T is inputted by the user's instruction, and the target number of images S is determined. Monitoring is performed at regular time intervals and the results are retained. The content held here is an evaluation value used to determine whether to automatically shoot within the monitoring period and indicates whether the image is worth shooting for the user. FIG. 10B(b) shows the result of automatically capturing an image (only one image) with an evaluation value equal to or higher than the threshold Th. Therefore, at time T1, it is determined that the number of captured images is insufficient, and control is performed to lower the determination threshold of the evaluation value. On the other hand, at time T2, as a result of lowering the evaluation value threshold, 5 images were automatically taken and it was determined that the number of images was too many compared to the target number, so the evaluation value judgment threshold was raised and the shooting frequency was lowered. It becomes. In this way, the evaluation values used in past automatic shooting judgments are monitored, and the judgment thresholds are updated as needed to obtain an appropriate number of shots. This suppresses excessive photographing processing within a short period of time. Furthermore, it is possible to obtain the desired number of images during the photographing time T, and to reduce the shortage of recording memory during the photographing time.

Further, as another control example, a configuration will be described in which captured image data is saved only when the evaluation value of the image exceeds a threshold value. During automatic photographing, the number of photographed images is monitored at regular intervals (for example, at times T1, T2, T3, and T4), and the threshold value of the image evaluation value is changed toward the target number of images. For example, if the number of captured images is insufficient at time T1, the threshold of the image evaluation value is lowered to make it easier to save. For example, if it is determined at time T2 that the number of images taken is low, the threshold of the evaluation value of the image is increased to make it difficult to save the image. This makes it possible to obtain the desired number of images during the photographing time T and to reduce the shortage of recording memory during the photographing time.

By controlling (changing) the shooting frequency according to the shooting situation in this way, it is possible to perform automatic shooting that allows an appropriate number of shots to be taken. As a result, in an imaging device that performs automatic shooting, it is possible to minimize the possibility of missing a video that the user wants to shoot.

Note that in the above, the shooting frequency is controlled to be changed according to the shooting situation, but the shooting frequency is also controlled to be changed in consideration of the communication performance between the camera 101 and the external device. You can.

Next, the second determination, which is based on a neural network, will be explained. As an example of a neural network, an example of a network using a multilayer perceptron is shown in FIG. Neural networks are used to predict output values from input values, and by learning input values and model output values for those inputs in advance, they can be used to predict output values from new input values. On the other hand, it is possible to estimate the output value based on the learned model. Note that the learning method will be described later. In FIG. 11, 1201 and the circles arranged vertically therein indicate neurons in the input layer, 1203 and the circles arranged vertically therein indicate neurons in the intermediate layer, and 1204 indicate neurons in the output layer. Arrows such as 1202 indicate connections connecting each neuron. In neural network-based determination, the input layer neurons are given features based on the subject in the current field of view, the scene, and the camera state, and then perform calculations based on the forward propagation law of a multilayer perceptron. Then, the value output from the output layer is obtained. Then, if the output value is equal to or greater than the threshold value, a determination is made to perform automatic imaging. The characteristics of the subject include the current zoom magnification, general object recognition results at the current angle of view, face detection results, number of faces in the current angle of view, degree of smiling face, degree of eyes closed, face angle, and face recognition. ID number, viewing angle of the subject, scene determination results, elapsed time since the previous shooting, current time, GPS location information and amount of change from the previous shooting position, current audio level, person making a voice, applause, It uses information such as whether there are cheers, vibration information (acceleration information, camera status), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), etc. Further, when there is information notification from the external device 501, the notification information (user's exercise information, arm action information, biological information such as heartbeat, etc.) is also used as a feature. This feature is converted into a numerical value within a predetermined range and given to each neuron in the input layer as a feature amount. Therefore, each neuron in the input layer is required as many as the number of features used.

Note that in the judgment based on this neural network, the output value can be changed by changing the connection weight between each neuron in a learning process described later, and the judgment result can be adapted to the learning result.

Further, the determination of automatic shooting also changes depending on the activation conditions of the first control unit 223 read in step S702 in FIG. For example, in the case of activation by tap detection or activation by a specific voice command, there is a very high possibility that the operation is because the user currently wants to take a picture. Therefore, the shooting frequency is set to increase.

<Determination of shooting method>
In determining the shooting method, it is determined whether to perform still image shooting, video shooting, continuous shooting, panorama shooting, etc., based on the camera status and the status of surrounding subjects detected in steps S901 to S904. do. For example, if the subject (person) is stationary, still image shooting is performed, and if the subject is moving, video shooting or continuous shooting is performed. In addition, if there are multiple subjects surrounding the camera, or if it is determined that the subject is a scenic spot based on the GPS information mentioned above, images can be taken one after another while operating the pan/tilt. A panoramic photographing process may be performed in which the images are combined to generate a panoramic image. Note that, similar to the determination method in <Determining whether or not to perform automatic photographing>, the photographing method may be determined by determining various information detected before photographing based on a neural network. Further, in this judgment process, the judgment conditions can also be changed by a learning process to be described later.

Returning to the explanation of FIG. 9, in step S909, if a determination is made to perform automatic photography in step S908, the process advances to step S910, and if a determination is not made to perform automatic photography, automatic photography mode processing is performed. finish.

In step S910, automatic photographing is started. At this time, photographing using the photographing method determined in step S908 is started. At this time, autofocus control is performed by the focus drive control unit 204. Further, using an aperture control section, a sensor gain control section, and a shutter control section (not shown), exposure control is performed so that the subject has appropriate brightness. Further, after photographing, the image processing unit 207 performs various known image processing such as auto white balance processing, noise reduction processing, and gamma correction processing to generate an image.

Note that when a predetermined condition is satisfied during this photographing, the camera may notify the person to be photographed that the photograph will be taken, and then the photograph may be taken. As a method of notification, for example, sound output from the audio output unit 218 or lighting of an LED by the LED control unit 224 may be used. The predetermined conditions include, for example, the number of faces within the angle of view, the degree of smile on the face, the degree of closed eyes, the gaze angle and face angle of the subject person, the facial recognition ID number, the number of people registered for personal authentication, and the time of shooting. General object recognition results, scene classification results, elapsed time since the previous shooting, shooting time, whether the current location is a scenic spot based on GPS information, audio level at the time of shooting, presence or absence of a person making a voice. , whether there is applause or cheering, vibration information (acceleration information, camera status), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), etc. By performing notification photography based on these conditions, it is possible to leave a preferable camera-looking image in a highly important scene.

Regarding such notification before photography, information on the photographed image or various information detected before photography can be determined based on a neural network, and the method and timing of notification can be determined. Further, in this judgment process, the judgment conditions can also be changed by a learning process to be described later.

In step S911, editing processing such as processing the image generated in step S910 and adding it to a video is performed. Specifically, image processing includes trimming processing based on a person's face or focus position, image rotation processing, HDR (high dynamic range) effect processing, blurring effect processing, color conversion filter effect processing, and the like. In image processing, a plurality of processed images may be generated based on the image generated in step S910 by a combination of the above processes, and stored separately from the image generated in step S910. Further, regarding video processing, processing may be performed in which a captured video or still image is added to an already generated edited video while applying special effects such as slide, zoom, and fade. Regarding the editing in step S911, it is also possible to determine the image processing method by determining information on the photographed image or various information detected before photographing based on a neural network. Further, in this judgment process, the judgment conditions can also be changed by a learning process to be described later.

In step S912, learning information generation processing for the captured image is performed. Here, information used for learning processing, which will be described later, is generated and recorded. Specifically, the zoom magnification at the time of shooting, the general object recognition result at the time of shooting, the face detection result, the number of faces in the shot image, the smiling degree of the face, the degree of eye closure, the face angle, and the face Authentication ID number, viewing angle of the subject, scene determination results, elapsed time since the previous shooting, shooting time, GPS location information and amount of change from the previous shooting position, audio level at the time of shooting, person making the voice, Whether there is applause or cheers, vibration information (acceleration information, camera status), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), video recording time, whether manual recording instructions were given, etc. be. It also calculates a score, which is the output of a neural network that quantifies the user's image preferences. This information is generated and recorded as tag information in the photographed image file. Alternatively, the information on each photographed image may be written into the nonvolatile memory 216 or stored in the recording medium 221 in a list format as so-called catalog data.

In step S913, past photographic information is updated. Specifically, regarding the number of shots for each area described in step S908, the number of shots for each person registered for personal authentication, the number of shots for each subject recognized by general object recognition, and the number of shots for each scene for scene discrimination, The count of the number of images corresponding to the image taken this time is increased by one.

<Learning process>
Next, learning tailored to the user's preferences in this embodiment will be explained. In this embodiment, a neural network as shown in FIG. 11 is used, and a machine learning algorithm is used to perform learning in accordance with the user's preferences in the learning processing unit 219. Neural networks are used to predict output values from input values, and by learning the actual values of input values and actual values of output values in advance, they can predict output values for new input values. value can be estimated. By using a neural network, learning is performed in accordance with the user's preferences for the aforementioned automatic shooting, automatic editing, and subject search. It also registers subject information (results of facial recognition, general object recognition, etc.), which also serves as feature data input to the neural network, and uses learning to change shooting notification control, low power consumption mode control, and automatic file deletion.

In this embodiment, the operations to which learning processing is applied are the following operations.
(1) Automatic shooting (2) Automatic editing (3) Subject search (4) Subject registration (5) Shooting notification control (6) Low power consumption mode control (7) Automatic file deletion (8) Image stabilization (9) Image Automatic Transfer Of the operations to which the above-described learning process is applied, automatic editing, automatic file deletion, and automatic image transfer are not directly related to the gist of the present invention, and therefore their explanations will be omitted.

<Automatic shooting>
Learn about automatic shooting. Automatic shooting involves learning to automatically take images that match the user's preferences. As explained using the flowchart of FIG. 9, the learning information generation process (step S912) is performed after the photographing (after step S910). Images to be trained are selected by a method described later, and learning is performed by changing the weights of the neural network based on learning information included in the images.

Learning is performed by changing the neural network that determines the automatic shooting timing and the neural network that determines the shooting method (still image shooting, video shooting, continuous shooting, panoramic shooting, etc.).

<Subject search>
Learning for object search will be explained. In object search, learning is performed to automatically search for objects that match the user's preferences. As explained using the flowchart of FIG. 9, in the subject search process (step S904), the importance level of each area is calculated, pan/tilt, and zoom are driven, and the subject search is performed. Learning is performed based on captured images and detected information during search, and is reflected in the learning results by changing the weights of the neural network. By inputting various detection information during search operations into a neural network and determining the level of importance, object searches that reflect learning are performed. In addition to calculating the importance level, it also controls, for example, the pan/tilt search method (speed, frequency of movement).

<Subject registration>
Learning for subject registration will be explained. In subject registration, learning is performed to automatically register and rank subjects that match the user's preferences. As learning, for example, facial recognition registration, general object recognition registration, gesture and voice recognition, and scene recognition using sound are registered. Authentication registration is performed for people and objects, and rankings are set based on the number and frequency of images taken, the number and frequency of manual shooting, and the frequency with which the subject appears during the search. The registered information will be registered as input for determination using each neural network.

<Photography notification control>
Learning for shooting notification will be explained. As described in step S910 of FIG. 9, immediately before photographing, when a predetermined condition is satisfied, the camera notifies the person to be photographed that the photograph will be taken, and then photographs the person. For example, by driving panning and tilting, the subject's line of sight can be visually guided, or by using a speaker sound emitted from the audio output section 218 or an LED lighting light from the LED control section 224, the subject's attention can be guided. do. Immediately after the above notification, based on whether or not the detection information of the subject (e.g. smile level, line of sight detection, gesture) is obtained, it is determined whether the detection information is used for learning and the weight of the neural network is changed. Learn by doing.

Each detection information immediately before shooting is input into a neural network, and it is used to determine whether or not to issue a notification, as well as each movement (sound (sound level/sound type/timing), light (lighting time, speed), camera direction ( Pan/tilt motion)) is determined.

<Low power consumption mode control>
Using FIGS. 7 and 8, as explained above, the power supply to the Main CPU (first control unit 223) is controlled to be turned ON/OFF. It also learns the transition conditions for . Learning the conditions for canceling low power consumption mode will be explained.

<Sound detection>
The user can learn by manually setting specific sounds, specific sound scenes and specific sound levels that the user wants to detect, for example, through communication using a dedicated application of the external device 301. In addition, multiple detection methods are set in advance in the audio processing unit, and the image to be learned is selected by the method described later, and the sound information included in the image is learned, and the sound judgment as the activation factor (specific sound command, etc.) is performed. You can also learn by setting sound scenes (such as ``cheers'', ``applause'', etc.).

<Environmental information detection>
It is possible to learn by manually setting environmental information changes that the user wants to use as activation conditions, for example, through communication using a dedicated application of the external device 301. For example, it can be activated based on specific conditions such as temperature, atmospheric pressure, brightness, humidity, absolute amount or amount of change in the amount of ultraviolet rays. It is also possible to learn determination thresholds based on each piece of environmental information. If it is determined from the camera detection information after activation based on environmental information that it is not the activation factor, the parameters of each determination threshold value are set to make it difficult to detect an environmental change.

Furthermore, each of the above parameters changes depending on the remaining capacity of the battery. For example, when the remaining battery power is low, it is difficult to make various judgments, and when the remaining battery power is large, it is easy to make various judgments. Specifically, even if the shaking state detection result or the sound scene detection result is not a factor that the user definitely wants to start the camera, if the battery level is high, it may be determined that the camera should be started.

Further, the determination of the low power consumption mode release condition can also be performed based on a neural network based on information on shaking detection, sound detection, time elapsed detection, various environmental information, remaining battery level, etc. In that case, learning is performed by selecting images to be trained using a method described later and changing the weights of the neural network based on learning information included in the images.

Next, learning of transition conditions to a low power consumption state will be explained. As shown in FIG. 7, in the mode setting determination in step S704, if it is determined that the mode is not one of "automatic shooting mode," "automatic editing mode," "automatic image transfer mode," "learning mode," and "automatic file deletion mode." , enter low power mode. The conditions for determining each mode are as described above, but the conditions for determining each mode also change through learning.

<Automatic shooting mode>
As described above, the importance level of each area is determined and automatic photography is performed while searching for a subject using panning and tilting, but if it is determined that there is no subject to be photographed, the automatic photography mode is canceled. For example, when the importance level of all areas or the sum of the importance levels of each area becomes less than or equal to a predetermined threshold, the automatic shooting mode is canceled. At this time, the predetermined threshold value is also lowered depending on the elapsed time after transitioning to the automatic shooting mode. The longer the time elapsed after transitioning to automatic shooting mode, the easier the transition to low power consumption mode becomes.

Further, by changing the predetermined threshold value depending on the remaining capacity of the battery, it is possible to perform low power consumption mode control that takes battery life into consideration. For example, when the battery level is low, the threshold value is increased to make it easier to shift to the low power consumption mode, and when the battery level is high, the threshold value is decreased to make it difficult to transition to the low power consumption mode. Here, a parameter (elapsed time threshold TimeC) for the next low power consumption mode release condition is set for the second control unit 211 (SubCPU) based on the elapsed time and the number of shots since the previous transition to automatic shooting mode. do. Each of the above threshold values changes by learning. Learning is performed, for example, by manually setting the shooting frequency, activation frequency, etc. through communication using a dedicated application of the external device 301.

Alternatively, the average value of the elapsed time from when the power button of the camera 101 is turned on until the power button is turned off may be accumulated, and distribution data for each time period may be accumulated, and each parameter may be learned. In that case, the time interval for returning from low power consumption mode or transitioning to a low power consumption state will be shortened for users who take a short time from power on to power off, and for users who take a long time from power on to power off. The interval is learned to be longer for .

It is also learned from detected information during the search. While it is determined that there are many important subjects set by learning, the time interval for returning from low power consumption mode or transitioning to low power consumption state will be shortened, and while there are few important subjects, the time interval will be shortened. , the interval is learned to be longer.

<Image shake correction>
Learning for image blur correction will be explained. Image blur correction is performed by calculating a correction amount in step S902 of FIG. 9, and driving pan/tilt in step S905 based on the correction amount. In image blur correction, learning is performed to perform corrections tailored to the user's shaking characteristics. For example, by using PSF (Point Spread Function) for a photographed image, it is possible to estimate the direction and magnitude of blur. In the learning information generation in step S912 in FIG. 9, the estimated direction and magnitude of blur are added to the image as information.

In the learning mode process in step S716 in FIG. ) movement information and vibration information (gyro output, acceleration output, camera status) as input to learn the weights of the neural network for image stabilization.In addition, environmental information (temperature, atmospheric pressure, illuminance, humidity), Sound information (sound scene determination, specific sound detection, sound level change), time information (elapsed time since startup, elapsed time since last shooting), location information (GPS position information, amount of change in position movement), etc. can also be input. You may also make a determination.

When calculating the amount of image blur correction in step S902, by inputting each of the above detection information to the neural network, it is possible to estimate the magnitude of blur at the time of photographing at that moment. When the estimated magnitude of blur is large, control such as increasing the shutter speed becomes possible. Furthermore, if the estimated magnitude of blur is large, the resulting image will be blurred, so a method such as prohibiting photography may be taken.

Additionally, since there is a limit to the pan/tilt drive angle, once the drive end is reached, no further correction can be made. It is possible to estimate the range required for pan/tilt driving to correct blur. If there is not enough room in the movable range during exposure, large blur can be suppressed by increasing the cutoff frequency of the filter that calculates the amount of image blur correction so that it does not exceed the movable range. In addition, if the movable range is likely to be exceeded, just before exposure, rotate the pan/tilt angle in the opposite direction to the direction in which the movable range is likely to be exceeded, and then start exposure to ensure the movable range and prevent blurring. You can also take pictures without. This allows image blur correction to be learned in accordance with the user's shooting characteristics and usage, thereby preventing blur in the shot image.

Furthermore, in the above-described <determination of photographing method>, it may be determined whether or not to perform panning photography, in which a moving subject is not blurred and a non-moving background appears flowing. In that case, the pan/tilt drive speed for photographing the subject without blur may be estimated from the detection information before photographing, and subject blur correction may be performed. At this time, the driving speed can be estimated by inputting each of the above detection information to a neural network that has already been trained. Learning is performed based on the information obtained by dividing the image into blocks and estimating the PSF of each block to estimate the direction and magnitude of blur in the block where the main subject is located.

Additionally, the amount of background flow can be learned from information on the image selected by the user. In that case, it is possible to estimate the magnitude of blur in blocks where the main subject is not located, and learn the user's preferences based on that information. By setting the shutter speed during photographing based on the learned preferred background panning amount, it is possible to automatically perform photographing that provides a panning effect that suits the user's preference.

Next, the learning method will be explained. Learning methods include ``in-camera learning'' and ``learning by linking with communication devices.''

The in-camera learning method will be explained below. In-camera learning in this embodiment includes the following methods.
(1) Learning using detection information during manual shooting (2) Learning using detection information during subject search <Learning using detection information during manual shooting>
As described in steps S907 to S913 in FIG. 9, in this embodiment, the camera 101 can perform two types of photography: manual photography and automatic photography. If there is a manual photographing instruction in step S907, information indicating that the photographed image is a manually photographed image is added in step S912. Furthermore, if it is determined in step S909 that automatic photography is ON and the image is captured, information is added to the captured image in step S912 to indicate that the captured image is an automatically captured image.

Here, in the case of manual photographing, there is a very high possibility that the photograph was taken based on the user's favorite subject, favorite scene, favorite place, or time interval. Therefore, learning is performed based on each feature data obtained during manual photography and the learning information of the photographed image. It also learns about extracting features from captured images, registering personal authentication, registering facial expressions for each individual, and registering combinations of people from the information detected during manual photography. Also, from the detection information during the subject search, for example, learning is performed to change the importance of nearby people and objects based on the facial expressions of the personally registered subjects.

<Learning based on detection information during object search>
During the object search operation, it is determined what kind of person, object, or scene the object registered for personal authentication is photographed at the same time, and at the same time, the ratio of time during which the object is photographed within the angle of view is calculated. For example, the time ratio in which person A, who is a subject registered for personal authentication, is photographed at the same time as person B, who is a subject registered for personal authentication, is calculated. If person A and person B are within the angle of view, various detection information is saved as learning data and learned in learning mode processing (step S716) so that the automatic shooting determination score is high.

In another example, the time ratio in which person A, who is a subject registered for personal authentication, is photographed at the same time as a subject "cat" determined by general object recognition is calculated. If the person A and the "cat" are within the angle of view, various detection information is stored as learning data and learned in learning mode processing (step S716) so that the automatic shooting determination score is high.

Furthermore, when a high degree of smile of Person A, who is a registered subject for personal authentication, is detected, or when an expression such as "joy" or "surprise" is detected, the subjects photographed at the same time are learned to be important. Alternatively, when facial expressions such as "angry" or "serious face" are detected, processing is performed such as not learning the subjects that are photographed at the same time because they are unlikely to be important.

Next, learning by cooperation with an external device in this embodiment will be explained. In this embodiment, there are the following methods for learning through cooperation with an external device.
(1) Learning by acquiring images with an external device (2) Learning by inputting judgment values into images via an external device (3) Learning by analyzing images stored in an external device ( 4) Learning from information uploaded to the SNS server with an external device (5) Learning by changing camera parameters with an external device (6) Learning from information whose images were manually edited with an external device <With an external device Learning by acquiring images＞
As explained in FIG. 3, the camera 101 and the external device 301 have communication means for performing first and second communications 302 and 303. Images are sent and received mainly through the first communication 302, and images in the camera 101 can be sent to the external device 301 via a dedicated application in the external device 301. Further, thumbnail images of image data stored in the camera 101 can be viewed using a dedicated application in the external device 301. The user can select an image he/she likes from among the thumbnail images, confirm the image, and send the image to the external device 301 by operating an image acquisition instruction.

At this time, since the user selects and acquires the image, there is a very high possibility that the acquired image is an image of the user's preference. Therefore, by determining that the acquired image is an image to be studied and learning based on the learning information of the acquired image, various types of learning according to the user's preference can be performed.

Here, an example of operation will be explained. FIG. 12 shows an example in which images in the camera 101 are viewed using a dedicated application of the external device 301. Thumbnail images (1604 to 1609) of image data stored in the camera are displayed on the display unit 407, and the user can select and obtain an image that he or she likes. At this time, buttons 1601, 1602, and 1603 forming a display method changing section for changing the display method are provided.

When the button 1601 is pressed, the display mode is changed to date and time priority display mode, and images are displayed on the display unit 407 in the order of the shooting date and time of the images in the camera 101. For example, an image with a newer date and time is displayed at a position indicated by 1604, and an image with an older date and time is displayed at a position indicated by 1609.

When button 1602 is pressed, the mode is changed to recommended image priority display mode. Based on the score calculated in step S912 of FIG. 9 that determines the user's preference for each image, the images in the camera 101 are displayed on the display unit 407 in the order of the highest score. For example, an image with a high score is displayed at a position indicated by 1604, and an image with a low score is displayed at a position indicated by 1609.

By pressing the button 1603, a person or object can be specified, and by subsequently specifying a specific person or object, only the specific object can be displayed. The settings of buttons 1601 to 1603 can also be turned on at the same time. For example, if all settings are ON, only the specified subject will be displayed, images with newer shooting dates and times will be given priority, and images with higher scores will be given priority and displayed. In this way, since the user's preferences are learned for the captured images, it is possible to extract only the user's preferred images from a large number of captured images with a simple confirmation process.

<Learning by inputting judgment values into images via an external device>
As explained above, the camera 101 and the external device 301 have communication means, and images stored in the camera 101 can be viewed using a dedicated application in the external device 301. Here, the user may assign a score to each image. Users can give high scores (for example, 5 points) to images that they like, and low scores (for example, 1 point) to images that they do not like. Create a structure that allows you to learn. The score for each image is used for relearning together with learning information within the camera. The neural network inputs the feature data from the specified image information and learns so that the output of the neural network approaches the score specified by the user.

In this embodiment, the user inputs the determination value into the photographed image via the external device 301, but the user may input the determination value directly into the image by operating the camera 101. In that case, for example, the camera 101 is provided with a touch panel display, and the user presses a GUI button displayed on the screen display section of the touch panel display to set a mode for displaying captured images. Then, the user can perform similar learning by inputting judgment values for each image while checking the captured images.

<Learning by analyzing images stored in an external device>
The external device 301 has a storage unit 404, and is configured to record images other than images captured by the camera 101 in the storage unit 404. At this time, the images stored in the external device 301 are easy for the user to view and can be easily uploaded to the shared server via the public wireless control unit 406, so there are many images that the user likes. Very likely to be included.

The control unit 411 of the external device 301 is configured to be able to process images stored in the storage unit 404 with the same ability as the learning processing unit 219 in the camera 101 using a dedicated application. Learning is then performed by communicating the processed learning data to the camera 101. Alternatively, a configuration may be adopted in which images and data to be learned by the camera 101 are transmitted and the learning is performed within the camera 101. Alternatively, a configuration may be provided in which the user selects and learns an image that the user wants to learn from among the images stored in the recording unit 404 using a dedicated application.

<Learning from information uploaded to SNS server by external device>
Next, we will explain how to use information in social networking services (SNS), which are services and websites that allow you to build social networks that focus on connections between people, for learning. When uploading an image to SNS, there is a technique in which a tag related to the image is input from the external device 301 and then transmitted along with the image. There is also a technique for inputting likes and dislikes for images uploaded by other users, and it can be determined whether the images uploaded by other users are the photos that the user who owns the external device 301 likes.

With a dedicated SNS application downloaded into the external device 301, it is possible to acquire images and information about the images that the user himself has uploaded as described above. Further, by inputting whether or not the user likes images uploaded by other users, the user's favorite images and tag information can be acquired. The images and tag information are analyzed and learned within the camera 101.

The control unit 411 of the external device 301 is configured to be able to acquire images uploaded by the user and images determined to be liked by the user as described above, and process them with the same ability as the learning processing unit 219 in the camera 101. . Learning is then performed by communicating the processed learning data to the camera 101. Alternatively, a configuration may be adopted in which images to be learned by the camera 101 are transmitted and the learning is performed within the camera 101.

Further, from the subject information set in the tag information (for example, subject object information such as a dog or cat, scene information such as a beach, facial expression information such as a smiley face, etc.), subject information that the user is likely to like is estimated. Then, learning is performed by registering the object to be detected and input into the neural network.

Furthermore, it is also possible to estimate image information that is currently popular in the world from statistical values of tag information (image filter information and subject information) in the SNS, and to learn it within the camera 101.

<Learning by changing camera parameters with an external device>
As explained above, the camera 101 and the external device 301 have communication means. Then, the learning parameters currently set in the camera 101 (such as the weight of the neural network and the selection of subjects to be input to the neural network) can be communicated to the external device 301 and stored in the storage unit 404 of the external device 301. can. Furthermore, using a dedicated application in the external device 301, learning parameters set in a dedicated server can be obtained via the public radio control unit 406 and set as learning parameters in the camera 101. Thereby, by saving the parameters at a certain point in the external device 301 and setting them in the camera 101, it is also possible to restore the learning parameters. Further, learning parameters held by other users can be acquired via a dedicated server and set in the user's own camera 101.

Further, a dedicated application of the external device 301 may be used to allow the user to register voice commands, authentication registration, and gestures, or to register important locations. These pieces of information are treated as input data for the shooting trigger and automatic shooting determination described in the automatic shooting mode process (FIG. 9). In addition, a configuration may be adopted in which the shooting frequency, activation interval, ratio of still images to moving images, favorite images, etc. can be set, and settings such as the activation interval described in <Low power consumption mode control> may be made.

<Learning from information on images manually edited with an external device>
It is also possible to provide a dedicated application of the external device 301 with a function that allows manual editing by user operation, and feed back the contents of the editing work to the learning. For example, it is possible to edit image effects (trimming processing, rotation processing, slide, zoom, fade, color conversion filter effect, time, still image/video ratio, BGM). Then, the automatic editing neural network is trained to determine whether to apply a manually edited image effect to the learning information of the image.

Next, the learning processing sequence will be explained. In the mode setting determination in step S704 in FIG. 7, it is determined whether learning processing should be performed, and if it is determined that learning processing should be performed, learning mode processing in step S716 is performed.

The learning mode determination conditions will be explained. Whether or not to shift to the learning mode is determined based on the elapsed time since the last learning process, the number of information that can be used for learning, whether a learning process instruction has been received via a communication device, etc. FIG. 13 shows a process flow for determining whether or not to shift to learning mode, which is determined in the mode setting determination process of step S704.

When the learning mode determination is instructed to start in the mode setting determination process of step S704, the process of FIG. 13 starts. In step S1401, it is determined whether there is a registration instruction from the external device 301. Registration here includes the above-mentioned <Learning by acquiring images with an external device>, <Learning by inputting judgment values into images via an external device>, and <Learning by storing images in an external device>. This is a determination as to whether or not there is a registration instruction for learning, such as "Learning by analyzing images that are currently available."

In step S1401, if there is a registration instruction from the external device 301, the process advances to step S1408, sets the learning mode determination to TRUE, sets to perform the process of step S716, and ends the learning mode determination process. If there is no registration instruction from the external device in step S1401, the process advances to step S1402.

In step S1402, it is determined whether there is a learning instruction from an external device. The learning instruction here is a determination as to whether or not there is an instruction to set learning parameters, such as <learning by changing camera parameters with an external device>. In step S1402, if there is a learning instruction from an external device, the process advances to step S1408, sets the learning mode determination to TRUE, sets the process of step S716 to be performed, and ends the learning mode determination process. If there is no learning instruction from the external device in step S1402, the process advances to step S1403.

In step S1403, the elapsed time TimeN since the previous learning process (recalculation of neural network weights) was performed is obtained, and the process advances to step S1404. In step S1404, the new number of data to be learned DN (the number of images designated to be learned during the elapsed time TimeN since the previous learning process was performed) is acquired, and the process advances to step S1405. In step S1405, a threshold value DT for determining whether to enter the learning mode is calculated from the elapsed time TimeN. It is set that the smaller the value of the threshold DT, the easier it is to enter the learning mode. For example, DTa, which is the value of the threshold DT when TimeN is smaller than a predetermined value, is set larger than DTb, which is the value of the threshold DT when TimeN is larger than the predetermined value. is set to be small. As a result, even when there is little learning data, if a long time has elapsed, it is easier to enter the learning mode, and by learning again, the camera can easily change the learning mode according to the usage time.

After calculating the threshold value DT in step S1405, the process advances to step S1406, and it is determined whether the number of data to be learned DN is larger than the threshold value DT. If the data number DN is larger than the threshold DT, the process advances to step S1407 and DN is set to 0. Thereafter, the process advances to step S1408, where the learning mode determination is set to TRUE, the process of step S716 (FIG. 7) is set to be performed, and the learning mode determination process is ended.

If DN is equal to or less than the threshold value DT in step S1406, the process advances to step S1409. Since there is no registration instruction from the external device or learning instruction from the external device, and the number of learning data is less than the predetermined value, the learning mode determination is set to FALSE, the process of step S716 is set not to be performed, and the learning mode is set. The determination process ends.

Next, processing in the learning mode processing (step S716) will be explained. A detailed flowchart showing the operation of the learning mode process is shown in FIG.

When the learning mode is determined in step S715 in FIG. 7 and the process proceeds to step S716, the process in FIG. 14 starts. In step S1501, it is determined whether there is a registration instruction from the external device 301. If there is a registration instruction from the external device 301 in step S1501, the process advances to step S1502. In step S1502, various registration processes are performed.

The various registrations are the registration of features to be input to the neural network, such as registration of face authentication, registration of general object recognition, registration of sound information, and registration of location information. When the registration process is completed, the process advances to step S1503, and the elements to be input to the neural network are changed from the information registered in step S1502. After completing the process in step S1503, the process advances to step S1507.

If there is no registration instruction from the external device 301 in step S1501, the process advances to step S1504, and it is determined whether there is a learning instruction from the external device 301. If there is a learning instruction from the external device 301, the process advances to step S1505, where the learning parameters communicated from the external device 301 are set in each determiner (neural network weights, etc.), and the process advances to step S1507.

If there is no learning instruction from the external device 301 in step S1504, learning (recalculation of neural network weights) is performed in step S1506. As explained using FIG. 13, the process of step S1506 is entered when the number of data to be learned DN exceeds the threshold value DT and re-learning of each determiner is performed. Relearning is performed using a method such as error backpropagation or gradient descent, the weights of the neural network are recalculated, and the parameters of each determiner are changed. Once the learning parameters are set, the process advances to step S1507.

In step S1507, the images in the file are rescored. In this embodiment, scores are assigned to all captured images stored in a file (recording medium 221) based on learning results, and automatic editing and automatic file deletion are performed according to the assigned scores. The structure is as follows. Therefore, when re-learning or setting learning parameters from an external device is performed, it is necessary to update the scores of captured images as well. Therefore, in step S1507, recalculation is performed to give a new score to the photographed image stored in the file, and when the process is completed, the learning mode process is ended.

In this embodiment, we have described a method for proposing videos that the user likes by extracting scenes that the user seems to like, learning their characteristics, and reflecting them in camera operations such as automatic shooting and automatic editing. However, the invention is not limited to this application. For example, it can also be used to propose a video that is different from the user's own preference. An example of how to achieve this is as follows.

<Method using a neural network that has learned preferences>
Regarding learning, the user's preferences are learned using the method described above. Then, in S908 of <Automatic Photography>, automatic photographing is performed when the output value of the neural network is a value indicating that it is different from the user's preference, which is teacher data. For example, if an image that the user likes is used as a teacher image, and learning is performed so that a high value is output when it shows features similar to the teacher image, conversely, if the output value is lower than a predetermined value, Perform automatic shooting. Similarly, in the subject search process and the automatic editing process, a process is performed in which the output value of the neural network becomes a value indicating that it is different from the user's preference, which is teacher data.

<Method using a neural network trained on situations different from one's preferences>
In this method, a situation different from the user's preference is learned as training data at the time of learning processing. For example, in the above description, a learning method using manually captured images as training data has been described, assuming that the manually captured images are scenes that the user prefers to capture. However, here, on the contrary, manually photographed images are not used as training data, and scenes in which manual photography has not been performed for a predetermined period of time or more are added as training data. Alternatively, if there is a scene in the training data that has similar characteristics to the manually captured image, it may be deleted from the training data. Further, an image having different characteristics from an image acquired by an external device may be added to the teacher data, or an image having characteristics similar to the acquired image may be deleted from the teacher data. By doing this, data that differs from the user's preferences is collected in the teacher data, and as a result of learning, the neural network becomes able to discriminate situations that differ from the user's preferences. Then, in automatic photography, by performing photography according to the output value of the neural network, it is possible to photograph a scene different from the user's preference.

As described above, by intentionally proposing a video that is different from the user's own preference, scenes that the user would not have manually photographed can be photographed, and the number of missed shots can be reduced. In addition, by proposing shooting scenes that are not in the user's imagination, it can be expected to make the user aware of the situation and expand the range of tastes.

Furthermore, by combining the above methods, it is possible to propose situations that are somewhat similar to the user's preferences but are partially different, and it is also easy to adjust the degree of adaptation to the user's preferences. The degree of adaptation to the user's preferences may be changed depending on the mode setting, the status of various sensors, and the status of detected information.

In this embodiment, a configuration for learning within the camera 101 has been described. However, the same learning effect can be achieved with a configuration in which the external device 301 side has a learning function, data necessary for learning is communicated to the external device 301, and learning is executed only on the external device side. In that case, as explained in <Learning by changing camera parameters with an external device> above, learning is performed by setting parameters such as neural network weights learned on the external device side to the camera 101 via communication. It may be configured.

Furthermore, both the camera 101 and the external device 301 are configured to have learning functions, so that, for example, learning information held by the external device 301 is transferred to the camera 101 at the timing when learning mode processing (step S716) is performed in the camera 101. The learning may be performed by communicating with the learning parameters and merging the learning parameters.

(Other embodiments)
The present invention also provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device reads the program. This can also be achieved by executing a process. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

Examples of embodiments of the invention are listed below.

(Embodiment 1) Imaging means that images a subject image and outputs image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information regarding the frequency of the photographing operation;
Equipped with
The imaging apparatus is characterized in that the control means changes a threshold value for determining whether or not to perform the imaging operation, depending on the information regarding the frequency.

(Implementation mode 2)
Further comprising a detection means for detecting information about the subject,
The imaging apparatus according to embodiment 1, wherein the control means determines whether to perform the photographing operation by comparing information about the subject with the threshold value.

(Implementation mode 3)
Embodiment 2 The imaging device according to embodiment 2, wherein the detection unit detects information about the subject based on at least one of detected sound and image data captured by the imaging unit.

(Implementation mode 4)
4. The imaging device according to any one of embodiments 1 to 3, wherein the initial value of the threshold is determined based on past learning results.

(Implementation mode 5)
5. The imaging apparatus according to any one of embodiments 1 to 4, wherein the information regarding the frequency of the photographing operation is the number of images taken per fixed period.

(Embodiment 6)
Embodiment 5 The imaging apparatus according to embodiment 5, wherein the control means determines the threshold value for the next fixed period based on the number of images taken in the past.

(Embodiment 7)
Further comprising determining means for determining a target number of shots in a predetermined period,
Any one of embodiments 1 to 6, wherein the control means changes a threshold value for determining whether or not to perform the photographing operation based on the target number of photographed images and information regarding the frequency. The imaging device according to item 1.

(Embodiment 8)
Embodiment 7 The imaging apparatus according to embodiment 7, wherein the control means changes the threshold value so that the number of shots increases linearly toward the target number of shots as the shooting time elapses.

(Implementation mode 9)
9. The imaging apparatus according to embodiment 7 or 8, wherein the determining unit determines the target number of images to be taken based on photographing conditions set based on manual input or voice input by a user.

(Embodiment 10)
10. The imaging apparatus according to embodiment 9, wherein the manual input or voice input by the user is performed using a smart device.

(Embodiment 11)
The imaging apparatus according to embodiment 9 or 10, wherein the imaging conditions include information on a total imaging time.

(Embodiment 12)
12. The imaging apparatus according to embodiment 11, wherein the photographing conditions further include information on a recording medium and remaining battery power.

(Embodiment 13)
an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
a detection means for detecting a face of a subject;
determination means for determining the state of the face of the subject detected by the detection means;
acquisition means for acquiring information regarding the frequency of the photographing operation;
Equipped with
The control means performs a photographing operation when the frequency is a first frequency, and performs a photographing operation when the frequency is a second frequency even if the facial condition of the subject determined by the determination means is the same. An imaging device characterized in that the imaging device is controlled so as not to perform a photographing operation.

(Embodiment 14)
An embodiment characterized in that the state of the subject's face includes the facial expression of the subject, the direction of the subject's face, the degree to which the subject's eyes are opened, the subject's line of sight, the subject's posture, and the state of the subject's movements. 14. The imaging device according to 13.

(Embodiment 15)
Further comprising changing means for changing the orientation of the imaging means in order to direct the imaging means toward the subject,
15. The imaging device according to any one of embodiments 1 to 14, wherein the changing unit changes a movable range in which the orientation of the imaging unit is changed depending on the frequency.

(Embodiment 16)
16. The imaging device according to Embodiment 15, wherein the changing unit rotates the imaging unit in a panning direction and a tilting direction.

(Embodiment 17)
further comprising a zoom means for enlarging or reducing the subject image on the imaging means,
17. The imaging device according to any one of embodiments 1 to 16, wherein the zoom means changes the enlargement or reduction control according to the frequency.

(Embodiment 18)
A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information regarding the frequency of the photographing operation,
A method for controlling an imaging apparatus, wherein in the control step, a threshold value for determining whether to perform the photographing operation is changed in accordance with information regarding the frequency.

(Embodiment 19)
A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a detection step of detecting the face of the subject;
a determination step of determining the state of the face of the subject detected in the detection step;
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information regarding the frequency of the photographing operation,
In the control step, even if the state of the face of the subject determined in the determination step is the same, the photographing operation is performed when the frequency is a first frequency, and when the frequency is a second frequency, the photographing operation is performed. 1. A method of controlling an imaging device, comprising controlling the imaging device so as not to perform a photographing operation.

(Embodiment 20)
A program for causing a computer to execute each step of the control method described in Embodiment 18 or 19.

(Embodiment 21)
A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to embodiment 18 or 19.

101: Camera, 301: Smart device, 501: Wearable device, 104: Tilt rotation unit, 105: Pan rotation unit

Claims (28)

an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information indicating the cumulative number of shots taken since the first time;
determining means for determining a target number of shots in a predetermined period;
Equipped with
The imaging apparatus is characterized in that the control means changes a threshold value for determining whether or not to perform the photographing operation based on the target number of photographed images and information indicating the cumulative number of photographed images. an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information indicating the cumulative number of shots taken since the first time;
a communication unit that communicates with an external device ;
Equipped with
The control means changes a threshold value for determining whether to perform the photographing operation based on communication performance of the communication unit with the external device and information indicating the cumulative number of photographed images. An imaging device that uses an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information indicating the cumulative number of shots taken since the first time;
Equipped with
The acquisition means acquires information indicating the cumulative number of images taken until a certain time elapses from the first time,
The control means includes a threshold value for determining whether to perform the photographing operation until a certain time period elapses from the first time, and a threshold value for determining whether or not to perform the photographing operation until a certain period of time elapses from the first time point. The imaging apparatus sets the threshold value to be used after the predetermined period of time has elapsed from the first time based on information indicating the cumulative number of images taken during the period of time. Further comprising a detection means for detecting information about the subject,
The imaging device according to any one of claims 1 to 3, wherein the control means determines whether or not to perform the photographing operation by comparing information about the subject with the threshold value. . The imaging device according to claim 4 , wherein the detection means detects information about the subject based on at least one of detected sound and image data captured by the imaging means. The imaging device according to any one of claims 1 to 5 , wherein the initial value of the threshold value is determined based on a past learning result. 7. The control means determines the threshold value for the next certain period based on information indicating the cumulative number of images taken since the first time. imaging device. Further comprising determining means for determining a target number of shots in a predetermined period,
The control means changes a threshold value for determining whether or not to perform the photographing operation based on the target number of photographed images and information indicating the cumulative number of photographed images from the first time. An imaging device according to any one of claims 1 to 7 . 9. The imaging apparatus according to claim 8 , wherein the control means changes the threshold value so that the number of shots increases linearly toward the target number of shots as the shooting time elapses. 10. The imaging apparatus according to claim 8 , wherein the determining unit determines the target number of images to be photographed based on photographing conditions set based on manual input or voice input by a user. The imaging apparatus according to claim 10 , wherein the manual input or voice input by the user is performed using a smart device. The imaging apparatus according to claim 10 or 11 , wherein the imaging conditions include information on a total imaging time. 13. The imaging apparatus according to claim 12 , wherein the photographing conditions further include information on a recording medium and remaining battery power. a detection means for detecting a face of a subject;
determination means for determining the state of the face of the subject detected by the detection means;
further comprising;
The control means performs a photographing operation if the cumulative number of photographed images from the first time is a first number even if the face condition of the subject determined by the determination means is the same; The imaging apparatus according to any one of claims 1 to 13 , wherein control is performed so that the photographing operation is not performed when the cumulative number of photographed images is a second number that is greater than the first number. A claim characterized in that the state of the subject's face is the expression of the subject's face, the direction of the subject's face, the degree to which the subject's eyes are opened, the subject's line of sight, the subject's posture, and the state of the subject's movements. 15. The imaging device according to 14 . Further comprising changing means for changing the orientation of the imaging means in order to direct the imaging means toward the subject,
4. The changing means changes the movable range for changing the orientation of the imaging means based on information indicating the cumulative number of images taken since the first time . The imaging device described in section. 17. The imaging apparatus according to claim 16 , wherein the changing means rotates the imaging means in a panning direction and a tilting direction. further comprising a zoom means for enlarging or reducing the subject image on the imaging means,
4. The imaging apparatus according to claim 1, wherein the zoom means changes the enlargement or reduction control based on information indicating the cumulative number of captured images. The acquisition means acquires information indicating the cumulative number of images taken until a certain time elapses from the first time,
3. The control means changes the threshold value based on information indicating the cumulative number of images taken until a certain period of time has elapsed from the first time. Imaging device. an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information regarding the frequency of the photographing operation;
a changing means for changing the direction of the imaging means in order to direct the direction of the imaging means toward the subject;
The control means changes a threshold value for determining whether to perform the photographing operation according to the information regarding the frequency,
The imaging device is characterized in that the changing unit changes a movable range in which the orientation of the imaging unit is changed depending on the frequency. an imaging means for imaging a subject image and outputting image data;
a control means for controlling whether or not to perform a photographing operation for recording image data outputted by the imaging means;
acquisition means for acquiring information regarding the frequency of the photographing operation;
a zoom means for enlarging or reducing the subject image on the imaging means;
The control means changes a threshold value for determining whether to perform the photographing operation according to the information regarding the frequency,
The image pickup apparatus is characterized in that the zoom means changes the control of enlargement or reduction according to the frequency. A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information indicating the cumulative number of shots taken from the first time;
a determination step of determining a target number of shots in a predetermined period;
has
In the control step, a threshold value for determining whether to perform the photographing operation is changed based on information indicating the target number of photographed images and the cumulative number of photographed images. . A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information regarding the frequency of the photographing operation;
a changing step of changing the orientation of the imaging means in order to direct the imaging means toward the subject;
In the control step, a threshold value for determining whether to perform the photographing operation is changed according to the information regarding the frequency;
A method for controlling an imaging device, characterized in that, in the changing step, a movable range in which the direction of the imaging means is changed is changed according to the frequency. A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information regarding the frequency of the photographing operation;
a zoom step for enlarging or reducing the subject image on the imaging means,
In the control step, a threshold value for determining whether to perform the photographing operation is changed according to the information regarding the frequency;
A method for controlling an imaging apparatus, characterized in that in the zooming step, the enlargement or reduction control is changed depending on the frequency. A method for controlling an imaging device comprising an imaging means for imaging a subject image and outputting image data, and a communication unit for communicating with an external device, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information indicating the cumulative number of shots taken from the first time;
has
In the control step, a threshold value for determining whether to perform the photographing operation is changed based on communication performance with the external device in the communication unit and information indicating the cumulative number of photographed images. Features: A method for controlling an imaging device. A method for controlling an imaging device including an imaging means for imaging a subject image and outputting image data, the method comprising:
a control step for controlling whether or not to perform a photographing operation for recording image data output by the imaging means;
an acquisition step of acquiring information indicating the cumulative number of shots taken from the first time;
has
In the acquisition step, information indicating the cumulative number of images taken until a certain time elapses from the first time is acquired;
In the control step, a threshold value for determining whether to perform the photographing operation until a certain time elapses from the first time, and a threshold value for determining whether or not to perform the photographing operation until a certain time elapses from the first time. A method for controlling an imaging apparatus, characterized in that the threshold value to be used after the certain period of time has elapsed from the first time is set based on information indicating the cumulative number of images taken during a period of time. A program for causing a computer to execute each step of the control method according to any one of claims 22 to 26 . A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to any one of claims 22 to 26 .

Images