JP2021111960A - Imaging apparatus, control method of the same, and program - Google Patents

Abstract

To solve the problem in which there is a possibility of missing a shot since automatic shooting means provided by an imaging apparatus alone cannot take a picture at a timing desired by a user.SOLUTION: The imaging apparatus includes sound-collecting means for collecting sound, analysis means for analyzing the sound collected by the sound collecting means, automatic shooting means for automatically shooting, and setting means for setting a shooting frequency of the automatic shooting means. If a result of the analysis by the analysis means is a specific voice instruction, the imaging apparatus sets the shooting frequency higher by the setting means after the operation according to the instruction.SELECTED DRAWING: Figure 5

Description

The present invention relates to an imaging device capable of receiving instructions using voice.

In recent years, a life log camera that automatically repeats shooting on a regular basis and an imaging device that automatically shoots by judging the shooting status by the camera itself have been proposed. The purpose of these devices is to automatically shoot an image of a scene desired by the user without the user being aware of it.

JP-A-2019-110525

In the device described in Patent Document 1, the shooting timing is determined from the information obtained by detecting the face of the subject, the number of shots taken in the past, the target number of shots, and the like, and shooting is automatically performed. There is.

However, since it is automatic, it does not always reflect the user's intention. Therefore, it is not possible to take a picture at the timing desired by the user by itself, and there is a possibility that a picture may be missed. Therefore, an object of the present invention is to reduce the occurrence of missed shots.

In order to achieve the above object, the imaging device of the present invention includes a sound collecting means for collecting sound, an analysis means for analyzing the sound collected by the sound collecting means, and an automatic photographing means for automatically taking a picture. And a setting means for setting the shooting frequency of the automatic shooting means, and if the result of analysis by the analysis means is a specific voice instruction, the setting is performed after performing an operation according to the instruction. It is characterized in that the shooting frequency is set higher by means.

According to the present invention, it is possible to reduce the occurrence of missed shots.

(A) It is a figure for demonstrating an example of the appearance of an image pickup apparatus. (B) It is a figure for demonstrating operation of an image pickup apparatus. It is a figure which shows the structure of the image pickup apparatus. It is a figure which shows the structure of the image pickup apparatus and an external device. It is a figure which shows the structure of an external device. It is a flowchart explaining the automatic shooting process. It is a flowchart explaining the voice recognition process. It is a flowchart explaining the frequency setting process. It is a figure for demonstrating the area division in a photographed image. It is a figure which shows an example of the screen displayed on an external device.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.

The embodiment described below is an example as a means for realizing the present invention, and may be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. Moreover, each embodiment can be combined as appropriate.

<Configuration of imaging device>
FIG. 1 is a diagram schematically showing an imaging device according to the first embodiment.

The image pickup device 101 shown in FIG. 1A is provided with an operation member capable of operating the power switch (hereinafter, referred to as a power button, but may be an operation such as tapping, flicking, or swiping on the touch panel). There is. The lens barrel 102, which is a housing including a group of photographing lenses for imaging and an image sensor, is attached to the image pickup device 101 and is provided with a rotation mechanism capable of rotationally driving the lens barrel 102 with respect to the fixed portion 103. The tilt rotation unit 104 is a motor drive mechanism capable of rotating the lens barrel 102 in the pitch direction shown in FIG. 1 (b), and the pan rotation unit 105 is a motor drive mechanism capable of rotating the lens barrel 102 in the yaw direction. Therefore, the lens barrel 102 can rotate in one or more axes. Note that FIG. 1B is an axis definition at the fixed portion 103 position. Both the angular velocity meter 106 and the accelerometer 107 are mounted on the fixed portion 103 of the image pickup apparatus 101. Then, the vibration of the image pickup apparatus 101 is detected based on the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit and the pan rotation unit are rotationally driven based on the detected shaking angles. As a result, the lens barrel 102, which is a movable portion, is configured to correct the runout and the tilt.

FIG. 2 is a block diagram showing the configuration of the image pickup apparatus of the present embodiment.

In FIG. 2, the first control unit 223 includes a processor (for example, CPU, GPU, microprocessor, MPU, etc.) and a memory (for example, DRAM, SRAM, etc.). These perform various processes to control each block of the image pickup apparatus 101, and control data transfer between each block. The non-volatile memory (EEPROM) 216 is a memory that can be electrically erased and recorded, and stores constants, programs, and the like for the operation of the first control unit 223.

In FIG. 2, the zoom unit 201 includes a zoom lens that performs scaling. The zoom drive control unit 202 drives and controls the zoom unit 201. The focus unit 203 includes a lens for adjusting the focus. The focus drive control unit 204 drives and controls the focus unit 203.

In the image pickup unit 206, the image pickup element receives the light incident through each lens group, and outputs the electric charge information corresponding to the amount of the light to the image processing unit 207 as analog image data. The image processing unit 207 applies image processing such as distortion correction, white balance adjustment, and color interpolation processing to the digital image data output by the A / D conversion, and outputs the applied digital image data. The digital image data output from the image processing unit 207 is converted into a recording format such as a JPEG format by the image recording unit 208, and transmitted to the memory 215 and the video output unit 217 described later.

The lens barrel rotation drive unit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

The device shake detection unit 209 is equipped with, for example, an angular velocity meter (gyro sensor) 106 that detects the angular velocity in the three-axis direction of the image pickup device 101, and an accelerometer (accelerometer) 107 that detects the acceleration in the three-axis direction of the device. .. The device shake detection unit 209 calculates the rotation angle of the device, the shift amount of the device, and the like based on the detected signal.

The voice input unit 213 acquires a voice signal collected from the periphery of the image pickup device 101 by using a microphone provided in the image pickup device 101, performs analog-to-digital conversion, and transmits the voice signal to the voice processing unit 214. The voice processing unit 214 performs voice-related processing such as optimization processing of the input digital voice signal. Then, the voice signal processed by the voice processing unit 214 is transmitted to the memory 215 by the first control unit 223. The memory 215 temporarily stores the image signal and the audio signal obtained by the image processing unit 207 and the audio processing unit 214.

The image processing unit 207 and the audio processing unit 214 read out the image signal and the audio signal temporarily stored in the memory 215, encode the image signal, encode the audio signal, and the like, and perform the compressed image signal, the compressed audio signal, and the like. To generate. The first control unit 223 transmits these compressed image signals and compressed audio signals to the recording / reproducing unit 220.

The recording / reproducing unit 220 records the compressed image signal, the compressed audio signal, and other control data related to shooting on the recording medium 221 with the image processing unit 207 and the audio processing unit 214. When the audio signal is not compressed and encoded, the first control unit 223 transfers the audio signal generated by the audio processing unit 214 and the compressed image signal generated by the image processing unit 207 to the recording / playback unit 220. It is transmitted and recorded on the recording medium 221.

The recording medium 221 may be a recording medium built in the image pickup apparatus 101 or a removable recording medium. The recording medium 221 can record various data such as a compressed image signal, a compressed audio signal, and an audio signal generated by the image pickup apparatus 101, and a medium having a capacity larger than that of the non-volatile memory 216 is generally used. For example, the recording medium 221 includes all types of recording media such as hard disks, optical disks, magneto-optical disks, CD-Rs, DVD-Rs, magnetic tapes, non-volatile semiconductor memories, and flash memories.

The recording / reproducing unit 220 reads (reproduces) a compressed image signal, a compressed audio signal, an audio signal, various data, and a program recorded on the recording medium 221. Then, the first control unit 223 transmits the read compressed image signal and compressed audio signal to the image processing unit 207 and the audio processing unit 214. The image processing unit 207 and the audio processing unit 214 temporarily store the compressed image signal and the compressed audio signal in the memory 215, decode them according to a predetermined procedure, and transmit the decoded signals to the video output unit 217 and the audio output unit 218. do.

The voice input unit 213 has a plurality of microphones mounted on the image pickup device 101, and the voice processing unit 214 can detect the direction of sound on a plane on which the plurality of microphones are installed, and is used for search and automatic shooting described later. Used. Further, the voice processing unit 214 detects a specific voice command. The voice command may be configured so that the user can register a specific voice in the image pickup apparatus in addition to some commands registered in advance. It also recognizes sound scenes. In sound scene recognition, sound scene determination is performed using a trained model trained by machine learning based on a large amount of voice data in advance. Specific algorithms for machine learning include the nearest neighbor method, the naive Bayes method, a decision tree, and a support vector machine. In addition, deep learning (deep learning) in which features and coupling weighting coefficients for learning are generated by themselves using a neural network can also be mentioned. As appropriate, any of the above algorithms that can be used can be applied to this embodiment.

In the present embodiment, for example, a neural network for detecting a specific scene such as "cheering", "applause", or "speaking" is set in the voice processing unit 214. .. Then, when a specific sound scene or a specific voice command is detected, the detection trigger signal is output to the first control unit 223 and the second control unit 211.

That is, the neural network of the voice processing unit 214 prepares the voice information of the scenes of "cheering", "clapping", and "speaking" in advance, and inputs and detects the voice information. Learn the trigger signal as an output.

A second control unit 211, which is provided separately from the first control unit 223 that controls the entire main system of the image pickup apparatus 101, controls the power supply of the first control unit 223.

The first power supply unit 210 and the second power supply unit 212 supply power for operating the first control unit 223 and the second control unit 211, respectively. By pressing the power button provided on the image pickup apparatus 101, power is first supplied to both the first control unit 223 and the second control unit 211. As will be described later, the first control unit 223 is the first power supply. It is controlled to turn off its own power supply to the unit 210. Even while the first control unit 223 is not operating, the second control unit 211 is operating, and information from the device shake detection unit 209 and the voice processing unit 214 is input. The second control unit is configured to perform a determination process of whether or not to start the first control unit 223 based on various input information, and when the activation is determined, a power supply instruction is given to the first power supply unit. .. In this embodiment, the power supply unit supplies electric power from the battery. That is, the image pickup device 101 is also a mobile terminal.

The audio output unit 218 outputs a preset audio pattern from a speaker built in the image pickup apparatus 101, for example, at the time of shooting.

The LED control unit 224 controls a preset lighting / blinking pattern of the LED provided in the image pickup apparatus 101, for example, at the time of shooting.

The video output unit 217 is composed of, for example, a video output terminal, and transmits an image signal in order to display the video on a connected external display or the like. Further, the audio output unit 218 and the video output unit 217 may be one combined terminal, for example, a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

The communication unit 222 communicates between the image pickup device 101 and the external device, and transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal, for example. In addition, it receives control signals related to shooting such as shooting start and end commands, pan, tilt, and zoom drive, and drives the imaging device 101 from instructions of an external device capable of intercommunication with the imaging device 101. In addition, information such as various parameters related to learning processed by the learning processing unit 219, which will be described later, is transmitted and received between the image pickup device 101 and the external device. The communication unit 222 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a WirelessUSB, and a GPS receiver.

<System configuration with external communication equipment>
FIG. 3 is a diagram showing a configuration example of a wireless communication system between the image pickup device 101 and the external device 301. The image pickup device 101 is a digital camera having a photographing function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

The image pickup device 101 and the smart device 301 include, for example, a communication 302 by a wireless LAN compliant with the IEEE802.11 standard series, and a control station and a subordinate station such as Bluetooth Low Energy (hereinafter referred to as "BLE") BLE. Communication is possible with the communication 303 having a master-slave relationship. The wireless LAN and BLE are examples of communication methods, and each communication device has two or more communication functions, for example, by one communication function that communicates in a relationship between a control station and a subordinate station. If it is possible to control the other communication function, another communication method may be used. However, without losing generality, the first communication such as wireless LAN can perform higher-speed communication than the second communication such as BLE, and the second communication consumes more than the first communication. It shall be at least one of low power consumption and short communication range.

<Configuration of external communication equipment>
The configuration of the smart device 301 as an example of the external communication device will be described with reference to FIG. The smart device 301 is a so-called mobile phone, that is, a mobile terminal.

The smart device 301 has, for example, a wireless LAN control unit 401 for wireless LAN, a BLE control unit 402 for BLE, and a public line control unit 406 for public wireless communication. In addition, the smart device 301 further includes a packet transmission / reception unit 403. The wireless LAN control unit 401 performs RF control of the wireless LAN, communication processing, a driver that performs various controls of communication by the wireless LAN conforming to the IEEE802.11 standard series, and protocol processing related to the communication by the wireless LAN. The BLE control unit 402 performs a driver that performs RF control of BLE, communication processing, various controls of communication by BLE, and protocol processing related to communication by BLE. The public line control unit 406 performs a driver for performing RF control of public wireless communication, communication processing, various controls of public wireless communication, and protocol processing related to public wireless communication. Public wireless communication conforms to, for example, IMT (International Multimedia Telecommunication) standards and LTE (Long Term Evolution) standards. The packet transmission / reception unit 403 performs processing for executing at least one of transmission and reception of packets related to communication by wireless LAN and BLE and public wireless communication. In this example, the smart device 301 is described as performing at least one of transmission and reception of packets in communication, but other communication formats such as circuit switching may be used in addition to packet switching. good.

The smart device 301 further includes, for example, a control unit 411, a storage unit 404, a GPS receiving unit 405, a display unit 407, an operation unit 408, a voice input voice processing unit 409, and a power supply unit 410. The control unit 411 controls the entire smart device 301, for example, by executing a control program stored in the storage unit 404. The storage unit 404 stores, for example, a control program executed by the control unit 411 and various information such as parameters required for communication. Various operations described later are realized by the control unit 411 executing the control program stored in the storage unit 404.

The power supply unit 410 supplies power to the smart device 301. The display unit 407 has a function capable of outputting visually recognizable information such as an LCD or LED, or sound output of a speaker or the like, and displays various information. The operation unit 408 is, for example, a button or the like that accepts an operation of the smart device 301 by a user. The display unit 407 and the operation unit 408 may be composed of a common member such as a touch panel.

The voice input voice processing unit 409 may be configured to acquire the voice emitted by the user from, for example, a general-purpose microphone built in the smart device 301, and acquire the user's operation command by voice recognition processing.

In addition, voice commands are acquired by the user's pronunciation via a dedicated application in the smart device. Then, it can be registered as a specific voice command for causing the voice processing unit 214 of the image pickup apparatus 101 to recognize the specific voice command via the communication 302 by the wireless LAN.

The GPS (Global Positioning System) 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude / latitude information) of the smart device 301. Alternatively, the position may be estimated by using WPS (Wi-Fi Positioning System) or the like to estimate the current position of the smart device 301 based on the information of the wireless network existing in the surrounding area. When the acquired current GPS position information is located within a preset position range (within a predetermined radius range), the movement information is notified to the image pickup apparatus 101 via the BLE control unit 402, which will be described later. Used as a parameter for automatic shooting and automatic editing. Further, when the GPS position information has a position change of a predetermined value or more, the movement information is notified to the image pickup apparatus 101 via the BLE control unit 402, and is used as a parameter for automatic shooting or automatic editing described later.

As described above, the image pickup device 101 and the smart device 301 exchange data with the image pickup device 101 by communication using the wireless LAN control unit 401 and the BLE control unit 402. For example, it transmits or receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. In addition, the smart device issues an operation instruction such as shooting of the imaging device 101, transmits voice command registration data, and performs predetermined position detection notification and location movement notification based on GPS position information. It also sends and receives learning data via a dedicated application in the smart device.

<Sequence of imaging operation>
FIG. 5 is a flowchart of the automatic photographing process of the image pickup apparatus 101 according to the present embodiment.

When the user operates the power button provided on the image pickup apparatus 101, the processing of this flowchart starts. In the present embodiment, the image pickup device 101 and the smart device 301 are always connected by wireless communication, and various operations can be performed from the dedicated application on the smart device 301. Further, the processing of each step in the following flowchart is realized by the first control unit 223 controlling each unit of the image pickup apparatus 101.

In S501, the first control unit 223 determines whether or not the automatic shooting is stopped. The stop of automatic shooting will be described in the flow chart of the voice recognition process described later. If the automatic shooting is stopped, it waits without doing anything and waits until the automatic shooting stop is released. If the automatic shooting is not stopped, the process proceeds to S502 and image recognition processing is performed.

In S502, the first control unit 223 causes the image processing unit 207 to perform image processing on the signal captured by the imaging unit 206 to generate an image for subject recognition.

Subject recognition such as person or object recognition is performed from the generated image.

When recognizing a person, the face or human body of the subject is detected. In the face detection process, a pattern for determining the face of a person is predetermined, and a portion matching the pattern included in the captured image can be detected as a face image of the person.

At the same time, the reliability indicating the certainty of the subject's face is also calculated, and the reliability is calculated from, for example, the size of the face region in the image, the degree of coincidence with the face pattern, and the like.

Similarly, for object recognition, it is possible to recognize an object that matches a pre-registered pattern.

There is also a method of extracting a feature subject by using a histogram of hue, saturation, etc. in the captured image. In this case, regarding the image of the subject captured within the shooting angle of view, the process of dividing the distribution derived from the histogram of the hue, saturation, etc. into a plurality of sections and classifying the captured images for each section is executed. Will be done.

For example, a histogram of a plurality of color components is created for the captured image, the image is divided by the mountain-shaped distribution range, the captured image is classified in the area belonging to the combination of the same sections, and the image area of the subject is divided. Be recognized.

By calculating the evaluation value for each image area of the recognized subject, the image area of the subject having the highest evaluation value can be determined as the main subject area.

By the above method, each subject information can be obtained from the imaging information.

In S503, the first control unit 223 calculates the image shake correction amount. Specifically, first, the absolute angle of the image pickup device is calculated based on the angular velocity and acceleration information acquired by the device shake detection unit 209. Then, the vibration isolation angle for moving the tilt rotation unit 104 and the pan rotation unit 105 in the angle direction that cancels the absolute angle is obtained, and is used as the image shake correction amount. The calculation method of the image shake correction amount calculation process here can be changed by a learning process described later.

In S504, the first control unit 223 determines the state of the image pickup apparatus. Based on the angle and the amount of movement detected by the angular velocity information, the acceleration information, the GPS position information, etc., it is determined what kind of vibration / movement state the image pickup apparatus is currently in.

For example, when the image pickup device 101 is attached to a car for shooting, the subject information such as the surrounding landscape changes greatly depending on the distance traveled.

Therefore, it can be used for automatic subject search, which will be described later, by determining whether or not the vehicle is in a "vehicle moving state" in which the vehicle is mounted on a car or the like and is moving at a high speed.

Further, it is determined whether or not the change in the angle is large, and it is determined whether or not the image pickup apparatus 101 is in the “placed shooting state” where there is almost no shaking angle.

In the "place-shooting state", it can be considered that the angle of the image pickup apparatus 101 itself does not change, so that the subject search for the stand-alone shooting can be performed.

Further, when the angle change is relatively large, it is determined to be in the "handheld state", and the subject can be searched for for handheld use.

In S505, the first control unit 223 performs the subject search process. The subject search is composed of the following processes.

(1) Area division The area division will be described with reference to FIG. As shown in FIG. 8A, the area is divided around the entire circumference centering on the position of the image pickup device (origin O is the position of the image pickup device). In the example of FIG. 8A, the tilt direction and the pan direction are each divided by 22.5 degrees. When divided as shown in FIG. 8A, the circumference in the horizontal direction becomes smaller and the area area becomes smaller as the angle in the tilt direction deviates from 0 degrees. Therefore, in the imaging device of the present embodiment, as shown in FIG. 8B, when the tilt angle is 45 degrees or more, the area range in the horizontal direction is set to be larger than 22.5 degrees. 8 (c) and 8 (d) show an example in which the area is divided within the shooting angle of view. The axis 1301 is the direction of the image pickup apparatus 101 at the time of initialization, and the area division is performed with this direction angle as a reference position. Reference numeral 1302 indicates an angle of view area of the captured image, and an example of the image at that time is shown in FIG. 8 (d). In the image projected at the angle of view, the image is divided as shown in 1303 to 1318 in FIG. 8D based on the area division.

(2) Calculation of importance level for each area For each area divided as described above, the importance level indicating the priority of searching is calculated according to the subject existing in the area and the scene situation of the area. The importance level based on the subject's situation is based on, for example, the number of people present in the area, the size of the person's face, the face orientation, the certainty of face detection, the facial expression of the person, and the personal authentication result of the person. calculate. In addition, the importance level according to the situation of the scene is, for example, general object recognition result, scene discrimination result (blue sky, backlight, evening scene, etc.), sound level and voice recognition result from the direction of the area, motion detection in the area. Information etc. Further, the vibration state of the image pickup device is detected by the state determination (S504) of the image pickup device, and the importance level can be changed according to the vibration state. For example, when it is determined that the subject is in the "placed shooting state", the subject is searched for a subject having a high priority (for example, a user of an imaging device) registered by face recognition, so that the subject is searched for a specific person. When face recognition is detected, it is determined that the importance level is high. In addition, the automatic shooting described later will also be performed with priority given to the above-mentioned face, and even if the user of the image pickup device wears the image pickup device and takes a lot of time to carry around and take a picture, the image pickup device is removed and the desk is used. By placing it on the top, you can leave many images of the user. At this time, since it is possible to search by pan / tilt, it is possible to leave an image of the user or a group photo of many faces just by installing it properly without considering the placement angle of the image pickup device. can. Under the above conditions alone, as long as there is no change in each area, the area with the highest importance level will be the same, and as a result, the area to be searched will not change forever. Therefore, the importance level is changed according to the past shooting information. Specifically, the importance level is lowered for the area that has been continuously designated as the search area for a predetermined time, and the importance level is lowered for the area that was photographed in S513, which will be described later. May be good.

(3) Determining the search target area After the importance level of each area is calculated as described above, the area with the high importance level is determined as the search target area. Then, the pan / tilt search target angle required to capture the search target area at the angle of view is calculated.

In S506, the first control unit 223 performs pan / tilt drive. Specifically, the pan / tilt drive amount is calculated by adding the image shake correction amount and the drive angle in the control sampling based on the pan / tilt search target angle, and the lens barrel rotation drive unit 205 calculates the tilt rotation. The unit 104 and the pan rotation unit 105 are driven and controlled, respectively.

In S507, the first control unit 223 controls the zoom unit 201 to drive the zoom. Specifically, the zoom is driven according to the state of the search target subject determined in S505. For example, when the subject to be searched is the face of a person, if the face on the image is too small, it may not be detected because it is smaller than the minimum detectable size, and the face may be lost. In such a case, the size of the face on the image is controlled to be increased by zooming to the telephoto side. On the other hand, if the face on the image is too large, the subject tends to deviate from the angle of view due to the movement of the subject or the imaging device itself. In such a case, zooming to the wide-angle side controls the size of the face on the screen to be smaller. By performing the zoom control in this way, it is possible to maintain a state suitable for tracking the subject.

In S505 to S507, a method of searching for a subject by pan / tilt or zoom drive has been described, but the subject search may be performed by an imaging system that uses a plurality of wide-angle lenses to shoot in all directions at once. In the case of an omnidirectional camera, enormous processing is required when performing image processing such as subject detection using all the signals obtained by imaging as input images. Therefore, a part of the image is cut out, and the subject search process is performed within the cut out image range. The important level for each area is calculated in the same manner as the above-mentioned method, the cutting position is changed based on the important level, and the automatic shooting determination described later is performed. This makes it possible to reduce power consumption by image processing and search for a subject at high speed.

In S508, the first control unit 223 reads the frequency parameter. The frequency parameter is a set value indicating the ease of automatic shooting. The user can set any frequency from the options such as "low", "medium", and "high" via the dedicated application of the smart device 301. When the frequency is set to "high", a larger number of images are taken per predetermined time than when the frequency is set to "low". The "Medium" setting captures the number of shots between the "Low" and "High" settings. In addition, it can be automatically changed by the frequency setting process described later.

In S509, the first control unit 223 determines whether the read frequency parameter has a predetermined value. For example, if "highest" is set as the frequency for performing automatic shooting, the process proceeds to S510, and if not, the process proceeds to S512. The setting that the frequency is "highest" is a setting that is automatically changed by the frequency setting process described later, and the frequency setting by a normal user using the dedicated application of the smart device 301 is "low" as described above. It is set from "Medium" and "High" options. That is, the frequency is not set to "highest" in the setting by user operation.

In S510, the first control unit 223 determines whether the frequency boost time from "highest" to returning the frequency parameter setting started in S705, which will be described later, has ended. If it is finished, the process proceeds to S511, and if not, the process proceeds to S512.

In S511, since the frequency boost time has expired, the first control unit 223 restores the frequency parameter to the frequency setting before being set to "highest". At this time, if more than a predetermined number of shots are taken by automatic shooting during the frequency boost time, it can be determined that the current scene is the scene to be shot, so the frequency boost time may be extended. By doing so, it is possible to continue shooting the scenes that the user wants to shoot.

In S512, the first control unit 223 determines whether or not to perform automatic photographing.

Here, the determination of whether or not to perform automatic shooting will be described. Judgment as to whether or not to perform automatic shooting is performed based on whether or not the importance score exceeds a predetermined value. The importance score is a parameter used for determining whether or not to perform automatic photographing, and is different from the importance level for determining the search area. Scores are added to the importance score according to the detection status of the subject and the passage of time. For example, consider a case where automatic shooting is performed when the importance score exceeds 2000 points. In this case, first, the importance score has an initial value of 0 points, and points are added according to the passage of time from the time when the automatic shooting mode is entered. If there is no subject with high priority, the number of points will increase at an increase rate of 2000 points after 120 seconds, for example. If 120 seconds elapse without detecting a high-priority subject, the number of points added over time reaches 2000 points, and shooting is performed. Further, if a subject having a high priority is detected during the lapse of time, 1000 points will be added. Therefore, in a state where a subject having a high priority is detected, it is easy to reach 2000 points, and as a result, the shooting frequency tends to increase.

Further, for example, when the smiling face of the subject is recognized, 800 points are added. The points added based on this smile are added even if the subject is not a high-priority subject. Further, in the present embodiment, the points to be added based on the smile will be described by taking as an example the case where the points are the same regardless of whether or not the subject has a high priority, but the present invention is not limited to this. For example, the score added according to the detection of the smile of the subject having a high priority may be higher than the score added according to the detection of the smile of the subject having a low priority. By doing so, it becomes possible to take a picture more in line with the user's intention. If the number of points exceeds 2000 due to the addition of points due to changes in the facial expressions of these subjects, automatic shooting is performed. In addition, even if the points added due to the change in facial expression do not exceed 2000 points, the points added after that time will reach 2000 points in a shorter time.

The points added with the passage of time will be described by taking, for example, a case where points are added so as to reach 2000 points in 120 seconds, and a case where only 2000/120 points are added every second, that is, points are added linearly with respect to time. Is not limited to this. For example, points may not be added up to 110 seconds out of 120 seconds, but may be increased so that points are added by 200 points per second to reach 2000 points in 10 seconds from 110 seconds to 120 seconds. By doing so, it is possible to prevent the number of points to be photographed from being reached regardless of the high or low priority due to the addition of points due to the change in the facial expression of the subject. In the case of the point addition method that linearly increases with the passage of time, points have already been added over time for a long time, so even if the points are added due to the change to the smile of a low-priority subject, the number of points to be shot will be reached. In many cases, it is difficult to reflect the high and low priorities. However, if the score of points added due to changes in facial expressions is lowered, the timing of changes in facial expressions will be missed, so it is desirable to avoid measures by lowering the points of points added. Therefore, do not add points until 110 seconds. In this way, 110 seconds elapse without adding points to the subject having a low priority. On the other hand, since 1000 points are added to a subject having a high priority at the time of detection, 1000 points are added even if there is no point added due to the passage of time up to 110 seconds. As a result, when points are added due to changes in facial expressions, the possibility that a low-priority subject will reach the number of points to be photographed can be suppressed as compared with a high-priority subject, and the high and low priorities function. Cheap. In the above explanation, the change in facial expression is taken as an example, but other criteria for adding points may be when the voice becomes louder or when the gesture becomes louder. Also for these, in order to make it easier for the high and low priorities to function, the difference in the point addition method as described above may be provided.

Further, even if the action of the subject does not exceed 2000 points, the image is always taken in 120 seconds with the passage of time, so that it is possible that the image is not taken at all for a certain period of time.

Further, when the subject is detected in the middle, the time for starting the increase may be advanced in 120 seconds. That is, for example, when a subject with a high priority is detected at 60 seconds, even if 1000 points are added by that, the score does not still exceed 2000 points, but the subject is detected instead of increasing to 110 seconds as it is. After that, the linear increase may be started after 30 seconds have passed. Alternatively, the linear increase may be started 20 seconds before 10 seconds before 120 seconds. By doing so, the possibility that a subject having a high priority is taken is increased, and it becomes easier to realize the shooting according to the user's intention.

When automatic shooting is performed, the importance score is reset to 0 points. Automatic shooting will not be performed until the number exceeds 2000 points again.

Here, the frequency parameter is used to control how the importance score increases over time. If the subject is not detected in the above example, it is set to take 120 seconds until automatic shooting. This was explained by taking the case where the frequency parameter is "medium" as an example, but in the frequency boost state (frequency parameter "highest"), the importance score is increased so that automatic shooting is performed in 60 seconds. Change the direction. In this case, 2000/60 points may be added every second, or 400 points may be added per second for the remaining 5 seconds up to 60 seconds without adding points until 55 seconds, for example. The advantages of doing the latter are as described above. To give another example of frequency, for example, when the frequency parameter is "high", it is increased to 2000 points in 100 seconds, and when the frequency parameter is "low", it is increased to 2000 points in 240 seconds. Design to let you. As described above, when the frequency parameter is "highest", at least one image is taken in the shortest time (60 seconds in the description of the present embodiment). Therefore, increasing the frequency of shooting means increasing the number of shots taken per hour by changing the method of adding points, and decreasing the frequency of shooting means increasing the number of shots per hour by changing the method of adding points. It is to reduce the number of shots.

The above is the description of the determination of whether or not to perform automatic shooting. Based on the above determination, if it is determined that automatic shooting is performed, the process proceeds to S513, and if it is determined that shooting is not performed, the process proceeds to S501.

In S513, the first control unit 223 executes a photographing process. The shooting process referred to here includes still image shooting and moving image shooting.

FIG. 6 is a flowchart of the voice recognition process of the image pickup apparatus 101 according to the present embodiment. When the voice emitted by the user is input to the microphone built in the image pickup apparatus 101, the voice input voice processing unit 409 performs voice recognition processing and acquires the user's operation command.

In S601, the first control unit 223 determines whether or not the wake word has been detected. The wake word is an activation command for starting voice command recognition that gives a specific instruction to the image pickup apparatus 101 by voice. When giving instructions by voice, it is necessary to generate a command word after wake word recognition and the recognition must be successful. If the wake word is detected, the process proceeds to S602, and if it is not detected, the process of S601 is repeated until it is detected.

In S602, the first control unit 223 stops the automatic photographing process. When the wake word is recognized, the command word is in the standby state, so the automatic shooting process is stopped. Stopping automatic shooting refers to subject search and shooting processing execution using pan / tilt operation and zoom operation. The purpose of stopping the automatic shooting is to stop the automatic shooting process and put it in the command word standby state in order to quickly respond to the command word instruction issued after the wake word. Further, when a shooting instruction is to be given by a voice instruction, it is possible to shoot in the direction in which the user is trying to shoot by stopping the pan / tilt.

In S603, the first control unit 223 sounds a recognition sound for indicating to the user that the wake word has been recognized successfully.

In S604, the first control unit 223 determines whether or not a command word has been detected. If the command word is detected, the process proceeds to S606, and if the command word is not detected, the process proceeds to S605.

In S605, the first control unit 223 detects the wake word and determines whether a predetermined time has elapsed since the command word standby state was entered. When the predetermined time has elapsed, the process proceeds to S601, the standby state of the command word is stopped, and the standby state of the wake word is set. If the predetermined time has not elapsed, S604 is repeated until the command word is detected.

In S606, the first control unit 223 determines whether or not the detected command word is a still image shooting command. This still image shooting command is a command for requesting the image pickup apparatus 101 to shoot and record a single still image. If it is determined that the command is still image shooting, the process proceeds to S607, and if not, the process proceeds to S608.

In S607, the first control unit 223 performs a still image photographing process. Specifically, the signal captured by the image pickup unit 206 is converted into, for example, a JPEG file by the image processing unit 207, and recorded on the recording medium 221 by the image recording unit 208.

In S608, the first control unit 223 determines whether or not the detected command word is a subject search command. If it is determined that the command is a subject search command, the process proceeds to S609, and if not, the process proceeds to S610.

In S609, the first control unit 223 performs the subject search process. If the search target area has already been determined by the subject search process in S505 and the subject is captured by the pan / tilt drive of S506 and the zoom drive of S507, the tracking of the subject is stopped and another subject is stopped. The subject search process is executed to search for. This is because if the user instructs the subject search while capturing the subject, it means that there is a subject to be photographed separately from the subject currently captured.

After the processing of S607 to S609 is completed, the frequency setting processing is performed in S610. The frequency setting process is a process of setting a frequency parameter for how many shots are taken within a predetermined time. The details of the processing contents will be described later, but in the frequency setting process executed in S610, the frequency of shooting is set to be higher.

In S611, the first control unit 223 determines whether or not the detected command word is a moving image recording start command. The moving image shooting command is a command that requests the image pickup device 101 to capture and record a moving image. If it is determined that the command is to start moving image recording, the process proceeds to S612, and if not, the process proceeds to S613.

In S612, the first control unit 223 starts taking a moving image by using the imaging unit 206 and records the moving image on the recording medium 221. During video recording, pan / tilt and zoom drive are not performed, the subject is not searched, and automatic shooting continues to be stopped.

In S613, the first control unit 223 determines whether or not the detected command word is a moving image recording stop command. If it is determined that the command is to stop moving image recording, the process proceeds to S614, and if not, the process proceeds to S615.

In S614, the first control unit 223 stops taking and recording a moving image using the imaging unit 206, and completes recording as a moving image file on the recording medium 221.

In S615, the first control unit 223 executes other processing in the voice command. For example, processing for a command to pan / tilt in a direction specified by the user and processing for a command to change various shooting parameters such as exposure compensation can be mentioned.

In S616 and S617, the first control unit 223 performs a restart process for the automatic shooting stopped in S602. As a result, the processes of S502 to S510 can be operated, and automatic shooting is restarted.

At this time, the frequency setting process is not executed in the case of instructions to start or stop recording of the moving image. This is because the signal from the imaging unit 206 is continuously recorded after the start of recording the moving image, so there is no point in setting the frequency setting high. Also, after the video recording is stopped, the fact that the user has instructed to stop recording indicates that the scene that should be recorded is over, so set the frequency unnecessarily to avoid shooting unnecessary images. Because.

Further, when the remaining battery level of the image pickup apparatus 101 is low, or when the image pickup apparatus 101 is heated to a predetermined temperature or higher due to heat generation, it is preferable not to operate the image pickup apparatus 206 or the like frequently. In such a situation, the frequency parameter according to S704 in FIG. 7, which will be described later, may not be set to “highest”.

FIG. 7 is a flowchart of the frequency setting process of the image pickup apparatus 101 according to the present embodiment. As a means for setting the frequency at which the user performs automatic shooting, there is a method of performing the automatic shooting via a dedicated application in the smart device 301. The processing of this flowchart is also started in response to the execution of S610 in FIG. Further, it is also started in response to the user instructing the frequency change via the dedicated application in the smart device 301.

In S701, the first control unit 223 determines whether the frequency is set via the dedicated application in the smart device 301. If the frequency is set via the dedicated application, the process proceeds to S702, and if not (for example, when the frequency is set in S610), the process proceeds to S703.

In S702, the first control unit 223 sets the frequency parameter instructed by the user. For example, as shown in FIG. 9, on the screen of the dedicated application in the smart device 301, the setting can be made by selecting "low", "medium", and "high" from the items of the automatic shooting frequency.

Here, the application screen of FIG. 9 will be described.

In the dedicated application of the smart device 301, still images and moving images are prepared as contents to be automatically captured. Furthermore, as the content to be automatically shot, it is possible to set whether to prioritize the still image or the moving image from the dedicated application. This setting can be changed by touching (flicking) the knob of the slider bar as shown in FIG. When it is set to give priority to still images, more still images are taken than moving images. Also, when it is set to give priority to moving images, more moving images are taken than still images.

It is also possible to set how many ranges the imaging device searches for a scene to be imaged from the front direction. In the example of FIG. 9, three patterns can be set: a range of 60 degrees at 30 degrees from the front to the left and right, a range of 180 degrees at 90 degrees from the front to the left and right, and the entire circumference. In addition, a numerical value may be input so that a finer range can be set.

Further, in the case of automatic imaging, there is a concern that the captured content will be too large. Therefore, a function for automatically deleting an image is provided so that the function can be turned on and off from the smart device 301. The images to be automatically deleted may be deleted in order from the oldest shooting date / time, or in ascending order of importance. The importance here is, for example, in the case of a still image, the parameters that are predicted to be an image that the user will want to leave, such as whether there is little blurring or whether a person is shown, are quantified. It is a thing. Further, in the case of a moving image, for example, whether or not a person is shown and whether or not a person's voice such as a conversation is recorded are quantified to calculate the importance. Then, the higher the total value, the higher the importance.

The above is the description of FIG. Returning to the description of FIG.

In S703, the first control unit 223 determines whether the frequency setting is called from the voice recognition process. If the frequency setting is called from the voice recognition process, the process proceeds to S704, and if not, the frequency setting process ends.

In S704, the first control unit 223 sets the frequency parameter even higher than the frequency that can be set in S702. The reason for doing this is that the timing at which the user instructs the shooting is at least the timing at which the shooting is desired. That is, since the situation is such that the user wants to shoot at the timing when the user instructs the shooting, it is considered that a scene to be shot is likely to occur in a period close to the time. Focusing on this point, the imaging device of the present embodiment uses a voice instruction by a user's voice command as a trigger, presumes that the scene should be shot for a certain period after the voice command is input, and increases the shooting frequency. This makes it possible to capture an image that the user wants to capture without missing it. In the present embodiment, it is described that the parameter with the frequency of "highest" is set, but the frequency may be increased stepwise each time the frequency is set by the voice command instruction. In this case, the upper limit of the frequency is the fastest frame speed of continuous shooting included in the imaging device 101.

In S705, the first control unit 223 sets the frequency boost time until the frequency parameter set to "highest" in S704 is returned to the original parameter, and starts the countdown. For example, if the frequency setting is set to "medium" and the frequency setting is set to "highest" by voice command instruction, assuming that the frequency boost time is 60 seconds, the frequency setting will be set after 60 seconds have passed. The setting returns to "Medium" (actual processing is performed in S511). The frequency boost time referred to here is the time for maintaining the highest frequency. This frequency boost time is set automatically, but the user may be able to set any time.

At this time, the frequency boost time may be returned to the original setting depending on whether or not a predetermined number of shots are taken by automatic shooting, in addition to returning the setting when the predetermined time elapses.

Further, if the frequency setting is set to "highest" again by the voice command before the countdown of the frequency boost time is completed, the predetermined time or the predetermined number of sheets until the frequency setting is restored is extended.

Further, as a determination to restore the frequency setting, it may be determined whether or not the subject search process is performed in all directions in the pan direction.

Although preferable examples of the present invention have been described above, the present invention is not limited to these examples, and various modifications and changes can be made within the scope of the gist thereof.

For example, in the above embodiment, an example of a voice command has been used as a means for instructing a user to take a picture. In addition to this, even when shooting is instructed by an instruction via a communication means from a smart device or a BLE remote controller, the frequency setting may be set to "highest" after executing the instruction. Also, when an instruction to execute processing according to this vibration pattern is detected by a specific vibration pattern using the acceleration sensor in the image pickup device, the frequency setting is set to "highest" after executing the instruction. You may do so. Further, even when the movement of the user's hand is analyzed through the imaging unit and a gesture instruction is received by a gesture, the frequency setting may be set to "highest" after the instruction is executed.

Further, the present embodiment has been characterized in that the subject is tracked by pan / tilt drive and zoom drive to capture an image desired by the user. Regarding this, for example, it is conceivable to adopt a 360 ° camera as an imaging means to constantly shoot in all directions and to obtain an image of a subject by cutting out an image in a necessary range from the captured image. In such a case, the moving image recording is always executed, and the frame rate of the moving image is increased after recording in the still image format in response to the input of the cutout instruction. Even in this case, the frame rate may be set to the highest rate that can be set, or may be a value that exceeds the settable value, as in the case of the shooting frequency in the above-described embodiment. Further, as the condition for returning the increased frame rate to the original value, the elapse of a certain period of time may be adopted as in the above-described embodiment. As a result, the image is recorded more frequently around the timing when the user wants to record the image, and as a result, it becomes easier to acquire an image in which, for example, the image is not out of focus with respect to the moving object.

If the shooting timing does not come within the frequency boost time, it is conceivable that no shot is shot. Therefore, when the still image shooting command is first received, pan / tilt and zoom drive are not performed, and one image is shot without searching for the subject. Then, while searching for the subject, three consecutive shots are taken. After that, for a predetermined time, the frequency is boosted and automatic shooting is performed. By doing so, when the user intentionally instructs the still image shooting by the still image shooting command, it is possible that no one shot is taken and at least four shots are taken.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (19)

A sound collecting means for collecting sound and
An analysis means for analyzing the sound collected by the sound collecting means, and an analysis means.
Automatic shooting means that automatically shoots,
A setting means for setting the shooting frequency of the automatic shooting means and
Have,
An imaging device characterized in that, when a specific voice instruction is obtained as a result of analysis by the analysis means, the imaging frequency is set higher by the setting means after performing an operation according to the instruction. The imaging device according to claim 1, wherein the automatic photographing means automatically pans, tilts, and zooms the imaging device, tracks a subject, and automatically captures a still image or a moving image. .. The imaging device according to claim 1, wherein the frequency set by the setting means is set at a higher frequency than the frequency that can be arbitrarily set by the user. The imaging apparatus according to claim 1, wherein when a predetermined time elapses after the photographing frequency is set higher by the setting means, the photographing frequency is restored to the original value. The fourth aspect of claim 4, wherein when the automatic photographing means takes a predetermined number of images or more in a state where the photographing frequency is set higher by the setting means, the predetermined time is extended. Imaging device. The imaging device according to claim 4, wherein when a specific voice instruction is recognized by the analysis means in a state where the photographing frequency is set higher by the setting means, the predetermined time is extended. The imaging apparatus according to claim 1, wherein the photographing frequency is set higher by the setting means, and then the photographing frequency is restored when a predetermined number of images are photographed by the automatic photographing means. The imaging according to claim 7, wherein when the automatic photographing means takes a predetermined number of images or more in a state where the photographing frequency is set higher by the setting means, the predetermined number of images is increased. Device. The imaging device according to claim 7, wherein when a specific voice instruction is recognized by the analysis means in a state where the photographing frequency is set higher by the setting means, the predetermined number of images is increased. Further, it has a rotating means for changing the direction of the image pickup apparatus, and after the shooting frequency is set higher by the setting means, the shooting frequency is restored when the subject in all directions is searched by the rotating means. The imaging device according to claim 1. The imaging device according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is a shooting instruction, the shooting frequency is set higher by the setting means. The imaging device according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to search for a subject, the shooting frequency is set higher by the setting means. The imaging according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to start recording a moving image, the frequency setting is not set high by the setting means. Device. The imaging according to claim 1, wherein when the specific voice instruction analyzed by the analysis means is an instruction to stop recording of a moving image, the frequency setting is not set high by the setting means. Device. When the remaining battery level of the image pickup apparatus is less than a predetermined amount, even if the voice analyzed by the analysis means is the specific voice instruction, the frequency setting is not set high by the setting means. The imaging device according to claim 1. When the temperature of the image pickup apparatus is higher than a predetermined temperature, even if the voice analyzed by the analysis means is the specific voice instruction, the frequency setting is not set high by the setting means. The imaging device according to claim 1. When a specific instruction is detected from a mobile terminal via a communication means, a specific vibration pattern using the acceleration sensor of the image pickup device is detected, or a specific instruction is given by a gesture instruction that realizes the instruction by the movement of the user's hand. The imaging device according to claim 1, wherein the frequency is set high by the setting means even when the frequency is increased. A control method for an image pickup device having a sound collecting means for collecting sound.
An analysis step for analyzing the sound collected by the sound collecting means, and
An automatic shooting step that automatically shoots and
It has a setting step for setting the shooting frequency in the automatic shooting step.
If the result of the analysis by the analysis step is a specific voice instruction, the control of the imaging device is characterized in that the setting step is executed after the operation according to the instruction is performed to set the shooting frequency high. Method. A computer-readable program for allowing a computer to function as each means of the imaging apparatus according to any one of claims 1 to 17.

Images