JP7393133B2 - Image processing device, image processing method, imaging device, program, storage medium - Google Patents

Description

The present invention relates to a technique for selecting images using machine learning.

When shooting still images and videos using an imaging device such as a camera, the user determines the subject to be shot through a finder, etc., checks the shooting conditions himself, adjusts the framing of the shot image, and operates the shutter button to capture the image. It is common to take pictures.

In contrast to such an imaging device that performs photography through user operations, Patent Document 1 discloses a camera called a so-called lifelog camera that periodically and continuously photographs images without the user giving a photography instruction. ing. A life log camera is used while attached to a user's body with a strap or the like, and records scenes that the user sees in daily life as images at regular time intervals. Photographing with a life log camera is not done at the user's intended timing, such as when the user releases the shutter, but at regular intervals, making it possible to capture unexpected moments that would not normally be photographed.

Special table 2016-536868 publication

When a user wears a life log camera and automatically takes pictures on a regular basis, the following problems may occur.

For example, when recording captured images on the camera itself, or when recording images on a smartphone or portable tablet device connected to the camera, there is a risk that the media capacity will be used up without the user noticing. If the media runs out of free space, the user will not be able to take a picture when he or she intends to do so, or the user will not be able to take a desired scene during automatic shooting.

Alternatively, even if you are transferring captured images to a server where there is less concern about running out of space, there is a possibility that a huge number of images will be captured due to automatic shooting, so be sure to check the captured images and avoid any images you do not like. It takes time to sort out failed images.

In order to solve these problems, it is conceivable to assign a score to each automatically captured image and select the image to be recorded or displayed according to the score. However, when using machine learning, the way scores are assigned may change over time. In this case, there may be a difference in the selection criteria between an image that has been selected as a recording or display target in the past and an image that has been newly selected as a recording or display target.

The present invention has been made in view of the above-mentioned problems, and its purpose is to provide an image processing device that can automatically select images that are considered unimportant to the user.

An image processing apparatus according to the present invention includes a storage means for storing an image obtained by imaging a subject, a learning means for learning a user's favorite image by machine learning, and an image processing device based on learning parameters of the learning means. , an evaluation means for acquiring a score that is an evaluation value of the image stored in the storage means; a selection means for selecting a deletion candidate from the images stored in the storage means by referring to the score ; an image processing apparatus, comprising: a deletion means for deleting the image selected by the selection means from the storage means; the image processing apparatus is in a learning mode in which the learning means learns a user's favorite image; and an automatic deletion mode in which the deletion means automatically deletes images stored in the storage means, and the evaluation means receives an instruction to set learning parameters of the learning means from an external device in the learning mode. If there is, the learning parameter is set, and if there is no instruction, the learned learning parameter is set, and then the re-evaluation is performed to re-score all the images, and the selection means: In the automatic deletion mode, if the learning parameters of the learning means have been updated , deletion candidates are selected by referring to the scores for all images, and if they have not been updated , the previous file is automatically deleted. The method is characterized in that deletion candidates are selected by referring to the scores for only images obtained after the processing .

According to the present invention, it is possible to provide an image processing device that can automatically select images that are considered unimportant to the user.

FIG. 1 is a diagram schematically showing an imaging device. 1 is a diagram showing the configuration of an imaging device. FIG. 2 is a diagram showing the configuration of an imaging device and external equipment. FIG. 3 is a diagram showing the configuration of external equipment. It is a flow chart explaining operation of a control circuit. 3 is a flowchart illustrating automatic shooting mode processing. FIG. 2 is a diagram illustrating a neural network. FIG. 3 is a diagram for explaining image display processing. It is a flowchart explaining learning mode determination. It is a flow chart explaining learning mode processing. 3 is a flowchart illustrating automatic file deletion mode processing.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

<Configuration of imaging device>
FIG. 1 is a diagram schematically showing an imaging device that is an embodiment of an image processing device of the present invention. The present invention is applicable not only to digital cameras and digital video cameras, but also to surveillance cameras, web cameras, mobile phones, and the like. In this embodiment, the description will be made assuming a configuration in which the imaging device itself also serves as an image processing device that performs machine learning, but an image processing device that is separate from the imaging device and can communicate with the imaging device It may also be configured to perform machine learning for this purpose.

The imaging device 101 shown in FIG. 1A is provided with an operation member (hereinafter referred to as a power button, but operations such as a tap, flick, or swipe on a touch panel) that can operate a power switch. There is. A lens barrel 102 , which is a housing that includes a group of photographic lenses and an image sensor that performs imaging, is attached to the imaging device 101 , and is provided with a rotation mechanism that can rotate the lens barrel 102 with respect to a fixed portion 103 . The tilt rotation unit 104 is a motor drive mechanism that can rotate the lens barrel 102 in the pitch direction shown in FIG. 1(b), and the pan rotation unit 105 is a motor drive mechanism that can rotate the lens barrel 102 in the yaw direction. Therefore, the lens barrel 102 is rotatable in one or more axial directions. Note that FIG. 1(b) shows the axis definition at the fixed part 103 position. Both the angular velocity meter 106 and the accelerometer 107 are mounted on the fixed part 103 of the imaging device 101. Then, vibrations of the imaging device 101 are detected based on the angular velocity meter 106 and the accelerometer 107, and the tilt rotation unit and the pan rotation unit are rotated based on the detected swing angle. Thereby, it is possible to correct the shake of the lens barrel 102, which is a movable part, and to correct the inclination.

FIG. 2 is a block diagram showing the configuration of the imaging device of this embodiment. In FIG. 2, the control circuit 221 includes a processor (eg, CPU, GPU, microprocessor, MPU, etc.) and memory (eg, DRAM, SRAM, etc.). These control each block of the imaging device 101 by executing various processes, and control data transfer between each block. The nonvolatile memory (EEPROM) 214 is an electrically erasable/recordable memory, and stores constants, programs, etc. for operation of the control circuit 221.

In FIG. 2, a zoom unit 201 includes a zoom lens that changes magnification. A zoom drive control circuit 202 drives and controls the zoom unit 201. Focus unit 203 includes a lens that performs focus adjustment. A focus drive control circuit 204 drives and controls the focus unit 203.

The imaging unit 206 includes an image sensor and an A/D converter, and the image sensor receives incident light through each lens group, and outputs charge information corresponding to the amount of light to the image processing circuit 207 as an analog image signal. The image processing circuit 207 is an arithmetic circuit equipped with multiple ALUs (Arithmetic and Logic Units), and performs image processing such as distortion correction, white balance adjustment, and color interpolation processing on digital image data output by A/D conversion. Apply the processing and output the applied digital image data. The digital image data outputted from the image processing circuit 207 is converted into a recording format such as JPEG format by the image recording circuit 208, and is sent to the memory 213 and a video output circuit 215, which will be described later.

The lens barrel rotation drive circuit 205 drives the tilt rotation unit 104 and the pan rotation unit 105 to drive the lens barrel 102 in the tilt direction and the pan direction.

The device shake detection circuit 209 is equipped with, for example, an angular velocity meter (gyro sensor) 106 that detects the angular velocity of the imaging device 101 in the three-axis directions, and an accelerometer (acceleration sensor) 107 that detects the acceleration of the device in the three-axis directions. Ru. The device shake detection circuit 209 calculates the rotation angle of the device, the amount of shift of the device, etc. based on the detected signal.

The audio input circuit 211 acquires audio signals around the imaging device 101 from a microphone provided in the imaging device 101, performs analog-to-digital conversion, and transmits the signal to the audio processing circuit 212. The audio processing circuit 212 performs audio-related processing such as optimization processing of the input digital audio signal. The audio signal processed by the audio processing circuit 212 is then transmitted to the memory 213 by the control circuit 221. The memory 213 temporarily stores the image signal and audio signal obtained by the image processing circuit 207 and the audio processing circuit 212.

The image processing circuit 207 and the audio processing circuit 212 read out image signals and audio signals temporarily stored in the memory 213, encode the image signals, encode the audio signals, etc., and convert them into compressed image signals and compressed audio signals. generate. The control circuit 221 transmits these compressed image signals and compressed audio signals to the recording and reproducing circuit 218.

The recording/reproducing circuit 218 records compressed image signals and compressed audio signals generated by the image processing circuit 207 and the audio processing circuit 212, as well as other control data related to photographing, on the recording medium 219. Furthermore, when the audio signal is not compressed and encoded, the control circuit 221 transmits the audio signal generated by the audio processing circuit 212 and the compressed image signal generated by the image processing circuit 207 to the recording/reproducing circuit 218. The information is recorded on the recording medium 219.

The recording medium 219 may be a recording medium built into the imaging device 101 or a removable recording medium. The recording medium 219 can record various data such as compressed image signals, compressed audio signals, and audio signals generated by the imaging device 101, and a medium with a larger capacity than the nonvolatile memory 214 is generally used. . For example, the recording medium 219 includes any type of recording medium such as a hard disk, an optical disk, a magneto-optical disk, a CD-R, a DVD-R, a magnetic tape, a nonvolatile semiconductor memory, and a flash memory.

The recording and reproducing circuit 218 reads (reproduces) compressed image signals, compressed audio signals, audio signals, various data, and programs recorded on the recording medium 219. The control circuit 221 then transmits the read compressed image signal and compressed audio signal to the image processing circuit 207 and the audio processing circuit 212. The image processing circuit 207 and the audio processing circuit 212 temporarily store the compressed image signal and the compressed audio signal in the memory 213, decode them according to a predetermined procedure, and send the decoded signals to the video output circuit 215 and the audio output circuit 216. do.

A plurality of microphones mounted on the imaging device 101 are connected to the audio input circuit 211, and the audio processing circuit 212 can detect the direction of sound on a plane in which the plurality of microphones are installed. This information is used for object search and automatic shooting, which will be described later. Furthermore, the voice processing circuit 212 detects a specific voice command. In addition to a plurality of voice commands registered in advance, the configuration may also be such that the user can register a specific voice in the imaging device. It also performs sound scene recognition. In sound scene recognition, sound scenes are determined using a network trained in advance through machine learning based on a large amount of audio data. For example, a network is set up in the audio processing circuit 212 to detect specific scenes such as "cheering", "claps", and "sounding". When a specific sound scene or specific audio command is detected, a detection trigger signal is output to the control circuit 221. The power supply circuit 210 supplies power for operating the control circuit 221.

The audio output circuit 216 outputs a preset audio pattern from a speaker built into the imaging device 101 during, for example, shooting. The LED control circuit 222 controls the LEDs provided in the imaging device 101 in a preset lighting/blinking pattern during, for example, photographing. The video output circuit 215 includes, for example, a video output terminal, and transmits an image signal to display a video on a connected external display or the like. Furthermore, the audio output circuit 216 and the video output circuit 215 may be connected to one terminal, such as a terminal such as an HDMI (registered trademark) (High-Definition Multimedia Interface) terminal.

The communication circuit 220 communicates between the imaging device 101 and an external device, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. It also receives control signals related to photography, such as photography start and end commands, pan/tilt, and zoom driving, and drives the imaging device 101 according to instructions from an external device that can communicate with the imaging device 101. Further, information such as various parameters related to learning to be processed by a learning processing circuit 217, which will be described later, is transmitted and received between the imaging device 101 and an external device. The communication circuit 220 is, for example, a wireless communication module such as an infrared communication module, a Bluetooth (registered trademark) communication module, a wireless LAN communication module, a Wireless USB, or a GPS receiver.

<Configuration with external communication devices>
FIG. 3 is a diagram showing a configuration example of a wireless communication system between the imaging device 101 and the external device 301. The imaging device 101 is a digital camera with a shooting function, and the external device 301 is a smart device including a Bluetooth communication module and a wireless LAN communication module.

The imaging device 101 and the external device 301 communicate via wireless LAN 302 based on the IEEE802.11 standard series, for example, and have a master-slave relationship between a control station and a subordinate station, such as Bluetooth Low Energy (hereinafter referred to as "BLE"). It is possible to communicate with the communication 303 having the following. Note that wireless LAN and BLE are examples of communication methods, and each communication device has two or more communication functions. For example, one communication function that performs communication in the relationship between a control station and a dependent station, Other communication methods may be used as long as it is possible to control the other communication function. However, without loss of generality, the first communication such as wireless LAN can be faster than the second communication such as BLE, and the second communication consumes less than the first communication. It is assumed that the power consumption is low or the communicable distance is short.

The configuration of the external device 301 will be explained using FIG. 4. The external device 301 includes, for example, a wireless LAN control circuit 401 for wireless LAN and a BLE control circuit 402 for BLE, as well as a public line control circuit 406 for public wireless communication. Furthermore, the external device 301 further includes a packet transmitting/receiving circuit 403. The wireless LAN control circuit 401 performs RF control of the wireless LAN, communication processing, a driver that performs various controls of wireless LAN communication based on the IEEE 802.11 standard series, and protocol processing related to wireless LAN communication. The BLE control circuit 402 performs BLE RF control, communication processing, a driver that performs various controls of BLE communication, and protocol processing regarding BLE communication. The public line control circuit 406 performs RF control of public radio communication, communication processing, a driver for performing various controls of public radio communication, and protocol processing related to public radio communication. Public wireless communication is based on, for example, the International Multimedia Telecommunications (IMT) standard or the Long Term Evolution (LTE) standard. The packet transmitting/receiving circuit 403 performs processing for transmitting and/or receiving packets related to wireless LAN, BLE communication, and public wireless communication. Note that in this example, the external device 301 will be described as one that transmits and/or receives packets during communication; however, other communication formats such as line switching may be used in addition to packet exchange. good.

The external device 301 further includes, for example, a control circuit 411, a storage circuit 404, a GPS reception circuit 405, a display device 407, an operation member 408, an audio input/processing circuit 409, and a power supply circuit 410. The control circuit 411 controls the entire external device 301 by executing a control program stored in the storage circuit 404, for example. The storage circuit 404 stores, for example, a control program executed by the control circuit 411 and various information such as parameters necessary for communication. Various operations described below are realized by the control circuit 411 executing a control program stored in the storage circuit 404.

A power supply circuit 410 supplies power to the external device 301. The display device 407 has a function of outputting visually recognizable information such as an LCD or LED, or outputting sound such as a speaker, and displays various information. The operation member 408 is, for example, a button that accepts an operation on the external device 301 by the user. Note that the display device 407 and the operation member 408 may be configured by a common member such as a touch panel, for example.

The voice input/processing circuit 409 may be configured to acquire the voice uttered by the user from, for example, a general-purpose microphone built into the external device 301, and acquire the user's operation command through voice recognition processing.

In addition, a voice command is acquired by the user's pronunciation via a dedicated application in the external device 301. Then, it can also be registered as a specific voice command for causing the voice processing circuit 212 of the imaging apparatus 101 to recognize the specific voice command via the wireless LAN communication 302.

A GPS (Global Positioning System) 405 receives a GPS signal notified from a satellite, analyzes the GPS signal, and estimates the current position (longitude/latitude information) of the external device 301 . Alternatively, the position estimation may be performed by using WPS (Wi-Fi Positioning System) or the like to estimate the current position of the external device 301 based on information on surrounding wireless networks. If the acquired current GPS position information is located within a preset position range (within a predetermined radius), the movement information is notified to the imaging device 101 via the BLE control circuit 402, and as described below. Use as a parameter for automatic shooting and automatic editing. Further, when there is a positional change of more than a predetermined value in the GPS positional information, movement information is notified to the imaging device 101 via the BLE control circuit 402, and is used as a parameter for automatic shooting and automatic editing, which will be described later.

As described above, the imaging device 101 and the external device 301 exchange data with the imaging device 101 through communication using the wireless LAN control circuit 401 and the BLE control circuit 402. For example, it transmits and receives data such as audio signals, image signals, compressed audio signals, and compressed image signals. In addition, the external device 301 issues operational instructions such as photographing to the imaging device 101, sends voice command registration data, and sends a notification of detection of a predetermined position based on GPS position information and notification of location movement. It also transmits and receives learning data via a dedicated application within the external device 301.

<Sequence of imaging operation>
FIG. 5 is a flowchart illustrating an example of the operation handled by the control circuit 221 of the imaging apparatus 101 in this embodiment.

When the user operates a power button provided on the imaging device 101, the power supply circuit 210 supplies power to the control circuit 221 and each block of the imaging device 101. When power is supplied, the process shown in FIG. 5 starts. In step S501 (hereinafter, "step S" is simply abbreviated as "S"), startup conditions are read. In this embodiment, the power may be started by manually pressing the power button, or the power may be started by an instruction from an external device (for example, 301) through external communication (for example, BLE communication). Alternatively, the power may be activated by detecting that the user has tapped the imaging device 101, or the power may be activated by detecting that a specific voice command has been input. Further, the activation condition read here is used as one parameter element during subject search and automatic shooting, and this will be described later. When the reading of the activation conditions is completed, the process advances to S502.

In S502, detection values of various sensors are read. The sensor detection value read here is the detection value of a sensor that detects vibrations, such as a gyro sensor or an acceleration sensor from the device shake detection circuit 209. It is also the rotational position of the tilt rotation unit 104 and the pan rotation unit 105. Further, it is the sound level detected by the sound processing circuit 212, the detection trigger for specific sound recognition, and the detected value of the sound direction.

Although not shown in FIGS. 1 to 4, a sensor that detects environmental information also acquires information. For example, it includes a temperature sensor that detects the temperature around the imaging device 101 at a predetermined period, and an atmospheric pressure sensor that detects changes in the air pressure around the imaging device 101. Further, an illuminance sensor that detects the brightness around the imaging device 101, a humidity sensor that detects the humidity around the imaging device 101, a UV sensor that detects the amount of ultraviolet rays around the imaging device 101, etc. may be provided. . In addition to the detected temperature information, atmospheric pressure information, brightness information, humidity information, and UV information, the amount of temperature change, amount of atmospheric pressure change, amount of brightness change, and humidity change is calculated by calculating the rate of change at a predetermined time interval from various detected information. It is used to judge the amount of ultraviolet rays and the amount of change in ultraviolet rays for automatic shooting, etc., which will be described later.

When the detection values of various sensors are read in S502, the process advances to S503. In S503, it is detected whether communication is instructed from an external device, and if there is a communication instruction, communication with the external device is performed. For example, it receives a remote operation from the external device 301 via wireless LAN or BLE, and transmits and receives data such as an audio signal, an image signal, a compressed audio signal, and a compressed image signal. Also, are there operational instructions from the external device 301 such as shooting with the imaging device 101, voice command registration data transmission, predetermined position detection notifications based on GPS position information, location movement notifications, and transmission/reception of learning data? Load something.

Further, the various sensors that detect the environmental information described above may be installed in the imaging device 101 or may be installed in the external device 301, and in that case, the environmental information is also read via BLE. When the communication from the external device is read in S503, the process advances to S504.

In S504, mode setting determination is performed. The mode set in S504 is determined and selected from the following.

(1) Manual shooting mode [Mode judgment conditions]
When it is detected that a command to set the manual shooting mode has been sent from the external device 301, the manual shooting mode is set.

[In-mode processing]
In manual shooting mode processing (S506), panning, tilting, or zooming is driven according to the user's input contents, and still images are taken or moving image recording is started according to the user's shooting instructions.

(2) Automatic shooting mode [Mode judgment conditions]
Based on each detection information (image, sound, time, vibration, location, physical change, environmental change) set by learning described later, the elapsed time since switching to automatic shooting mode, past shooting information, etc. When it is determined that automatic photography should be performed, automatic photography mode is set.

[In-mode processing]
In automatic photographing mode processing (S508), the subject is automatically searched for by driving panning, tilting, and zooming based on each detection information (image, sound, time, vibration, location, change in body, and change in environment). Then, when it is determined that the timing is right for the user's preferred photography, photography is automatically performed. Note that when a user gives a shooting instruction, shooting is performed in accordance with the instruction.

(3) Learning mode [Mode judgment conditions]
If it is determined that learning should be performed based on the elapsed time since the last learning process, the information associated with images that can be used for learning, the number of learning data, etc., the system will be set to learning mode. . Alternatively, this mode is also set when there is an instruction to set learning parameters via communication from the external device 301.

[In-mode processing]
In the learning mode process (S510), learning is performed in accordance with the user's preferences. Learning is performed in accordance with the user's preferences using a neural network based on information such as each operation on the external device 301 and learning data notifications from the external device 301. Information on each operation on the external device 301 includes, for example, image acquisition information from the imaging device 101, information on manual editing instructions given via a dedicated application, and judgments input by the user for images in the imaging device. There is value information.

(4) File automatic deletion mode [Mode judgment conditions]
If it is determined that automatic file deletion should be performed based on the elapsed time since the last automatic file deletion, the amount of data recorded on the recording medium 219 or non-volatile memory 214, and whether or not learning data has been updated, the file will be automatically deleted. mode is set.

[In-mode processing]
In the automatic file deletion mode process (S512), files to be automatically deleted are specified from among the images in the recording medium 219 or the nonvolatile memory 214 based on the tag information of each image, the date and time of photographing, and the like.

Note that details of the automatic shooting mode processing, learning mode processing, and automatic file deletion mode processing will be described later.

In S505 of FIG. 5, it is determined whether the mode setting determination in S504 is set to manual shooting mode. If it is determined that the mode is manual shooting mode, the process advances to S506 and manual shooting mode processing is performed. In the manual shooting mode process, the imaging device 101 is driven according to the user's input content as described above. When the process ends, the process returns to S502.

On the other hand, if it is determined in S505 that the mode is not manual shooting mode, the process proceeds to S507, where it is determined whether the mode setting is automatic shooting mode, and if it is automatic shooting mode, the process proceeds to S508, where automatic shooting mode processing is performed. . When the process ends, the process returns to S502. If it is determined in S507 that the mode is not automatic shooting mode, the process advances to S509.

In S509, it is determined whether the mode setting is learning mode, and if it is learning mode, the process advances to S510, where learning mode processing is performed. When the process is completed, the process returns to S502 and the process is repeated. If it is determined in S509 that the mode is not learning mode, the process advances to S511.

In S511, it is determined whether the mode setting is automatic file deletion mode, and if it is automatic file deletion mode, the process advances to S512, where automatic file deletion mode processing is performed. When the process is completed, the process returns to S502 and the process is repeated. If it is determined in S511 that the file automatic deletion mode is not in effect, the process returns to S502 and repeats the process.

<Automatic shooting mode processing>
The details of the automatic shooting mode process in S508 in FIG. 5 will be explained using FIG. 6. As described above, the following processing is controlled by the control circuit 221 of the imaging apparatus 101 in this embodiment.

In S601, the image processing circuit 207 performs image processing on the image signal captured by the imaging unit 206 to generate an image for object recognition. Subject recognition such as person and object recognition is performed on the generated image.

When recognizing a person, the face or human body of the subject is detected. In face detection processing, a pattern for determining a person's face is determined in advance, and a portion of the area included in the captured image that matches this pattern can be detected as a person's face image. . In addition, the reliability level indicating the certainty that the subject's face is true is also calculated at the same time. The reliability is calculated from, for example, the size of the face area in the image, the degree of matching with the face pattern, and the like.

Similarly, for object recognition, it is possible to recognize objects that match a pre-registered pattern. There is also a method of extracting characteristic objects by using a histogram of hue, saturation, etc. in a captured image. In this case, the distribution derived from the histogram of hue, saturation, etc. of the image of the subject captured within the shooting angle of view is divided into multiple sections, and the captured image is classified for each section. be done.

For example, a histogram of multiple color components is created for a captured image, the mountain-shaped distribution range is divided, images captured in areas that belong to the same combination of sections are classified, and the image area of the subject is recognized. be done.

By calculating the evaluation value for each image area of the recognized object, it is possible to determine the image area of the object with the highest evaluation value as the main object area.

With the above method, each subject information can be obtained from the imaging information.

In S602, a shake correction amount is calculated. Specifically, first, the absolute angle of attitude change of the imaging device 101 is calculated based on the angular velocity and acceleration information acquired by the device shake detection circuit 209. Then, a shake correction angle for moving the tilt rotation unit 104 and pan rotation unit 105 in an angular direction that cancels out the absolute angle is determined and used as the shake correction amount.

In S603, the state of the imaging device 101 is determined. What kind of vibration/motion state the imaging device 101 is currently in is determined based on the angle and amount of movement detected using angular velocity information, acceleration information, GPS position information, and the like. For example, when the imaging device 101 is attached to a car to take a picture, subject information such as surrounding scenery changes greatly depending on the distance traveled.

Therefore, it is possible to determine whether the device is in a "vehicle moving state" in which the device is attached to a car or the like and is moving at a high speed, and can be used for automatic object search, which will be described later.

Furthermore, it is determined whether the change in angle is large or not, and it is determined whether the imaging device 101 is in a "stationary shooting state" in which there is almost no shaking angle. When the camera is in the "stationary shooting state", it can be considered that there is no change in the angle of the imaging device 101 itself, so it is possible to search for a subject for stationary shooting. Furthermore, if the angle change is relatively large, it is determined that the camera is in a "hand-held state", and a hand-held object search can be performed.

In S604, subject search processing is performed. The control circuit 221 performs area division around the entire periphery around the position of the imaging device 101 (the origin O in FIG. 1 is the position of the imaging device). For each divided area, an importance level indicating the priority for searching is calculated depending on the objects existing within the area and the scene situation of the area.

The importance level based on the situation of the subject is based on, for example, the number of people in the area, the size of the faces of the people, the orientation of the faces, the certainty of face detection, the facial expressions of the people, and the results of personal identification of the people. Calculated by In addition, the importance level depending on the situation of the scene includes, for example, general object recognition results, scene discrimination results (blue sky, backlight, sunset view, etc.), sound levels and voice recognition results from the direction of the area, and motion detection within the area. Calculated from information etc. Further, in the state determination of the imaging device 101 (S603), the vibration state of the imaging device 101 is detected, and the importance level can also be changed depending on the vibration state. For example, if it is determined that the camera is in a "stationary shooting state", the face of a specific person will be searched based on the high-priority subject (for example, the user of the imaging device) registered in face recognition. When detected, the importance level is determined to be high. In addition, automatic shooting, which will be described later, will also be performed with priority given to the face of a specific person, so even if the user of the imaging device 101 spends a lot of time carrying around the imaging device and taking pictures, he or she may remove the imaging device. By placing it on a desk, etc., you can keep many images of the user. At this time, since it is possible to search by panning and tilting, it is possible to leave an image of the user or a group photo of many faces by simply setting it up appropriately, without having to consider the placement angle of the imaging device. can.

Note that with only the above conditions, as long as there is no change in each area, the area with the highest level of importance will remain the same, and as a result, the area to be searched will remain the same forever. Therefore, the importance level is changed depending on past shooting information. Specifically, the importance level of an area that has been designated as a search area for a predetermined period of time is lowered, and the importance level of an area that has been photographed in S610, which will be described later, is lowered for a predetermined period of time. good.

Once the importance level of each area is calculated as described above, an area with a high importance level is determined as a search target area. Then, the pan/tilt search target angle required to capture the search target area at the angle of view is calculated.

In S605, pan/tilt driving is performed. Specifically, the pan/tilt drive amount is calculated by adding the drive angle in control sampling based on the image blur correction amount and the pan/tilt search target angle. The lens barrel rotation drive circuit 205 drives and controls the tilt rotation unit 104 and the pan rotation unit 105, respectively.

In S606, the zoom unit 201 is controlled to drive the zoom. Specifically, the zoom is driven according to the state of the search target object determined in S604. For example, when the subject to be searched is a person's face, if the face on the image is too small, it is less than the minimum detectable size and cannot be detected, and there is a risk that the face will be lost. In such a case, control is performed to increase the size of the face on the image by zooming toward the telephoto side. On the other hand, if the face in the image is too large, the subject is likely to move out of the field of view due to movement of the subject or the imaging device 101 itself. In such a case, control is performed to reduce the size of the face on the screen by zooming to the wide-angle side. By performing zoom control in this manner, a state suitable for tracking the subject can be maintained.

In S604 to S606, a method of searching for an object using pan/tilt or zoom driving was described, but the object search may also be performed using an imaging system that uses a plurality of wide-angle lenses to take images in all directions at once. In the case of an omnidirectional camera, performing image processing such as object detection using all the signals obtained by imaging as input images requires an enormous amount of processing. Therefore, a configuration is adopted in which a part of the image is cut out and a search process for a subject is performed within the cut out image range. Similar to the method described above, the importance level for each area is calculated, the cutout position is changed based on the importance level, and automatic shooting determination, which will be described later, is performed. This makes it possible to reduce power consumption through image processing and to search for objects at high speed.

In S607, it is determined whether or not there is a user (manual) shooting instruction while the automatic shooting mode is set. If there is a shooting instruction, the process advances to S610. In this case, the user (manually) may issue a shooting instruction by pressing the shutter button, by lightly tapping (tapping) the housing of the imaging device 101 with a finger, inputting a voice command, or by instructions from an external device. It's okay. The shooting instruction by tap operation is a shooting instruction method in which the vibration when the user taps the housing of the imaging device 101 is detected by the device shaking detection circuit 209 as a continuous high-frequency acceleration in a short period of time, and is used as a trigger for shooting. . Voice command input is a photography instruction method in which when a user utters a command word (for example, "take a photo") instructing a predetermined photography, the voice processing circuit 212 recognizes the voice and uses it as a trigger for photography. The shooting instruction from an external device is a shooting instruction method in which a shutter instruction signal transmitted from a smartphone or the like connected to the imaging device 101 via a dedicated application is used as a trigger, for example.

If there is no photographing instruction in S607, the process advances to S608 and automatic photographing determination is performed. In the automatic photographing determination, it is determined whether or not to perform automatic photographing.

The determination as to whether or not to perform automatic photographing is performed based on a neural network, which is a type of machine learning. As an example of a neural network, an example of a network using a multilayer perceptron is shown in FIG. Neural networks are used to predict output values from input values, and by learning input values and model output values for those inputs in advance, they can be used to predict output values from new input values. On the other hand, it is possible to estimate the output value based on the learned model. Note that the learning method will be described later.

In FIG. 7, 701 and its vertically aligned circles are neurons of the input layer, 703 and its vertically aligned circles are neurons of the intermediate layer, and 704 are neurons of the output layer. Arrows such as 702 indicate connections connecting each neuron. In neural network-based determination, input layer neurons are given features based on the subject in the current field of view, the scene, and the state of the imaging device, and calculations are performed based on the forward propagation law of a multilayer perceptron. Obtain the value output from the output layer through . Then, if the output value is equal to or greater than the threshold value, a determination is made to perform automatic imaging.

The characteristics of the subject include the current zoom magnification, the general object recognition result at the current angle of view, the face detection result, the number of faces in the current angle of view, the degree of smile/eyes closed of the face, the face angle, and the face recognition ID. The number, the viewing angle of the subject person, the scene discrimination result, the detection result of a specific composition, etc. are used. It also uses information such as the elapsed time since the last shooting, the current time, GPS location information, the amount of change from the previous shooting location, the current audio level, the person making the sound, whether there is applause or cheering, etc. It's okay. Further, vibration information (acceleration information, state of the imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), etc. may be used. This feature is converted into a numerical value within a predetermined range and given to each neuron in the input layer as a feature amount. Therefore, each neuron in the input layer is required as many as the number of features used above.

Note that in the judgment based on this neural network, the output value changes by changing the connection weight between each neuron through a learning process to be described later, so that the judgment result can be adapted to the learning result.

Further, the determination that automatic photography is performed also changes depending on the activation conditions read in S501 of FIG. For example, in the case of activation by tap detection or activation by a specific voice command, there is a very high possibility that the operation is because the user currently wants to take a picture. Therefore, the shooting frequency is set to increase.

In S609, if a determination is made to photograph by the automatic photographing determination in S608, the process advances to S610; if not, the photographing mode process is ended and the process advances to S502 in FIG.

In S610, photographing is started. At this time, in the case of manual shooting, shooting is performed as a still image or in a shooting method manually set by the user, and in the case of automatic shooting, shooting is started at the timing determined in S608. At this time, autofocus control is performed by the focus drive control circuit 204. Further, using an aperture control circuit, a sensor gain control circuit, and a shutter control circuit (not shown), exposure control is performed so that the subject has appropriate brightness. Further, after photographing, the image processing circuit 207 performs various image processing such as auto white balance processing, noise reduction processing, and gamma correction processing to generate an image.

Note that when a predetermined condition is satisfied during this photographing, a method may be adopted in which the imaging device 101 notifies the person to be photographed that the photograph will be taken, and then photographs the person. The notification method may be, for example, using sound from the audio output circuit 216 or LED lighting from the LED control circuit 222, or a motion motion that visually guides the subject's line of sight by driving panning and tilting. You may do so. The predetermined conditions include, for example, the number of faces within the angle of view, the degree of smiling faces and closed eyes, the gaze angle and face angle of the subject person, the face authentication ID number, the number of people registered for personal authentication, etc. . In addition, the general object recognition results at the time of shooting, the scene discrimination results, the elapsed time since the previous shooting, the shooting time, whether the current position based on GPS information is a scenic spot, the audio level at the time of shooting, and the voice output. These include the presence or absence of a person present, whether there is applause, cheers, etc. The information also includes vibration information (acceleration information, status of the imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), and the like. By performing notification photography based on these conditions, it is possible to leave a favorable image of the camera's line of sight in a highly important scene.

It also has multiple predetermined conditions, and can change the audio, the LED lighting method (color, flashing time, etc.), and the pan/tilt motion method (movement method and drive speed) according to each condition. May be changed.

In S611, editing processing such as processing the image generated in S610 and adding it to a video is performed. Specifically, image processing includes cropping processing based on a person's face and focus position, image rotation processing, and addition processing of various effects such as HDR (high dynamic range) effects, bokeh effects, and color conversion filter effects. It is. For image processing, a plurality of images may be generated based on the image generated in S610 by a combination of the above processes, and the images may be saved separately from the image generated in S610. Further, regarding video processing, processing may be performed in which a captured video or still image is added to an already generated edited video while applying special effects such as slide, zoom, and fade. In the editing in S611, information on the photographed image or various information detected before photographing can be determined based on a neural network, and the image processing method can be determined. Further, in this judgment process, the judgment conditions can be changed by a learning process described later.

In S612, processing is performed to generate learning data from the captured images. Here, information used for learning processing, which will be described later, is generated and recorded. Specifically, in this captured image, the zoom magnification at the time of shooting, general object recognition results at the time of shooting, face detection results, number of faces in the captured image, degree of smiling face/eyes closed, face angle, face These include the authentication ID number, the viewing angle of the subject person, etc. In addition, the scene determination results, elapsed time since the previous shooting, shooting time, GPS location information, amount of change from the previous shooting position, audio level at the time of shooting, person making a voice, whether there is applause, cheers, etc. etc. Further, the information includes vibration information (acceleration information, state of the imaging device), environmental information (temperature, atmospheric pressure, illuminance, humidity, amount of ultraviolet rays), video shooting time, whether or not manual shooting was instructed, and so on. It also calculates a score, which is the output of a neural network that quantifies the user's image preferences.

This information is generated and recorded as tag information in the photographed image file. Alternatively, the information on each photographed image may be written in the nonvolatile memory 214 or stored in the recording medium 219 in a list format as so-called catalog data.

In S613, past shooting information is updated. Specifically, the number of shots for each area explained in S608, the number of shots for each person registered for personal authentication, the number of shots for each subject recognized by general object recognition, and the number of shots for each scene for scene discrimination, The count of the number of images corresponding to the image taken this time is increased by one.

<Learning mode processing>
Next, learning tailored to the user's preferences in this embodiment will be explained.

In this embodiment, a neural network as shown in FIG. 7 is used, and a machine learning algorithm is used to perform learning in accordance with the user's preference in the learning processing circuit 217. The learning processing circuit 217 uses, for example, NVIDIA's Jetson TX2. Neural networks are used to predict output values from input values, and by learning the actual values of input values and actual values of output values in advance, they can predict output values for new input values. value can be estimated. By using a neural network, the system performs learning tailored to the user's preferences for the above-mentioned automatic shooting and subject search.

It also performs subject registration (facial recognition, general object recognition, etc.), which also serves as feature data input to the neural network.

Learning for automatic shooting in this embodiment will be explained. Automatic shooting involves learning to automatically take images that match the user's preferences. As explained using the flowchart of FIG. 6, the process of generating learning data (S612) is performed after photographing. Learning is performed by selecting images to be trained using a method described later, and changing the connection weights between neurons of the neural network based on learning data included in the images.

Next, the learning method will be explained. Learning methods include "learning within the imaging device" and "learning by linking with communication equipment." The method of learning within the imaging device will be described below.

The learning within the imaging device in this embodiment includes the following methods.

(1) Learning based on detection information when user instructs photography As described in S607 to S613 in FIG. 6, in this embodiment, the imaging device 101 can perform two types of photography: manual photography and automatic photography. If there is a manual shooting instruction in S607 (performed based on the three determinations as described above), in S612 information indicating that the captured image is a manually captured image is added. Furthermore, if it is determined in S609 that automatic photography is ON and a photograph is taken, information is added to the photographed image in S612 indicating that the photographed image is an automatically photographed image. Furthermore, information indicating that the image is a manually shot image is also added to the image shot in the manual shooting mode process of S506.

When manual photography is performed here, there is a very high possibility that the photography is based on the user's favorite subject, favorite scene, favorite place, or time interval. Therefore, learning is performed based on each feature data obtained during manual photography and the learning data of the photographed image.

It also learns about extracting features from captured images, registering personal authentication, registering facial expressions for each individual, and registering combinations of people from the information detected during manual photography. Also, from the detection information during the subject search, for example, learning is performed to change the importance of nearby people and objects based on the facial expressions of the personally registered subjects.

Next, learning by cooperation with an external communication device in this embodiment will be explained. In this embodiment, there are the following methods for learning through cooperation with an external communication device.

(2) Learning by acquiring images with an external communication device As explained in FIG. 3, the imaging device 101 and the external device 301 have communication means for communicating 302 and 303. Images are mainly sent and received through communication 302, and the external device 301 can acquire images in the imaging device 101 through communication via an internal dedicated application. Further, the external device 301 can view thumbnail images of image data stored in the imaging device 101 via an internal dedicated application. Thereby, the user can select an image he/she likes from among the thumbnail images, confirm the image, and operate the acquisition instruction to acquire the image to the external device 301.

At this time, since the image selected by the user is acquired by a transmission instruction (transmission request), there is a very high possibility that the acquired image is an image of the user's preference. Therefore, the acquired image is determined to be an image to be learned, learning data is generated from the acquired image in the same manner as in S612 of FIG. 6, and learning is performed based on this learning data. This allows the user to perform various types of learning as desired.

An example of operation will be explained. FIG. 8 shows an example in which images in the imaging device 101 are viewed through a dedicated application of the external device 301, which is a smart device. Thumbnail images (804 to 809) of image data stored in the imaging device 101 are displayed on the display device 407, and the user can select and acquire an image that he or she likes. At this time, change button icons 801, 802, and 803 for changing the display method are provided. When the change button icon 801 is pressed, the display order is changed to date and time priority display mode, and images in the imaging device 101 are displayed on the display device 407 in the order of shooting date and time. For example, image 804 is displayed with a newer date and time, and image 809 is displayed with an older date and time. When the change button icon 802 is pressed, the mode is changed to the recommended image priority display mode. Based on the score calculated in S612 of FIG. 6, which is the evaluation result of determining the user's preference for each image, the images in the imaging device 101 are displayed on the display device 407 in order of the highest score. For example, image 804 is displayed with a high score, and image 809 is displayed with a low score. By pressing the change button icon 803, a person or object can be specified, and by subsequently specifying a specific person or object, only the specific object can be displayed.

Settings of the change button icons 801 to 803 can also be turned on at the same time. For example, if all settings are turned on, only the specified subject will be displayed, and the image with the latest shooting date and time will be given priority, and Images with higher scores will be prioritized and displayed.

In this way, since the user's preferences are learned for captured images as well, it is possible to easily extract only the user's preferred images from a large number of captured images with a simple confirmation process.

(3) Learning by inputting judgment values into images via an external communication device As explained above, the imaging device 101 and the external device 301 have communication means, and the images can be viewed through a dedicated application within the external device 301. Here, the configuration may be such that the user assigns a score to each image. The user can give a high score (for example, 5 points) to an image that he or she likes, and a low score (for example, 1 point) to an image that he or she does not like. 101 will be configured to learn. The score of each image is used for relearning together with learning data within the imaging device. The neural network inputs the feature data from the specified image information and learns so that the output of the neural network approaches the score specified by the user.

In the present embodiment, the user inputs a score to a captured image via the external device 301, but a configuration may also be adopted in which the user operates the imaging device 101 and directly inputs a score to the image. In that case, for example, the imaging apparatus 101 is provided with a touch panel display, and the user presses a GUI button displayed on the touch panel display to set a mode for displaying captured images. Similar learning can be performed by the user inputting points for each image while checking the captured images.

(4) Learning by changing parameters with an external communication device As explained above, the imaging device 101 and the external device 301 have a communication means, and the learning parameters currently set in the imaging device 101 can be changed. It can be transmitted to the external device 301 and stored in the storage circuit 404 of the external device 301. Examples of learning parameters include connection weights between neurons in a neural network, selection of subjects to be input to the neural network, and the like. Further, the configuration is such that learning parameters set in a dedicated server can be obtained via a dedicated application in the external device 301 via the public line control circuit 406 and set as learning parameters in the imaging device 101. shall be. With this, it is possible to save the parameters at a certain point in the external device 301 and return the learning parameters by setting them in the imaging device 101, or to save the learning parameters held by other users via a dedicated server. It is also possible to acquire and set it in the own imaging device 101.

Next, the learning processing sequence will be explained. In the mode setting determination at S504 in FIG. 5, it is determined whether or not learning processing should be performed. If learning processing is to be performed, it is determined that the learning mode is selected, and learning mode processing is performed at S510.

The learning mode determination conditions will be explained. Whether or not to shift to the learning mode is determined based on the elapsed time since the last learning process, the number of information that can be used for learning, whether a learning process instruction has been received via a communication device, etc. FIG. 9 shows a process flow for determining whether to shift to learning mode, which is determined in the mode setting determination process of S504.

When the learning mode determination is instructed to start in the mode setting determination process of S504, the process of FIG. 9 starts. In S901, it is determined whether there is a learning instruction from an external device. The determination of the presence or absence of a learning instruction here is a determination of whether or not there was an instruction to set learning parameters, as in <(4) Learning by changing parameters with an external communication device>. In S901, if there is a learning instruction from the external device 301, the process advances to S907, sets the learning mode determination to TRUE, sets to perform the process of S510, and ends the learning mode determination process. If there is no learning instruction from the external device 301 in S901, the process advances to S902.

In S902, the elapsed time TimeN since the last learning mode process was performed is acquired, and the process advances to S903. In S903, the new number of data to be learned DN (the number of images designated to be learned during the elapsed time TimeN since the previous learning process was performed) is acquired, and the process advances to S904. In S904, a threshold value DT is calculated from TimeN. Alternatively, a table for obtaining the threshold value DT from TimeN may be prepared. For example, the threshold value DTa when TimeN is smaller than a predetermined value is set larger than the threshold value DTb when TimeN is larger than the predetermined value, and the threshold value is set to become smaller as time passes. As a result, even when there is little learning data, learning is performed again when a large amount of time has elapsed, thereby making it easier for the imaging device to change to the learning mode when the usage time becomes longer. Note that it is preferable to increase the threshold value DT so as not to shift to the learning mode for a while after the learning mode processing is performed.

When the threshold value DT is calculated in S904, the process proceeds to S905, and it is determined whether the number of data to be learned DN is greater than or equal to the threshold value DT. If the data number DN is equal to or greater than the threshold DT, the process advances to S906 and DN is set to 0. Thereafter, the process proceeds to S907, where the learning mode determination is set to TRUE, the process of S510 is set to be performed, and the learning mode determination process is ended.

If the data number DN is less than the threshold DT in S905, the process advances to S908. In S908, since there is neither a registration instruction from the external device 301 nor a learning instruction from the external device, and the number of learning data is less than a predetermined value, the learning mode determination is set to FALSE and the process of S510 is set not to be performed. Then, the learning mode determination process ends.

Next, the processing in the learning mode processing (S510) will be explained. FIG. 10 shows a detailed flow of the learning mode process.

When the learning mode is determined in S509 of FIG. 5 and the process proceeds to S510, the process of FIG. 10 starts. In S1001, it is determined whether there is a learning parameter setting instruction from the external device 301. If there is an instruction to set learning parameters from the external device 301, the process advances to S1006, where the learning parameters transmitted from the external device are set in each determiner (such as connection weights between neurons of a neural network), and the process advances to S1007. If there is no learning instruction from the external device 301 in S1001, the process advances to S1002.

In S1002, one of the learning data is selected and machine learning is performed. This learning data includes learning data generated from captured images that have been added with information indicating that they are manually captured images, learning data generated from images acquired with an external communication device, and learning data determined via an external communication device. Contains learning data generated from captured images with input values. Learning is performed using a method such as error backpropagation or gradient descent, the connection weights between neurons in the neural network are recalculated, and the parameters of each judger are changed. If the user has given a score to the image that generated the learning data, learning will be performed taking that score into consideration.

In S1003, it is determined whether learning has been performed using all the learning data prepared for machine learning. If there is still learning data remaining, the process returns to S1002, and if learning has been performed using all the learning data, the process proceeds to S1004.

In S1004, the learning parameters obtained by machine learning are stored in the nonvolatile memory 214 in association with the reference number of times.

In S1005, the latest learning parameters stored in S1004 are set in each determiner (such as connection weights between neurons of a neural network), and the process advances to S1007.

In S1007, the images in the recording medium 219 or the non-volatile memory 214 are re-scored (re-evaluated). In this embodiment, scores are assigned to all captured images stored in the recording medium 219 or non-volatile memory 214 based on new learning results, and automatic editing is performed according to the assigned scores. and automatic file deletion. In other words, when re-learning or setting learning parameters from an external device is performed, the scores of captured images also need to be updated. Therefore, in S1007, a new score is recalculated for the photographed image stored in the recording medium 219 or the nonvolatile memory 214, and when the process is completed, the learning mode process is ended. Note that the recalculation for assigning a new score may be performed in response to a user's instruction.

Although the present embodiment has been described based on a configuration in which learning is performed within the imaging device 101, the learning function is provided on the external device 301 side, data necessary for learning is sent to the external device 301, and only the learning function is provided on the external device 301 side. A similar learning effect can also be achieved with a configuration that performs learning. In that case, as explained in <(4) Learning by changing parameters with external communication device> above, parameters such as connection weights between neurons of the neural network learned on the external device side are transmitted to the imaging device 101. It is also possible to have a configuration in which learning is performed by setting as follows.

Further, a configuration may be adopted in which both the imaging device 101 and the external device 301 have learning processing functions. For example, a configuration may be adopted in which learning data held by the external device 301 is communicated to the imaging device 101 at the timing when learning mode processing is performed within the imaging device 101, and learning is performed by merging the learning parameters.

<File automatic deletion process>
Next, details of automatic file deletion processing will be explained using FIG. 11. FIG. 11 is a flowchart showing the operation of the automatic file deletion process executed in S512 of FIG.

In S511 of FIG. 5, it is determined that automatic file deletion should be performed based on the elapsed time since the last automatic file deletion, the amount of data recorded on the recording medium 219 or nonvolatile memory 214, whether learning data has been updated, etc. If so, the process advances to S512, and the automatic file deletion mode operation shown in FIG. 11 is executed.

In S1102, it is determined whether the learning data referenced when the previous automatic file deletion process was executed has been updated. If the learning data, that is, the learning parameters have been updated, the user's preferences may have been updated, and the deletion candidates may also have changed. Therefore, if the learning parameters have been updated since the last time, the process advances to S1103, and if they have not been updated, the process advances to S1108.

In S1103, since the learning parameters have been updated since the previous automatic file deletion process, there is a possibility that the user's preferences have also been updated. Therefore, the scores assigned in S1007 of FIG. 10 are referred to for all the captured images stored in the recording medium 219 or the non-volatile memory 214, and the scores of the captured images whose scores are lower than the first reference value are determined. Select an image as a deletion candidate image. This first reference value may be a constant value, a value set by the user, or a value set according to the histogram distribution of scores of all captured images. Note that to simplify the explanation, all captured images stored in the recording medium 219 or non-volatile memory 214 are covered, but instructions from the user, date, or folder in which the images are stored are The target photographed images may be limited by, for example, the following.

In S1104, if the image that remains without being deleted is equal to or greater than the second reference value, the process advances to S1105, and if it is less than the second reference value, the process advances to S1106. This step S1104 is a process for determining whether there are too many images to be recorded. Therefore, this second reference value may be a fixed value, but as the number of shots increases, it is desirable that the number of images to be recorded increases, so as the number of shots increases, this second reference value may also be increased. is desirable.

In S1105, deletion candidate images are selected using evaluation criteria different from those in S1103. Since the score based on the learning data has already been calculated, for example, the similarity between the captured images that were not selected as deletion candidate images in S1103 is compared, and if there is a captured image with a high degree of similarity, one of the captured images ( For example, similar images with lower scores are deleted. Examples of criteria for determining similarity include, but are not limited to, the location where the image was taken, the date and time the image was taken, the composition of the subject in the captured image, and the type of subject in the captured image. Note that if you want to keep all images with high scores, S1104 and S1105 may be omitted.

In S1106, if there are images that were not deleted in the previous file automatic deletion process but became deletion candidates in the current file automatic deletion process, the user is notified that these images have become deletion candidates. . Examples of notification processing include a method of notification using a speaker sound emitted from the audio output circuit 216, a method of notification using an LED lighting light from the LED control circuit 222, and the like. Further, as a method for checking the deletion candidates, there may be a method of displaying a list of deletion candidates recorded in the camera 101 via a dedicated application of the external device 301, which is a smart device. For example, a list of images to be deleted is displayed, and images that have not been targeted for deletion in the past are displayed in color or with icons so that the user can visually recognize deletion candidates. This is because the user may believe that images that were not deleted in the past automatic file deletion process will not be deleted in the current automatic file deletion process. Then, if there is an instruction from the user, the image is excluded from deletion targets in accordance with the instruction.

Then, in S1107, the image to be deleted is deleted from the recording medium 219 or the nonvolatile memory 214, and the operation of this flow ends.

Note that if the learning parameters have not been updated in S1102, the process advances to S1108. In S1108, among the captured images stored in the recording medium 219 or the non-volatile memory 214, only the captured images obtained after the previous file automatic deletion process are targeted, and in S1007 of FIG. See the assigned score. Then, a photographed image whose score is lower than the first reference value described above is selected as a deletion candidate image. This is because if the learning parameters have not been updated, the determination result will not change even if the determination is performed on a captured image that has already been determined. Furthermore, by reducing the number of captured images that refer to scores, the processing load can be reduced.

Then, in step S1109, the image to be deleted is deleted from the recording medium 219 or the nonvolatile memory 214, and the operation of this flow ends.

Note that when selecting images to be deleted, the images may be selected and deleted based on the image viewing record or image transfer record in addition to the score assigned in S1007 of FIG. 10. This is because, for example, there is no disadvantage to deleting images that the user has not viewed for a long time or images that have already been transferred.

Further, although the step of notifying the user of deletion candidate images in S1106 is provided, a configuration may be adopted in which automatic deletion is performed without notification. Further, the conditions for proceeding to S1103 may include not only the case where the learning parameters have been updated but also the case where the remaining capacity of the recording medium 219 or the nonvolatile memory 214 is below a threshold value.

As described above, in this embodiment, the range of images that are subject to automatic file deletion processing is wider when the learning parameters have been updated than when the learning parameters have not been updated. In other words, even if the captured images stored in the recording medium 219 or the non-volatile memory 214 are under the same conditions, if the learning parameters are updated when performing automatic file deletion processing, the learning parameters will not be updated. The number of images that are subject to automatic file deletion processing increases compared to the case where no files are deleted. In addition, in this embodiment, an example was given in which deletion candidate images are selected from only captured images obtained after the previous automatic file deletion process if the learning parameters have not been updated. However, it is not limited to this.

For example, in automatic file deletion processing, a predetermined number of images with the highest scores may be left and the remaining images may be deleted. With such specifications, even if the learning parameters are not updated, as the number of captured images increases, there is a possibility that images that were not deleted in the previous automatic file deletion process will be deleted in the next automatic file deletion process. There is. In such a case, if the learning parameters have been updated, all captured images stored in the recording medium 219 will be targeted in the next automatic file deletion process. On the other hand, if the learning parameters have not been updated, the next automatic file deletion process will use images captured after the previous automatic file deletion process and images acquired before the previous automatic file deletion process. Only some of the captured images are targeted. The captured images that are obtained after the previous file automatic deletion process and are subject to deletion are those that have a relatively low score among the captured images obtained before the previous file automatic deletion process. It is an image. Therefore, among the captured images obtained before the previous automatic file deletion process, only those captured images with relatively low scores may be subjected to the next automatic file deletion process.

Note that although the present embodiment has been described assuming a configuration in which files are automatically deleted within the imaging apparatus 101, the present invention is not limited to this. The file may be automatically deleted in the external device 301 to which the captured image is transferred from the imaging device 101 or in the server to which the captured image is transferred via the external device 301. In this case, the imaging apparatus 101 does not need to automatically delete images, and can be configured to transfer all captured images to the external device 301 or the server.

Further, if the external device 301 or the server has sufficient capacity to record captured images, a configuration may be adopted in which captured images are automatically selected for the user to view instead of automatically deleting files. In other words, when a user tries to view a captured image without automatically deleting the captured image, only the images with a high score will be displayed to the user instead of the images with a low score based on the learning parameters. The specification can be such that it can be viewed. In this case, in steps S1103, S1105, and S1108, a non-display candidate image is selected instead of selecting a deletion candidate image. In this case, since the photographed image remains without being deleted, even if the image had a low score in the past, it is possible to salvage an image that would have a high score in the evaluation based on the new learning parameters.

As explained above, according to the present embodiment, images to be recorded or displayed can be appropriately selected according to updates of learning data, so if the user's preferences change, the image to be recorded or displayed can be appropriately selected. You can now select images. This makes it possible to keep images that the user likes and automatically delete images that are not. This makes it possible to automatically delete images that the user does not like. The inconvenience of not being able to take pictures can be resolved.

(Other embodiments)
The present invention also provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device reads the program. This can also be achieved by executing a process. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

101: Imaging device, 102: Lens barrel, 104: Tilt rotation unit, 105: Pan rotation unit, 217: Learning processing circuit, 219: Recording medium, 221: Control circuit, 301: External device

Claims (13)

a storage means for storing an image obtained by imaging a subject;
A learning means that uses machine learning to learn the user's favorite images;
evaluation means for acquiring a score that is an evaluation value of the image stored in the storage means based on learning parameters of the learning means;
selection means for selecting deletion candidates from the images stored in the storage means with reference to the score ;
Deletion means for deleting the image selected by the selection means from the storage means;
An image processing device comprising :
The image processing device has a learning mode in which the learning means learns a user's favorite image, and an automatic deletion mode in which the deletion means automatically deletes images stored in the storage means,
In the learning mode, if there is an instruction to set a learning parameter of the learning means from an external device, the evaluation means sets the learning parameter, and if there is no instruction, after setting the learned learning parameter. , re-evaluate the scores for all images,
In the automatic deletion mode, when the learning parameters of the learning means have been updated , the selection means selects deletion candidates by referring to the scores for all images, and when the learning parameters have not been updated , An image processing device that selects deletion candidates by referring to the score for only images obtained after a previous automatic file deletion process . 5. The selection means selects the deletion candidate based on at least one of a viewing record of the image stored in the storage means and a record of transfer to an external device, in addition to the score. 1. The image processing device according to 1 . 3. The image processing apparatus according to claim 1 , wherein the learning means sets learning parameters based on an image photographed according to a manual photographing instruction from a user. 3. The image processing apparatus according to claim 1 , wherein the learning means sets learning parameters based on an image requested to be transmitted from an external device. 3. The learning means learns based on a score, which is an evaluation value of each image, input by a user and stored in the storage means, and sets learning parameters. image processing device. The selection means selects similar images from among the images other than the selected images when the number of images other than those selected by the selection means among the images stored in the storage means is equal to or greater than a predetermined threshold. The image processing apparatus according to any one of claims 1 to 5 , wherein the image processing apparatus further selects at least one of the similar images. 7. The image processing apparatus according to claim 6 , further comprising a notification unit that visibly notifies a user of the image further selected by the selection unit. The selection means determines whether or not to select the deletion candidate from the images stored in the storage means based on the amount of data of the images stored in the storage means. The image processing apparatus according to any one of claims 1 to 7 . an imaging means for imaging a subject;
An image processing device according to any one of claims 1 to 8 ,
An imaging device comprising: Claim 9 , further comprising a control unit that searches for a subject based on at least one of images, sounds, time, vibrations, changes in the body, and past photographic information, and causes the imaging unit to automatically perform photographing. The imaging device described in . a storage step of storing an image obtained by imaging the subject in a storage means;
A learning process that uses machine learning to learn the user's favorite images;
an evaluation step of acquiring a score, which is an evaluation value of the image stored in the storage means, based on the learning parameters in the learning step;
a selection step of selecting deletion candidates from images stored in the storage means with reference to the score ;
a deletion step of deleting the image selected in the selection step from the storage means;
An image processing method comprising :
The image processing method has a learning mode in which the user's favorite images are learned in the learning step, and an automatic deletion mode in which the images stored in the storage means are automatically deleted in the deletion step,
In the evaluation step, in the learning mode, if there is an instruction to set learning parameters for the learning step from an external device, the learning parameters are set, and if there is no instruction, after setting the learned learning parameters. , re-evaluate the scores for all images,
In the selection step, in the automatic deletion mode, if the learning parameters of the learning step have been updated , deletion candidates are selected by referring to the scores for all images, and if they have not been updated , An image processing method characterized in that deletion candidates are selected by referring to the scores for only images obtained after a previous automatic file deletion process . A program for causing a computer to function as each means of the image processing apparatus according to claim 1 . A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8 .

Images