JP7433788B2 - Control device, imaging device, control method, program - Google Patents

Description

The present invention relates to a control device for controlling focus adjustment means.

BACKGROUND ART Line-of-sight detection technology for detecting a user's gaze point is known. In the field of imaging devices, it is expected that the use of line of sight detection technology will make it easier to reflect the user's intentions in controlling the imaging device.

Patent Document 1 describes an imaging device that uses information regarding a user's line of sight to select a focus detection point. In the imaging device of Patent Document 1, it is possible to more easily reflect the user's intention in focus adjustment.

Japanese Patent Application Publication No. 2002-62470

However, in the method described in Patent Document 1, only information regarding the user's line of sight and defocus information are used for focus adjustment. That is, in the method described in Patent Document 1, what the user is gazing at is not taken into consideration in the control, so the user's intention may not be fully reflected.

An object of the present invention is to realize focus adjustment control that reflects the user's intention.

The control device of the present invention includes: a display acquisition unit that acquires display information regarding a display of a display unit that displays an image related to a subject; a line-of-sight acquisition unit that acquires line-of-sight information regarding a user's line of sight; and the line-of-sight information and the display information. and control means for controlling the focus adjustment means using an optical system , the image of the subject is acquired using an optical system and an image sensor, and the display means displays an image of the subject in a first area of the display area. displays an image, and displays information regarding the settings of the optical system and the image sensor in a second area of the display area, and the control means controls the control means to display the information about the settings of the optical system and the image sensor in a second area of the display area, and when the user's line of sight is on the second area, the control means If it is determined that focus adjustment is not to be performed for the position corresponding to the user's current line of sight, and a shooting preparation instruction is given during the determined period, the camera will not use the information regarding the user's current line of sight. , the focus adjustment means is controlled, and the control means receives time-series data regarding the line-of-sight information and the display information, and corresponds to the user's current line-of-sight in consideration of the correlation between the data. Using a neural network that outputs information regarding whether or not to perform focus adjustment for a position, it is determined whether or not to perform focus adjustment for the position corresponding to the current line of sight of the user. If a shooting preparation instruction is issued during a period in which it is determined that focus adjustment is to be performed on a position corresponding to the user's current line of sight, control of the focus adjustment means is performed using information regarding the user's current line of sight. A control device characterized by performing the following.

According to the present invention, it is possible to realize focus adjustment control that reflects the user's intention.

1 is an external view of a digital camera. FIG. 2 is a functional block diagram showing the configuration of a digital camera. It is a figure showing the display of a display part. FIG. 3 is a diagram illustrating the movement of the user's line of sight in the display area of the display unit. FIG. 3 is a diagram illustrating the movement of the user's line of sight in the display area of the display unit. It is a flowchart regarding AF control. It is an external view of a smartphone. FIG. 7 is a diagram showing a display when a photographing application is activated.

Hereinafter, embodiments of an imaging device having a control device of the present invention will be described based on the accompanying drawings.

[Example 1]
FIG. 1 shows a rear external view of a digital camera 100 as an example of an imaging device according to this embodiment. The display unit (display means) 28 is an electronic viewfinder that displays images and various information. The shutter button 61 is used to instruct photography. The connector 112 is a connector for connecting the digital camera 100 to the connection cable 111 for connecting to an external device such as a personal computer or a printer. The operation unit 70 is made up of operation members such as various switches, buttons, and a touch panel that accept various operations from the user. The controller wheel 73 is a rotatable operating member included in the operating section 70 . The power switch 72 is a push button for switching between power on and power off.

The recording medium 200 includes, for example, a memory card, a hard disk, etc., and stores images etc. taken by the digital camera 100. The recording medium slot 201 is a slot for removably storing the recording medium 200. The recording medium 200 stored in the recording medium slot 201 can communicate with the digital camera 100, and can perform recording and reproduction. The lid 202 is the lid of the recording medium slot 201. FIG. 1 shows a state in which the lid 202 is opened and a portion of the recording medium 200 is taken out and exposed from the recording medium slot 201.

FIG. 2 is a block diagram showing a configuration example of the digital camera 100 according to this embodiment. In FIG. 2, the photographic lens 103 includes an optical system including a zoom lens, a focus lens, and an aperture, and a focus adjustment means that changes the focused position by driving the focus lens. The shutter 101 is a mechanical shutter. The image sensor 22 is composed of a CCD, CMOS element, etc. that converts an optical image into an electrical signal. A/D converter 23 converts analog signals into digital signals.

The image processing section 24 performs predetermined pixel interpolation, resizing processing such as reduction, and color conversion processing on the data from the A/D converter 23 or the data from the memory control section 15. Further, the image processing unit 24 performs predetermined calculation processing using the captured image data. Based on the calculation results obtained by the image processing section 24, the system control section 50 performs exposure control and distance measurement control. As a result, TTL (through-the-lens) AF (autofocus) processing, AE (automatic exposure) processing, and EF (flash pre-emission) processing are performed. Further, the image processing section 24 includes a display acquisition means 24a and a line of sight acquisition means 24b. The display acquisition means 24a acquires information regarding the display of the display element 28a of the display section 28 (display information). The image processing unit 24 includes, for example, a CPU, FPGA, GPU, and the like.

The display acquisition means 24a acquires, for example, image (video) data currently displayed by the display element 28a as display information. The line-of-sight acquisition means 24b analyzes image data of the user's eyeballs transmitted from a line-of-sight detection unit 28d, which will be described later, and detects line-of-sight information regarding the user's line of sight. The line of sight information may include the coordinates of the user's gaze point and the movement of the user's gaze point. The movement of the user's gaze point is, for example, a change in the coordinates of the user's gaze point acquired at regular intervals. In the digital camera 100 of this embodiment, display information and line-of-sight information are transmitted to the system control unit 50.

Output data from the A/D converter 23 is written into the memory 32 via the image processing section 24 and the memory control section 15, or only via the memory control section 15. The memory 32 stores, for example, image data obtained by the image sensor 22 and converted into digital data by the A/D converter 23, and image data to be displayed on the display unit 28. The memory 32 has a storage capacity sufficient to store a predetermined number of still images, a predetermined period of moving images, and audio.

Furthermore, the memory 32 also serves as an image display memory (video memory). The D/A converter 13 converts the display data stored in the memory 32 into an analog signal and supplies it to the display section 28 . In this way, the display image data written in the memory 32 is displayed on the display unit 28 via the D/A converter 13. The display unit 28 performs display according to the analog signal from the D/A converter 13 on a display such as an LCD. A digital signal that has been A/D converted by the A/D converter 23 and stored in the memory 32 is converted into an analog signal by the D/A converter 13, and is sequentially transferred to the display section 28 for display. Live view display (LV display) can be performed. Hereinafter, the image displayed in live view will be referred to as an LV image.

The display section 28 includes a display element 28a, an eyepiece optical system 28b, an eyepiece detection section 28c, and a line of sight detection section 28d. The display element 28a is composed of a liquid crystal panel or the like, and displays image data for display supplied from the memory control section 15. The display image data includes an image of the subject space obtained through the photographic lens 103 and information such as various settings of the digital camera 100. The user of the digital camera 100 can enlarge and observe the image data displayed on the display element via the eyepiece optical system 28b. The eye proximity detection unit 28c detects the approach (eye proximity) of an eye (object). The system control unit switches display/non-display of the display unit 28 according to the state detected by the eye proximity detection unit 28c. The line of sight detection unit 28d acquires image data of the user's eyeball via the eyepiece optical system 28b. Image data of the user's eyeballs acquired by the line of sight detection section 28d is transmitted to the image processing section 24 via the A/D converter 29.

The nonvolatile memory 56 is a memory as an electrically erasable/recordable recording medium, and for example, an EEPROM or the like is used. The nonvolatile memory 56 stores constants, programs, etc. for the operation of the system control unit 50. The program here refers to a computer program for executing various flowcharts described later in this embodiment.

The system control unit 50 is at least one processor or circuit that controls the entire digital camera 100. By executing the program recorded in the nonvolatile memory 56 described above by the system control unit 50, each process of the present embodiment, which will be described later, is executed. For example, a RAM is used as the system memory 52. In the system memory 52, constants and variables for the operation of the system control unit 50, programs read from the nonvolatile memory 56, and the like are expanded. The system control unit 50 also performs display control by controlling the memory 32, the D/A converter 13, the display unit 28, and the like.

The system timer 53 is a timer that measures the time used for various controls and the time of a built-in clock. The shutter button 61 and the operation unit 70 are operation means for inputting various operation instructions to the system control unit 50.

The shutter button 61 has a first shutter switch 62 and a second shutter switch 64. The first shutter switch 62 is turned on when the shutter button 61 provided on the digital camera 100 is pressed half-way (instruction to prepare for photographing) during operation, and generates a first shutter switch signal SW1. Focus adjustment by AF (autofocus) is started by the first shutter switch signal SW1. Focus adjustment is performed by issuing a command to measure a distance to a subject that is the target of focus adjustment, and then issuing a command to move the focus lens to a position where the subject is in focus. The second shutter switch 64 is turned ON when the operation of the shutter button 61 is completed, that is, when the shutter button 61 is fully pressed (photographing instruction), and generates a second shutter switch signal SW2. The system control unit 50 starts a series of photographic processing operations from reading out signals from the image sensor 22 to writing image data to the recording medium 200 in response to the second shutter switch signal SW2.

Each operating member of the operating unit 70 is assigned an appropriate function for each scene by selecting and operating various function icons displayed on the display unit 28, and acts as various function buttons. Examples of the function buttons include an end button, a return button, an image forwarding button, a jump button, a narrowing down button, and an attribute change button. For example, when the menu button is pressed, various setting menu screens are displayed on the display unit 28. The user can intuitively perform various settings using the menu screen displayed on the display unit 28, four-way buttons (up, down, left, right), and the SET button.

The power supply control unit 80 includes a battery detection circuit, a DC-DC converter, a switch circuit for switching the block to be energized, and the like, and detects whether or not a battery is attached, the type of battery, and the remaining battery level. Further, the power supply control section 80 controls the DC-DC converter based on the detection result and the instruction from the system control section 50, and supplies the necessary voltage to each section including the recording medium 200 for a necessary period. The power supply unit 30 includes a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li battery, an AC adapter, or the like.

The recording medium I/F 18 is an interface with a recording medium 200 such as a memory card or a hard disk. The recording medium 200 is a recording medium such as a memory card for recording photographed images, and is composed of a semiconductor memory, an optical disk, a magnetic disk, or the like. The communication unit 54 can be connected to external equipment via a wireless or wired cable, and transmits and receives video signals, audio signals, and the like. The communication unit 54 can also be connected to a wireless LAN (Local Area Network) and the Internet. The communication unit 54 can transmit images captured by the image sensor 22 (including through-the-lens images) and images recorded on the recording medium 200, and can also receive image data and other various information from external devices. can.

Note that although the digital camera 100 of this embodiment is a so-called integrated digital camera in which an optical system and an image sensor are integrated, the present invention is not limited thereto. The invention is also applicable to so-called interchangeable lens cameras in which a lens device (interchangeable lens) having an optical system can be replaced with a camera body having an image sensor. In the case of an interchangeable lens camera, the lens device and the camera body are configured to be able to communicate with each other. Further, the focus adjustment means is provided in the lens device, and the system control section is provided in the camera body, and the system control section of the camera body is configured to be able to control the focus adjustment means by communicating with the lens device.

Next, the features of this embodiment will be described. The digital camera 100 of this embodiment performs AF control using display information and line-of-sight information in order to realize AF control that reflects the user's intention. Here, "AF control" means controlling at least one of a series of AF operations in the digital camera 100. The series of AF operations includes selection of an AF target area, distance measurement for the selected area, and driving of the focus lens within the photographic lens 103.

When photographing using the digital camera 100, various settings of the digital camera 100 are displayed on the display unit 28 together with the image of the subject. While checking the subject displayed on the display unit 28, the user can perform framing to obtain a desired composition, select a subject to focus on, and change the settings of the digital camera 100.

In this way, since the user makes adjustments regarding photography using the digital camera 100 while looking at the display on the display unit 28, the user's intentions are likely to be reflected in the user's line of sight. For example, in framing when the subject is a stationary object, the user tends to gaze at the main subject and then at the desired position to place the main subject after framing. When framing a moving subject, there is a tendency to frame while tracking the subject's movement with the line of sight. When performing focus adjustment, the user tends to gaze at the main subject until the focus adjustment operation is completed. When configuring the digital camera 100, the user tends to focus on the entire image and the area where information related to the settings of the digital camera 100 displayed on the display unit 28 is displayed.

Although the user's line of sight information alone can only provide information about the position (coordinates) that the user gazed at on the display element 28a of the display unit 28, when the line of sight information and display information are used together, it is possible to obtain information about what is ahead of the user's line of sight. You can see if it is displayed. Therefore, by controlling AF using display information and line-of-sight information, the digital camera 100 can infer the user's intention from the position and movement of the user's line of sight with respect to the information displayed on the display unit 28. .

In the AF control in the digital camera 100 of this embodiment, the system control unit 50 performs an AF operation on the position where the user's line of sight is currently displayed on the display unit 28 based on display information and line of sight information. It is determined whether or not the AF operation is possible (hereinafter simply referred to as whether or not the AF operation is possible). That is, the system control unit 50 determines whether the subject that the user wants to focus on can be considered to exist ahead of the user's current line of sight based on the display information and the line of sight information. do. The AF operation to be performed differs depending on whether the AF operation is determined to be "possible" or not.

FIG. 3 is an example of information displayed in the display area 300 of the display element 28a of the display unit 28. In the digital camera 100 of this embodiment, the display acquisition means acquires the image (video) displayed in the display area 300 as display information. That is, FIG. 3 is also an example of display information acquired by the display acquisition means 24a.

The display area 300 includes a first area 301 that displays a subject and a second area 302 that displays information regarding settings of the digital camera 100. In the example of FIG. 3, information other than the image of the subject is displayed in the second area 302. In this embodiment, the second area 302 displays information related to the settings of the digital camera 100, such as the shooting mode, shutter speed, F value, exposure, ISO sensitivity, and remaining battery power.

In the first area 301, an AF frame 303 is displayed as an index representing a subject area that is a focus adjustment target (target). The example shown in FIG. 3 shows an initial state before AF control of this embodiment is performed, and the AF frame is arranged at the center of the screen. Note that the AF frame 303 does not need to be displayed in the display area 300.

FIG. 4 is a diagram showing the movement of the user's line of sight within the display area 300. That is, FIG. 4 is also an example of line-of-sight information acquired by the line-of-sight acquisition means 24b.

FIG. 4(a) shows the movement of the user's line of sight when the user moves the line of sight within the screen and then stops on a specific subject (bird). In such a case, it is highly likely that the user started to look at the main subject after looking around the first area 301 of the display area 300. In other words, there is a high possibility that the user is trying to select a bird as the main subject.

Further, FIG. 4(b) shows the movement of the user's line of sight when the user gazes at a specific object (bird) and then moves his or her line of sight to another position. Since there is a subject (leaf) at the point where the user shifts the gaze point, the user is thinking of changing the main subject from the bird to the leaf in the first area 301 of the display area 300. Probability is high.

In the case of the movement of the line of sight as shown in FIGS. 4(a) and 4(b), there is a high possibility that the user's intention regarding focus adjustment is reflected in the user's line of sight. That is, there is a high possibility that the current position of the user's line of sight and the position the user wants to focus on match. Therefore, the system control unit 50 allows the AF operation. Note that FIGS. 4(a) and 4(b) only show an example in which the AF operation is "possible", and the line of sight movements that make the AF operation "possible" are shown in FIGS. 4(a) and (b). Not limited to cases.

Figure 5(a) shows the movement of the user's line of sight when the user focuses on a specific subject (a bird) and then stagnates on another subject, or when the user moves the line of sight without tracking a certain subject. Showing. In such a case, the user may be trying to change the main subject from the bird to another subject in the first area 301 of the display area 300, but it is still not possible to select which subject should be the main subject. It's highly likely that you haven't. Therefore, in the case of such a movement of the line of sight, it is considered that the user has no intention to change the subject on which he or she wants to focus at the moment.

Moreover, FIG. 5(b) shows the movement of the line of sight when the user gazes at the second area 302 after gazing at a specific subject (bird). In such a case, there is a high possibility that the user is not trying to change the main subject from the bird to another subject, but is instead focusing on the information displayed in the second area 302. Therefore, in the case of such a movement of the line of sight, it is considered that the user has no intention to change the subject on which he or she wants to focus at the moment.

In the case of movements of the line of sight as shown in FIGS. 5A and 5B, there is a high possibility that the user's intention regarding focus adjustment is not reflected in the user's current line of sight. That is, there is a high possibility that the current position of the user's line of sight and the position on which the user wants to focus are different. Therefore, the system control unit 50 determines that the AF operation is not possible. Note that FIGS. 5(a) and 5(b) only show an example in which the AF operation is determined to be "no", and the line of sight movements that indicate that the AF operation is not performed are shown in FIGS. 5(a) and (b). Not limited to cases.

As described above, by using line of sight information and display information, it is possible to infer the user's intention with high accuracy.

In this embodiment, a pre-trained machine learning model that receives at least line-of-sight information and display information is used to determine whether the AF operation is possible. It is preferable to use a neural network with at least three or more layers as the machine learning model. This is because the display information and line of sight information used as input have a high degree of freedom, and the amount of features to be used for AF control can be enormous.

Note that by performing AF control based on the movement of the subject and the movement of the user's line of sight, it is possible to more easily reflect the user's intention. Therefore, it is preferable that both display information and line-of-sight information can be treated as time-series data. Therefore, it is preferable to use a neural network having recursive connections (recurrent neural network) as the learning model. Recursive combination is a neural network that adds the output of the middle layer when data at time (t-1) in a certain time series data is input to the input of the middle layer when data at time t is input. Refers to structure. By having recursive coupling, it becomes possible to perform estimation that takes into account the correlation of time series data. The data length of the input time series data is not particularly limited.

In this embodiment, a multilayer neural network with recursive connections is used as a machine learning model. The neural network of this embodiment can input time-series data of line-of-sight information and time-series data of display information. It is designed to output a vectorized vector. The learning of the machine learning model of this embodiment includes at least (1) time-series data of line-of-sight information, (2) time-series data of display information, and (3) a vector indicating the determination result of whether AF is possible (if yes, then (1) , 0), and if not (0, 1)). For example, the training data set is based on the transition of line-of-sight information with respect to the transition of display information included in one time-series data, and determines whether there is a subject to be the AF target ahead of the line-of-sight at the last point in the time-series data. This is prepared by determining whether or not it can be considered to be the case. When generating a training data set, the judgment as to whether or not a subject to be AF-targeted can be considered to exist is made by automatically determining the pattern as explained using Figures 4 and 5. You can go, or people can go.

Although the present embodiment describes an example in which a machine learning model is used to determine whether or not an AF operation is possible, the present invention is not limited to this. This may be done by setting a predetermined logic for determining whether or not the AF operation is possible, and determining whether the line-of-sight information and display information match the logic. The logic for determining whether or not AF operation is possible is that if the user's viewpoint is in the second area of the display area 300, AF operation is rejected, and if the movement of the user's viewpoint does not match the movement of the subject. It is possible to set the AF operation to be disabled.

Next, AF control using the result of determination as to whether AF operation is possible will be described.

As a first example, a case where the AF setting is one-shot AF will be described. One-shot AF is a system in which focus is adjusted at the timing when a user issues an instruction to prepare for photographing, and the focus is fixed during the preparation period.

In the case of one-shot AF, if AF control is performed at a timing not intended by the user, the user may feel uncomfortable. Therefore, when the user issues an instruction to prepare for shooting during a period in which the AF operation is determined to be "possible," the system control unit 50 determines whether the user is currently gazing at the AF frame 303, as shown in FIG. displayed in the location where it is estimated to be. Thereafter, the system control unit 50 issues a distance measurement command to the subject within the AF frame 303 and a focus lens drive command.

On the other hand, if the user issues an instruction to prepare for photographing during a period in which the AF operation is determined to be "no", AF control is performed without using information regarding the current user's line of sight. Specifically, the AF frame 303 is displayed not at the position where the user's line of sight is currently, but at the position where the user is estimated to be gazing at a timing when AF operation was determined to be "possible" in the recent past. let Thereafter, the system control unit 50 issues a distance measurement command to the subject within the AF frame 303 and a focus lens drive command.

As a second example of AF operation, a case where the AF setting is continuous AF will be described. Continuous AF is a method of adjusting the focus so that the subject within the AF frame remains in focus. Note that in this embodiment, continuous AF is performed while a shooting preparation instruction is given, but continuous AF may be performed regardless of whether or not a shooting preparation instruction is given.

In the case of continuous AF, the user is expected to move his or her line of sight in hopes of continuing to capture the desired subject. Therefore, it is important to continue displaying the AF frame 303 at the position desired by the user. When continuous AF is being performed, the AF frame 303 is displayed at the position where the user is estimated to be gazing while the AF operation is determined to be "possible". If the position where the user is estimated to be gazing moves during the period in which the AF operation is determined to be "possible", the AF frame 303 is also moved. Then, the system control unit 50 issues a distance measurement command to the subject within the AF frame 303 and a focus lens drive command at predetermined intervals.

On the other hand, when continuous AF is being performed, AF control is performed without using information regarding the current user's line of sight during a period in which the AF operation is determined to be "unavailable." Specifically, the AF frame 303 is displayed not at the position where the user's line of sight is currently, but at the position where the user is estimated to be gazing at a timing when AF operation was determined to be "possible" in the recent past. keep letting it happen. Then, the system control unit 50 issues a distance measurement command to the subject within the AF frame 303 and a focus lens drive command at predetermined intervals.

FIG. 6 is a flowchart regarding AF control in the digital camera 100 of the present embodiment described above. The process shown in FIG. 6 is realized by the system control unit 50 executing a program recorded in the nonvolatile memory 56 using the system memory 52 as a work memory. The process in FIG. 6 is started when the user's line of sight is detected by the line of sight detection unit 28c.

In S601, the display acquisition unit 24a of the image processing unit 24 acquires display information on the display unit 28. Furthermore, the line-of-sight acquisition means 24b acquires line-of-sight information using the output of the line-of-sight detection unit 28d.

In S602, the system control unit 50 uses the display information and the line of sight information to determine whether or not to perform the AF operation by regarding the current user's line of sight as the point of interest. If it is determined in S602 that the AF operation is "possible", the process advances to S603, and if it is determined that the AF operation is "not possible", the process advances to S604.

In S603, the system control unit 50 stores the current user's viewpoint based on the line-of-sight information in the system memory 52 as an AF position to be an AF target (position where an AF frame should be displayed). Note that the AF position may be set to a predetermined initial value (for example, a position including the center of the screen).

In S604, the system control unit 50 determines whether the user has given an instruction to prepare for photographing. If the user has given an instruction to prepare for photographing, the process advances to S605; otherwise, the process advances to S610.

In S605, the system control unit 50 determines whether the AF setting is one-shot AF or continuous AF. If it is one-shot AF, the process advances to S606, and if it is continuous AF, the process advances to S611.

In S606, the system control unit 50 reads the AF position from the system memory 52 and executes an AF operation for the AF position. Specifically, an AF frame is displayed at the AF position within the display of the display section 28, a distance measurement instruction for the AF position is given, and then a focus lens drive command is given based on the distance measurement result.

In S607, the system control unit 50 determines whether the user's instruction to prepare for imaging continues. If the photography preparation instruction is being continued, the process advances to S608, and if the photography preparation instruction has been canceled, the process advances to S610.

In S608, the system control unit 50 determines whether the user has given a shooting instruction. If a photographing instruction has not been given, the process returns to S607, and if a photographing instruction has been given, the process advances to S609. That is, when the AF setting is one-shot AF, the AF operation based on the user's line of sight is not performed while the shooting preparation state continues after the AF operation is performed once. This makes it possible to perform AF control that reflects the user's intentions at the timing that the user intended.

In S609, the system control unit 50 executes a predetermined photographing sequence and acquires an image.

In S610, the system control unit 50 determines whether the user's line of sight is continuously detected. If the user's line of sight is no longer detected, the process ends, and if the user's line of sight is continuously detected, the process returns to S601.

If it is determined in S605 that the AF setting is continuous AF, in S611 the system control unit 50 reads the AF position from the system memory 52 and executes the AF operation for the AF position in the same manner as in S606. .

In S612, the system control unit 50 determines whether the user has given a shooting instruction. If a photographing instruction has not been given, the process advances to S610, and if a photographing instruction has been given, the process advances to S613. That is, when the AF setting is continuous AF, the AF operation based on the user's line of sight continues to be performed while the shooting preparation state continues after the AF operation is performed once. This makes it possible to perform AF control while tracking the subject while reflecting the user's intention.

As described above, the digital camera 100 of this embodiment includes the system control section 50, the image processing section 24, and the display section 28 as a control device for controlling the focus adjustment means. This makes it possible to implement AF control that reflects the user's intentions.

Note that when the AF setting is one-shot AF, it is preferable that the AF frame 303 is not displayed on the display unit 28 during a period when the user does not give an instruction to prepare for shooting. This is because if the AF frame is displayed at a position not intended by the user or if the AF frame 303 moves during a period when the user does not intend to start AF, the user may feel uncomfortable.

Furthermore, if the AF position changes due to a change in the line of sight between objects at greatly different distances, the object that the user originally wanted to focus on may become significantly blurred due to defocus. In this case, not only is AF control performed that is not intended by the user, but it also takes a long time to refocus on the subject intended by the user. For this reason, distance information of the subject is used to determine whether or not AF operation is possible, and if the user's line of sight moves from a different position by more than a predetermined distance within a predetermined time, the position corresponding to the current user's line of sight is It is preferable not to perform focus adjustment. Thereby, the out-of-focus rate for the subject intended by the user can be reduced.

Further, as in this embodiment, when the display area 300 displays a first area 301 in which a subject is displayed and a second area 302 in which information regarding settings of the digital camera 100 is displayed, the user can It is conceivable to frequently move the line of sight to the area 302. Therefore, if it is determined that the user is looking at the second area, it is preferable that the AF operation not be performed on the current user's viewpoint on the display area 300. Thereby, the out-of-focus rate for the subject intended by the user can be reduced.

Although FIG. 3 shows an example in which the first area 301 and the second area 302 are vertically divided and arranged, the display of the display area 300 is not limited to this. A second area where information regarding the settings of the digital camera 100 is displayed may be superimposed on the first area where the image of the subject is displayed.

Preferably, the configuration is such that the user can select an area to focus on via the operation unit 70, and when the area is selected via the operation unit 70, the processing shown in FIG. 6 is not performed. . That is, when a region is selected via the operation unit 70, it is preferable that AF control using line-of-sight information and display information is not performed. This is because the user's intention is more strongly reflected in the operation via the operation unit 70 than in the user's line of sight. Thereby, the user's intention can be reflected in focus adjustment control with higher precision.

[Example 2]
Next, an imaging device according to a second embodiment will be described.

FIG. 7 shows a smartphone 700 as an imaging device of this embodiment.

7(a) shows the front of the smartphone 700, and FIG. 7(b) shows the back of the smartphone 700.

The smartphone 700 includes a display section 701, an operation section 702, a front imaging section 703, and a back imaging section 704.

The display unit 701 is composed of a liquid crystal panel or the like, and includes a touch sensor. That is, the user can operate the smartphone 700 by touching the display unit 701 in addition to the operation unit 702.

The front imaging unit 703 is mainly used to photograph the user. The rear imaging unit 704 is mainly used to photograph a subject facing the user. Both the front imaging unit 703 and the back imaging unit 704 include an optical system and an imaging element (not shown).

FIG. 8 shows the smartphone 700 of this embodiment in a state in which a photography application has been activated.

The display unit 701 of the smartphone 700 when an application for photographing is started includes a first area 801 that displays an image (video) of the subject, and a second area 802 that displays virtual buttons and the like for operating the application. Is displayed.

At this time, the smartphone 700 acquires the image (video) displayed on the display unit 701 as display information. Furthermore, the smartphone 700 analyzes the user's face obtained through the front imaging unit 703 as line-of-sight information, and acquires the user's line-of-sight information.

Then, the smartphone 700 determines whether AF operation is possible or not, and performs AF control in the same manner as in the first embodiment.

In this way, the effects of the present invention can also be obtained with the form of this embodiment.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various combinations, modifications, and changes can be made within the scope of the invention.

28 Display section (display means)
24a Display acquisition means 24b Line of sight acquisition means 50 System control unit (control means)

Claims (10)

a display acquisition means for acquiring display information regarding the display of the display means for displaying an image related to the subject;
a line-of-sight acquisition means for acquiring line-of-sight information regarding the user's line of sight;
a control means for controlling a focus adjustment means using the line of sight information and the display information;
has
The image of the subject is acquired using an optical system and an image sensor,
The display means displays an image of the subject in a first area of the display area, and displays information regarding settings of the optical system and the image sensor in a second area of the display area,
The control means determines that, when the user's line of sight is on the second area, the focus adjustment is not performed for the position corresponding to the user's current line of sight;
If a shooting preparation instruction is issued during the determined period, controlling the focus adjustment means without using information regarding the user's current line of sight;
The control means inputs time-series data regarding the line-of-sight information and the display information, and determines whether or not to perform focus adjustment with respect to a position corresponding to the current line-of-sight of the user, taking into account the correlation between the data. determining whether or not to perform focus adjustment to a position corresponding to the user's current line of sight, using a neural network that outputs information regarding the current line of sight of the user;
If a shooting preparation instruction is given during a period in which it is determined that focus adjustment is to be performed on a position corresponding to the user's current line of sight, the focus adjustment means uses information regarding the user's current line of sight. A control device characterized by controlling. On the display means, an indicator indicating that the camera is subject to focus adjustment is displayed during a period when the photography preparation instruction is being issued;
The control device according to claim 1, wherein the indicator is not displayed during a period when the photographing preparation instruction is not given. The control device further controls a focus adjustment means using distance information of the subject,
According to claim 1 or 2, when the user's line of sight moves from a position where the distance is different by a predetermined value or more, focus adjustment is not performed to the position corresponding to the user's current line of sight. Control device as described. If it is determined that the focus adjustment is not to be performed on the position corresponding to the current line of sight of the user, the control means may be configured to adjust the focus on the position corresponding to the current line of sight of the user. The control device according to any one of claims 1 to 3, wherein the control device controls the focus adjustment means based on information regarding the user's line of sight when determined. 2. The control means controls the focus adjustment means based on the instruction when an instruction regarding a position for focus adjustment is given through an operation via an operation section. 4. The control device according to any one of 4 . 6. The control device according to claim 1, wherein the display means is an electronic viewfinder. 7. The control device according to claim 1, wherein the line of sight information is information regarding the user's line of sight with respect to the display on the display means. an imaging unit including an optical system and an imaging element;
a focus adjustment means for adjusting the focus of the imaging section;
A control device according to any one of claims 1 to 7 ,
An imaging device comprising: obtaining display information regarding a display of a display means for displaying an image related to a subject;
obtaining gaze information regarding the user's gaze;
controlling a focus adjustment means using the line of sight information and the display information;
has
The image of the subject is acquired using an optical system and an image sensor,
The display means displays an image of the subject in a first area of the display area, and displays information regarding settings of the optical system and the image sensor in a second area of the display area,
In the step of controlling the focus adjustment means, if the user's line of sight is on the second area, it is determined that the focus adjustment is not performed on the position corresponding to the user's current line of sight. ,
If a shooting preparation instruction is issued during the determined period, controlling the focus adjustment means without using information regarding the user's current line of sight;
In the step of controlling the focus adjustment means, time-series data regarding the line-of-sight information and the display information is input, and the focus adjustment means is adjusted to a position corresponding to the user's current line-of-sight by taking into account the correlation between the data. Determining whether or not to perform focus adjustment with respect to a position corresponding to the current line of sight of the user using a neural network that outputs information regarding whether to perform focus adjustment;
If a shooting preparation instruction is given during a period in which it is determined that focus adjustment is to be performed on a position corresponding to the user's current line of sight, the focus adjustment means uses information regarding the user's current line of sight. A control method characterized by controlling. A program for causing a computer to execute the control method according to claim 9 .

Images