WO2020137602A1

WO2020137602A1 - Imaging device, imaging method, and program

Info

Publication number: WO2020137602A1
Application number: PCT/JP2019/048877
Authority: WO
Inventors: 高弘佐藤
Original assignee: ソニー株式会社
Priority date: 2018-12-28
Filing date: 2019-12-13
Publication date: 2020-07-02
Also published as: JP2024019284A; US11539892B2; US20230276120A1; EP3904956A4; US20220060635A1; JPWO2020137602A1; EP3904956A1

Abstract

This technology relates to an imaging device, an imaging method, and a program which make it possible to easily perform imaging with attention paid to a subject or a specific portion of the subject. This imaging device displays a previous announcement frame that previously announces a specific region as a focusing object on an image acquired by an imaging unit according to the kind of a subject. This technology is applicable to an imaging device.

Description

Imaging device, imaging method, and program

The present technology relates to an image capturing apparatus, an image capturing method, and a program, and particularly to an image capturing apparatus, an image capturing method, and a program capable of easily performing image capturing focusing on a subject or a specific portion of the subject.

Patent Document 1 proposes an imaging device that detects a pupil region, which is a region of a person's pupil, and focuses on the detected pupil region.

International Publication No. 2015/045911

The imaging device described in Patent Document 1 detects only the human pupil region, and it was difficult to detect the animal pupil region.

The present technology has been made in view of such a situation, and makes it possible to easily perform imaging by focusing on a subject or a specific portion of the subject.

The image pickup device according to one aspect of the present technology includes a display control unit that displays a notice frame that gives a notice of a specific area to be focused on the image acquired from the image pickup unit according to the type of subject.

In one aspect of the present technology, a notice frame that gives advance notice of a specific area to be focused is displayed on the image acquired from the imaging unit according to the type of subject.

It is a figure showing an example of processing which detects a pupil of a person. It is a figure which shows the example of the process which detects a pupil of an animal. It is a figure which shows the instruction|indication method of the focus start in the case of the animal pupil detection mode. It is a figure which shows the instruction|indication method of the focus start in the case of the animal pupil detection mode. It is a block diagram showing an example of main composition of an imaging device to which this art is applied. It is a block diagram which shows the structural example of a digital signal processing part. It is a figure which shows the selection method when several pupils are detected in a person's pupil detection mode. It is a figure which shows the selection method when several pupils are detected in the pupil detection mode of an animal. It is a figure which shows the relationship between the direction of a subject's face, and the number of the pupil regions detected. 7 is a flowchart illustrating an image capturing process in a human pupil detection mode. FIG. 11 is a flowchart illustrating an image pickup process in a person's pupil detection mode following FIG. 10. It is a flowchart explaining the imaging process of the pupil detection mode of an animal. FIG. 13 is a flowchart illustrating an imaging process in an animal pupil detection mode following FIG. 12. 13 is a flowchart illustrating a pupil region selection process of step S114 of FIG.

Hereinafter, modes for carrying out the present technology will be described. The description will be given in the following order.
1. Outline of this technology 2. Configuration example of imaging device 3. 3. Selection method when multiple pupils are detected Operation example of imaging device

<1. Overview of this technology>
An imaging device according to an embodiment of the present technology has a detection mode for each specific region that detects a region of a specific region of a subject and uses the detected region of the specific region for focusing. Examples of the detection mode for each specific part include a human pupil detection mode and an animal pupil detection mode.

FIG. 1 is a diagram showing an example of processing for detecting the pupil of a person.

FIG. 1 shows a screen displayed on the display unit of the imaging device when the human eye detection mode is set. The screen on the left side of FIG. 1 is a screen displayed before the user performs an operation for instructing to start focusing. The screen on the right is a screen that is displayed after the user performs an operation to instruct the start of focusing. The face of the person is displayed on the screen.

When the human pupil detection mode is set, as shown on the left side of FIG. 1, a face area overlapping the focus frame F is detected in the face area that is a face area, and the detected face area is detected. A face notice frame PF for advance notice is displayed. In the example on the left side of FIG. 1, the face notice frame PF is displayed so as to partially overlap the focus frame F. The focus frame F is a focus setting frame for setting the focus.

In this state, when the user performs an operation for instructing to start focusing, as shown on the right side of FIG. 1, the pupil region, which is the pupil region, is detected for the face region, and the detected pupil region is detected. Is displayed instead of the face notice frame PF. In addition, focusing is performed on the pupil region surrounded by the pupil frame AE.

FIG. 2 is a diagram showing an example of processing for detecting the pupil of an animal.

FIG. 2 shows a screen displayed on the display unit of the imaging device when the animal pupil detection mode is set. The screen on the left side of FIG. 2 is a screen displayed before the user performs an operation for instructing to start focusing. The screen on the right is a screen that is displayed after the user performs an operation to instruct the start of focusing. The face of an animal (cat) is displayed on the screen.

When the animal pupil detection mode is set, as shown on the left side of FIG. 2, the pupil region is detected for the inside and outside of the focus frame F, and the pupil notice frame for notifying the detected pupil region is made. PE is displayed. At this time, the detection of the pupil area is performed by giving priority to the inside of the focus frame F.

In this state, when the user performs an operation for instructing to start focusing, a pupil frame AE indicating the pupil region is displayed in place of the pupil notice frame PE in the pupil region, as shown on the right side of FIG. In addition, focusing is performed on the pupil region surrounded by the pupil frame AE.

The pupil frame AE is displayed in a different display method from the pupil frame PE, for example, the pupil frame PE is displayed in a white frame and the pupil frame AE is displayed in a green frame. The same applies to the face preview frame PF and the pupil frame AE.

As described above, when the human pupil detection mode is set, the face area is detected before the operation of instructing the start of focusing is performed by the user, and the face notice frame PF is added to the detected face area. Is displayed.

On the other hand, when the animal's pupil detection mode is set, the pupil area is detected before the operation of instructing the start of focusing is performed by the user, and the pupil notice frame PE is displayed in the detected pupil area. It

By displaying the notice frame before the user gives an instruction to start focusing, it is possible to know in advance that an intended position or an unintended position has been detected. It is possible to choose not to do it. As a result, it becomes possible to easily perform image pickup focusing on a specific part of the subject.

3 and 4 are diagrams showing a focusing start instruction method when the animal pupil detection mode is set.

In the image pickup apparatus, the focus start can be instructed by the user performing an operation such as pressing the pupil AF (autofocus) button P, pressing the AF-ON button Q, or pressing the shutter button R halfway. ..

As shown on the left side of FIG. 3, the pupil AF button P is configured as, for example, a central button located at the center of a cross button provided on the back surface of the imaging device. The pupil AF button P is a dedicated button for instructing to start focusing on the pupil, which is a specific part of the subject.

As described above, before the user performs the operation to instruct the start of focusing, the image of the subject on the display unit of the image pickup apparatus displays the pupil notice frame in the detected pupil region.

When the user presses the pupil AF button P in this state, a screen as shown on the right side of FIG. 3 is displayed on the display unit.

As shown on the right side of FIG. 3, the focus frame F and the pupil frame AE are displayed on the screen in an overlapping manner with the image of the cat facing forward. On the right side of FIG. 3, the focus frame F is displayed in the center of the screen near the back of the cat.

When the user presses the pupil AF button P, the range up to about 50% of the screen is set as the pupil region detection range W1 with the center of the focus frame F as a reference. The detection range W1 is not actually displayed. The focus frame F itself may be set in a wide range such as the detection range W1.

When the user presses the pupil AF button P and a pupil region is detected within the detection range W1, the pupil displayed before the pupil AF button P is pressed in the pupil region. Instead of the notice frame, the pupil frame AE is displayed.

Also, the AF-ON button Q is provided, for example, on the upper part of the back surface of the imaging device, as shown on the left side of FIG. The AF-ON button Q is a button for instructing to start focusing on the pupil in the focus frame F.

The shutter button R is provided, for example, on the upper surface of the imaging device, as shown on the left side of FIG. When the shutter button R is half-pressed by the user, it becomes a button for instructing to start focusing on the pupil in the focus frame F, like the AF-ON button Q, and when fully pressed by the user, It becomes a button for instructing the shutter.

When the user presses the AF-ON button Q or the shutter button R is half-pressed in this state, a screen as shown on the right side of FIG. 4 is displayed on the display unit. It

As shown on the right side of FIG. 4, a focus frame F and a pupil frame AE are displayed on the screen, overlaid on the image of the cat facing forward and the dog lying next to the cat. On the right side of FIG. 4, the focus frame F is displayed in the left eye of the cat.

When the user presses the AF-ON button Q, the range from the center of the focus frame F to the vicinity of the focus frame F is set as the pupil area detection range W2. The detection range W2 is not actually displayed.

When the user presses the AF-ON button Q and a pupil area is detected within the detection range W2, the pupil area is displayed before the AF-ON button Q is pressed. The pupil frame AE is displayed instead of the pupil notice frame.

As described above, in the case of the animal pupil detection mode, the image capturing apparatus displays the pupil notice frame for notifying the pupil region before the user performs the operation to instruct the start of focusing. When the user performs an operation for instructing to start focusing, a pupil frame AE indicating a pupil area is displayed, and focusing is performed on the pupil area.

Therefore, it is possible to immediately understand the position of the animal's eyes in the image. As a result, the user can easily perform the imaging focusing on the eyes of the animal.

Note that the positions where the buttons are installed in the imaging device are not limited to the positions shown in FIGS. 3 and 4, and may be other positions.

Also, in the above description, an example of detecting the pupil as a specific part of the subject has been described, but when the subject is a small animal such as a bird, the subject itself may be detected.

<2. Configuration example of imaging device>
FIG. 5 is a block diagram showing a main configuration example of the image pickup apparatus.

The image pickup apparatus 100 shown in FIG. 5 has a detection mode for each specific part of the subject including a human pupil detection mode and an animal pupil detection mode. It should be noted that the image capturing apparatus 100 can be provided with a detection mode according to a specific part, without being limited to a human eye or an animal eye. The user can select and set a desired detection mode from the detection modes of the subject and the specific part of the subject.

As shown in FIG. 5, the imaging device 100 is configured to include a lens 101, a diaphragm 102, an imaging element 103, an analog signal processing unit 104, an A/D conversion unit 105, and a digital signal processing unit 106. The image pickup apparatus 100 is configured to include a lens driver 121, a TG (Timing Generator) 122, a gyro 123, and a system controller 131.

The image pickup apparatus 100 is also configured to include a display unit 141, a storage unit 142, an input unit 143, an output unit 144, a communication unit 145, an operation unit 146, and a drive 147.

The lens 101 adjusts the focus to the subject and collects the light from the in-focus position. The diaphragm 102 adjusts the exposure.

The image sensor 103 captures an image of a subject to obtain a captured image. That is, the image sensor 103 photoelectrically converts light from the subject and outputs it as an image signal to the analog signal processing unit 104. The image sensor 103 can capture a still image or a moving image by such photoelectric conversion.

The analog signal processing unit 104 performs analog signal processing on the image signal obtained by the image sensor 103. The A/D converter 105 performs A/D conversion on the image signal subjected to the analog signal processing to obtain image data which is a digital signal.

The digital signal processing unit 106 performs digital signal processing on the image data obtained by the A/D conversion unit 105. The digital signal processing unit 106 performs, as digital signal processing, at least processing of detecting a subject or a region of a specific portion of the subject from a moving image supplied as image data, and setting a focus region. Hereinafter, the specific part of the subject will be simply referred to as the specific part.

The digital signal processing unit 106 also performs processing such as controlling the display of a frame or the like indicating the region of the subject or the specific part based on the detection result of the region of the subject or the specific part. Details of these processes will be described later.

Note that the content of digital signal processing is arbitrary, and processing other than the above may be performed. For example, the digital signal processing unit 106 may perform color mixture correction, black level correction, white balance adjustment, demosaic processing, matrix processing, gamma correction, and YC conversion as digital signal processing. Further, the digital signal processing unit 106 may perform codec processing, which is processing relating to encoding and decoding of image data, as digital signal processing.

The lens driver 121 drives the lens 101 and the diaphragm 102 to control the focal length or exposure. The TG 122 drives the image sensor 103 by generating a synchronization signal and supplying it to the image sensor 103, and controls image capturing. The gyro 123 is a sensor that detects the position and orientation of the imaging device 100. The gyro 123 outputs the detected sensor information to the A/D conversion unit 105.

The system controller 131 is composed of, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. Controls the processing unit. Further, the system controller 131 receives an operation input by the user based on the signal supplied from the operation unit 146, and performs processing or control corresponding to the operation input.

For example, the system controller 131 can control the focal length or the exposure based on the detection result of the subject or the region of the specific portion supplied from the digital signal processing unit 106.

The display unit 141 is configured as, for example, a liquid crystal display or the like, and displays an image corresponding to the image data stored in the memory of the digital signal processing unit 106. For example, the display unit 141 can display a captured image obtained by the image sensor 103, a stored captured image, and the like.

The storage unit 142 stores the image data stored in the memory of the digital signal processing unit 106. At that time, the storage unit 142 stores the encoded data encoded by the digital signal processing unit 106 in order to reduce the data amount. The encoded data stored in the storage unit 142 is read by the digital signal processing unit 106, decoded, and displayed on the display unit 141, for example.

The input unit 143 has an external input interface such as an external input terminal, and various data (for example, image data or encoded data) supplied from outside the imaging device 100 via the external input interface is input to the digital signal processing unit 106. Output.

The output unit 144 has an external output interface such as an external output terminal, and outputs various data supplied via the digital signal processing unit 106 to the outside of the imaging device 100 via the external output interface.

The communication unit 145 performs predetermined communication, which is at least one of wired communication and wireless communication, with other devices, and exchanges data with other devices via the predetermined communication. For example, the communication unit 145 outputs various data (for example, image data and encoded data) supplied from the digital signal processing unit 106 to another device via predetermined communication. The communication unit 145 also acquires various data from another device via predetermined communication, and outputs the acquired data to the digital signal processing unit 106.

The operation unit 146 is configured by an arbitrary input device such as a key, a button, or a touch panel. The operation part 146 includes the pupil AF button P, the AF-ON button Q, or the shutter button R described above with reference to FIG. 3 or 4. The operation unit 146 receives an operation input from the user and outputs a signal corresponding to the operation input to the system controller 131.

The drive 147 reads out information (programs, data, etc.) stored in a removable recording medium 148 such as a semiconductor memory mounted on itself. The drive 147 supplies the information read from the removable recording medium 148 to the system controller 131. Further, the drive 147 causes the removable recording medium 148 to store information (image data, encoded data, etc.) supplied via the system controller 131 when the writable removable recording medium 148 is attached to itself. You can

The lens 101, the diaphragm 102, and the lens driver 121 described above are formed as a replaceable lens 151 that is detachable (replaceable) from the image pickup apparatus 100 and is a case separate from the image pickup apparatus 100. May be.

FIG. 6 is a block diagram showing a configuration example of the digital signal processing unit 106.

The digital signal processing unit 106 has a memory 211, a subject detection unit 212, a region setting unit 213, a display control unit 214, and a codec processing unit 215.

The memory 211 stores the image data supplied from the A/D conversion unit 105. The image data is, for example, image data of each frame of a moving image or image data of a still image.

The subject detection unit 212 detects a subject or a region of a specific part from the image data stored in the memory 211 based on a signal corresponding to a user's operation input supplied from the system controller 131. The subject detection unit 212 outputs the detection result of the region of the subject or the specific part to the region setting unit 213 and the display control unit 214.

The subject detection unit 212 includes a person detection unit 212-1, an animal detection unit 212-2, and an animal detection unit 212-3.

When the detection mode for each specific part of the subject is the human pupil detection mode, the person detection unit 212-1 detects the face area of the person and displays the detection result of the detected face area on the area setting unit 213 and the display unit. Output to the control unit 214. The person detection unit 212-1 detects the pupil area of the person based on the detection result of the face area when the operation of instructing the start of focusing is performed by the user, and the detection result of the pupil area is set to the area setting unit. 213 and the display control unit 214.

The animal detection units 212-2 and 212-3 differ in the types of animals to be detected. The animal detection units 212-2 and 212-3 detect the pupil region of the target animal when the detection mode for each specific part of the subject is the animal pupil detection mode, and the detection result of the detected animal pupil region is detected. Is output to the area setting unit 213 and the display control unit 214.

For example, methods such as deep learning are used to detect the eyes of animals. When the user performs an operation for instructing to start focusing, the animal detection units 212-2 and 212-3 detect the pupil of the animal according to the focus frame and display the detection result of the detected pupil area of the animal. It is output to the area setting unit 213 and the display control unit 214.

For example, the animal detection unit 212-2 detects the pupil area of animals such as dogs and cats. The animal detection unit 212-3 detects the pupil region of animals such as lizards and frogs. Not only the animal detection units 212-2 and 212-3, but other animal detection units may be provided depending on, for example, the types of animals having the same characteristics at the time of detection.

The area setting unit 213 sets any one of the area of the specific region of the subject detected by the subject detection unit 212 and the region indicated by the focus frame as the focus region according to the detection mode for each specific region of the subject. .. The area setting unit 213 supplies information on the set focusing area to the system controller 131.

The display control unit 214 generates a focus frame according to a signal corresponding to a user's operation input supplied from the system controller 131, and superimposes the focus frame on the image from the memory 211 to display the focus frame on the display unit 141. Information on the focus frame is output to the subject detection unit 212.

In addition, the display control unit 214, based on a signal corresponding to a user's operation input supplied from the system controller 131, a predetermined frame (face frame, notice) corresponding to the face or pupil region detected by the subject detection unit 212. Frame, or pupil frame) is generated. The display control unit 214 superimposes the generated predetermined frame on the image from the memory 211 and causes the display unit 141 to display it. Information on the face frame, the notice frame, or the pupil frame is output to the subject detection unit 212 as necessary.

Note that the display control unit 214 generates an image of a GUI (Graphical User Interface) such as a menu, a button, or a cursor, and displays the image together with a captured image, a captured image, and the like.

The codec processing unit 215 performs processing relating to encoding and decoding of image data of moving images and still images stored in the memory 211.

<3. Selection method when multiple eyes are detected>
FIG. 7 is a diagram showing a selection method when a plurality of pupils are detected in the human pupil detection mode.

FIG. 7A shows a screen displayed before the user performs an operation to instruct the start of focusing. FIG. 7B shows a screen that is displayed after the user performs an operation to instruct the start of focusing.

When the person's pupil detection mode is set, as shown in A of FIG. 7, a face area overlapping the focus frame F is detected in the face area, and a face notice frame PF indicating the detected face area is detected. Is displayed.

In this state, when the user performs an operation of instructing to start focusing, as shown in B of FIG. 7, the pupil area is detected for the face area, and the pupil frame AE indicating the detected pupil area is detected. Is displayed. In addition, focusing is performed on the pupil region surrounded by the pupil frame AE.

At this time, in the detected face area, the pupil area located in front of the imaging device 100 is detected. Therefore, when the left pupil is located in front of the imaging device 100, even if the user adjusts the focus frame F to the right pupil, as shown on the left side of FIG. Will be displayed in the eyes. Further, when the right pupil is located in front of the imaging device 100, the user may or may not adjust the focus frame F to the right pupil, as shown on the right side of FIG. 7B. The pupil frame AE is displayed on the right pupil.

In this way, in the human pupil detection mode, the user can select the face and focus on the front pupil in the selected face area regardless of the position of the focus frame.

FIG. 8 is a diagram showing a selection method when a plurality of pupils are detected in the animal pupil detection mode.

FIG. 8A shows a screen that is displayed before the user performs an operation for instructing to start focusing. FIG. 8B shows a screen displayed after the user performs an operation to instruct the start of focusing.

When the animal pupil detection mode is set, as shown in A of FIG. 8, the pupil region is detected for the inside and outside of the focus frame F, and the pupil notice frame for notifying the detected pupil region is provided. PE is displayed. At this time, the detection of the pupil area is performed by giving priority to the inside of the focus frame F.

However, when a plurality of pupil regions are detected, the pupil notice frame PE is displayed in a pupil region located in front of the imaging device 100 and close to the center (center position) of the focus frame F. The details of the selection of the pupil region will be described later with reference to FIG.

When the operation of instructing to start focusing is performed by the user, as shown in B of FIG. 8, the pupil frame AE is displayed in the selected pupil region, and focusing is performed on the pupil region indicated by the pupil frame AE. Done.

Note that it is possible to set in advance not to display the eye notice frame PE. In this case, it is possible to prevent the expression of the subject from being hidden by the display of the eye notice frame PE.

FIG. 9 is a diagram showing the relationship between the orientation of the subject's face and the number of pupil regions.

FIG. 9 shows images P1 to P7 with the fox as a subject. The foxes shown in images P1 to P7 have different face orientations (angles). The solid line rectangle indicates the front pupil region detected in each image, and the broken line rectangle indicates the back pupil region detected in each image.

Image P1 shows a fox with his face facing to the left. In the image P2, the fox whose face is facing diagonally forward left is shown. In the image P3, a fox whose face is directed slightly diagonally to the front left is shown. The image P4 shows a fox whose face is facing forward. In the image P5, a fox whose face is facing slightly diagonally forward right is shown. The image P6 shows a fox with his face facing diagonally forward right. The image P7 shows a fox with his face facing to the right.

Among these images, an image P1, an image P2, an image P6, and an image P7 in which a fox whose face is directed diagonally forward or laterally (left and right) are shown, respectively, in front of the imaging device 100. It shows a case where only the located pupil region is detected.

Two pupil areas are detected in each of the images P3 and P5 in which the fox whose face is facing slightly diagonally forward is shown, and one of the pupil areas is in front of the imaging device 100. It is easy to determine whether or not the other pupil region is located at the back.

　It is difficult to distinguish which of the left and right pupil regions is the front one in image P4, which shows the fox with its face facing forward. In this case, as described above, the notice frame or the pupil frame is displayed in the pupil region located in front of the imaging device 100 and close to the center of the focus frame.

<4. Operation example of imaging device>
FIG. 10 is a flowchart illustrating an image capturing process in the human pupil detection mode of the image capturing apparatus 100.

The image processing in the human pupil detection mode in FIG. 10 is started when the power is turned on by operating the power button, for example. The detection mode for each specific part of the subject is preset as a human pupil detection mode from a setting screen or the like.

In step S11 of FIG. 10, the system controller 131 determines whether to end the process, for example, whether the power button has been operated.

If it is determined in step S11 to end the process, the imaging process ends.

If it is determined in step S11 that the process is not completed, the process proceeds to step S12.

In step S12, the image sensor 103 acquires the electrical signal of each pixel of the image by photoelectrically converting the light from the subject condensed through the lens 101 and the diaphragm 102 on a pixel-by-pixel basis. The image signal, which is an electric signal of each pixel of the image, is output to the memory 211 of the digital signal processing unit 106 via the analog signal processing unit 104 and the A/D conversion unit 105.

In step S13, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a live view image.

In step S14, the person detection unit 212-1 detects a face area from the image data stored in the memory 211. The person detection unit 212-1 supplies the detected face area information to the area setting unit 213 and the display control unit 214.

The user gives an instruction to start focusing by pressing the pupil AF button, pressing the AF-ON button, or pressing the shutter button halfway. Note that the instruction to start focusing is issued for each image capturing unit. The operation unit 146 receives an operation input from the user and outputs a signal corresponding to the operation input to the system controller 131.

In step S15, the system controller 131 determines whether or not the user has pressed the eye AF button. If it is determined in step S15 that the pupil AF button has been pressed, the process proceeds to step S16.

In step S16, the system controller 131 forcibly changes the focus frame to a wide range.

In step S17, the person detection unit 212-1 detects the pupil area for the face area in the focus frame under the control of the system controller 131. Information on the detected pupil region is output to the region setting unit 213 and the display control unit 214.

In step S18, the area setting unit 213 determines whether a pupil area has been detected. When it is determined in step S18 that the pupil area has been detected, the process proceeds to step S19.

In step S19, the area setting unit 213 sets the pupil area detected by the person detection unit 212-1 as the focus area. Information on the set focus area is supplied to the system controller 131.

If it is determined in step S18 that the pupil region has not been detected, the process proceeds to step S20.

In step S20, the area setting unit 213 sets the face area detected by the person detecting unit 212-1 as the focus area. Information on the set focus area is output to the system controller 131.

On the other hand, if it is determined in step S15 that the eye AF button has not been pressed, the process proceeds to step S21.

In step S21, the system controller 131 determines whether the user has half-pressed the shutter button or whether the AF-ON button has been pressed.

If it is determined in step S21 that the shutter button has been half-pressed or the AF-ON button has been pressed, the process proceeds to step S22.

In step S22, the system controller 131 does not change the focus frame.

In step S23, the person detection unit 212-1 detects the pupil area for the face area in the focus frame under the control of the system controller 131. Information on the detected pupil region is output to the region setting unit 213 and the display control unit 214.

In step S24, the area setting unit 213 determines whether a pupil area has been detected. When it is determined in step S24 that the pupil area has been detected, the process proceeds to step S25.

In step S25, the area setting unit 213 sets the pupil area detected by the person detecting unit 212-1 as the focus area. Information on the set focus area is supplied to the system controller 131.

If it is determined in step S24 that the pupil area has not been detected, the process proceeds to step S26.

In step S26, the area setting unit 213 sets the face area or the focus frame detected by the person detecting unit 212-1 as the focus area. Information on the set focus area is output to the system controller 131.

If it is determined in step S20 that the shutter button has not been half pressed and the AF-ON button has not been pressed, the process proceeds to step S27.

In step S27, the area setting unit 213 sets the face area detected by the person detecting unit 212-1 as the focus area. Information on the set focus area is output to the system controller 131.

In step S28, the display control unit 214 generates a face notice frame in the face area detected by the person detection unit 212-1 and superimposes the face notice frame on the live view image to display it on the display unit 141.

After steps S19, S20, and S25, the process proceeds to step S29 in FIG.

In step S29, the system controller 131 controls the lens driver 121 to drive the optical system such as the lens 101 and the diaphragm 102 so that the focus area is in focus. Then, a process progresses to step S30.

After step S27 in FIG. 11, the process proceeds to step S30.

In step S30, the system controller 131 determines whether or not focus is achieved.

If it is determined in step S30 that the subject is in focus, the process proceeds to step S31.

In step S31, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a live view image. Further, the display control unit 214 superimposes a focus frame (pupil frame, face frame, or focus frame), which is a frame surrounding the set focus region, on the live view image and displays it on the display unit 141.

If it is determined in step S30 that the subject is out of focus, step S31 is skipped and the process proceeds to step S32.

After step S29 in FIG. 10, the process proceeds to step S32.

In step S32, the system controller 131 determines whether or not the shutter button has been fully pressed based on a signal corresponding to an operation input from the operation unit 146. If it is determined in step S32 that the shutter button has been fully pressed, the process proceeds to step S33.

In step S33, the image sensor 103 photoelectrically converts the light from the subject condensed through the optical system such as the lens 101 and the diaphragm 102 on a pixel-by-pixel basis to acquire an electric signal of each pixel of the image. The image signal, which is an electric signal of each pixel of the image, is output to the memory 211 of the digital signal processing unit 106 via the analog signal processing unit 104 and the A/D conversion unit 105.

In step S34, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a captured image.

In step S35, the codec processing unit 215 encodes the image data stored in the memory 211. The codec processing unit 215 supplies the encoded image data to the storage unit 142.

In step S36, the codec processing unit 215 causes the storage unit 142 to record the encoded image data. Then, the process returns to step S11, and the subsequent processes are repeated.

Also, when it is determined in step S32 that the shutter button has not been fully pressed, the process returns to step S11, and the subsequent processing is repeated.

FIG. 12 is a flowchart illustrating the image pickup processing of the animal eye detection mode of the image pickup apparatus 100.

The imaging processing in the animal pupil detection mode of FIG. 12 is started when the power is turned on by operating the power button, for example. As the detection mode for each specific part of the subject, the pupil detection mode of the animal is preset on the setting screen or the like. In FIG. 12, an example in which the animal detection unit 212-2 that detects the eyes of a cat or a dog detects the eyes of an animal will be described.

In step S111 of FIG. 12, the system controller 131 determines whether to end the process, for example, whether the power button has been operated.

If it is determined in step S111 to end the process, the imaging process ends.

If it is determined in step S111 that the process is not completed, the process proceeds to step S112.

In step S112, the image sensor 103 acquires an electrical signal of each pixel of the image by photoelectrically converting the light from the subject condensed through the lens 101 and the diaphragm 102 on a pixel-by-pixel basis. The image signal, which is an electric signal of each pixel of the image, is output to the memory 211 of the digital signal processing unit 106 via the analog signal processing unit 104 and the A/D conversion unit 105.

In step S113, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a live view image.

In step S114, the animal detection unit 212-2 detects the pupil area from the image data stored in the memory 211. The animal detection unit 212-2 supplies the detected pupil area information to the area setting unit 213 and the display control unit 214.

In step S115, the display control unit 214 performs a pupil region selection process detected by the animal detection unit 212-2. By the pupil area selection processing, the pupil area in which the notice frame is displayed is selected from the plurality of detected pupil areas.

In step S116, the system controller 131 determines whether or not the user has pressed the eye AF button.

If it is determined in step S116 that the eye AF button has been pressed, the process proceeds to step S117.

In step S117, the system controller 131 forcibly changes the focus frame to a wide range.

In step S118, the animal detection unit 212-2 detects a pupil region in the focus frame that has been changed in a wide range under the control of the system controller 131. Information on the detected pupil region is output to the region setting unit 213 and the display control unit 214.

In step S119, the area setting unit 213 determines whether a pupil area has been detected. If it is determined in step S119 that the pupil region has been detected, the process proceeds to step S120.

In step S120, the area setting unit 213 sets the pupil area detected by the animal detection unit 212-2 as the focus area. Information on the set focus area is supplied to the system controller 131.

On the other hand, if it is determined in step S116 that the eye AF button has not been pressed, the process proceeds to step S121.

In step S121, the system controller 131 determines whether the user has half-pressed the shutter button or whether the AF-ON button has been pressed.

If it is determined in step S121 that the shutter button has been half-pressed or the AF-ON button has been pressed, the process proceeds to step S122.

In step S122, the system controller 131 does not change the focus frame.

In step S123, the animal detection unit 212-2 detects the pupil area in the focus frame under the control of the system controller 131. Information on the detected pupil region is output to the region setting unit 213 and the display control unit 214.

In step S124, the area setting unit 213 determines whether a pupil area has been detected. When it is determined in step S124 that the pupil region has been detected, the process proceeds to step S125.

In step S125, the area setting unit 213 sets the pupil area detected by the animal detection unit 212-2 as the focus area. Information on the set focus area is supplied to the system controller 131.

If it is determined in step S125 that the pupil area has not been detected, the process proceeds to step S126.

In step S126, the area setting unit 213 sets the focus frame as the focus area as another condition. Information on the set focus area is output to the system controller 131.

If it is determined in step S121 that the shutter button has not been half pressed and the AF-ON button has not been pressed, the process proceeds to step S127.

In step S127, the area setting unit 213 determines whether a pupil area has been detected. When it is determined in step S127 that the pupil region has been detected, the process proceeds to step S128.

In step S128, the area setting unit 213 sets the pupil area detected by the animal detection unit 212-2 as the focus area. Information on the set focus area is supplied to the system controller 131.

In step S129, the display control unit 214 generates a pupil preview frame in the pupil region detected by the animal detection unit 212-2, superimposes the pupil preview frame on the live view image, and causes the display unit 141 to display it.

After steps S120 and S125, the process proceeds to step S130 in FIG.

In step S130, the system controller 131 controls the lens driver 121 to drive the optical system such as the lens 101 and the diaphragm 102 so that the focus area is in focus. Then, a process progresses to step S131.

After step S126 of FIG. 12, the process proceeds to step S131.

In step S131, the system controller 131 determines whether or not focus is achieved.

If it is determined in step S131 that the subject is in focus, the process proceeds to step S132.

In step S132, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a live view image. In addition, the display control unit 214 causes the display unit 141 to display a focus frame (pupil frame or focus frame), which is a frame surrounding the set focus region, on the live view image.

If it is determined in step S131 that the subject is out of focus, step S132 is skipped and the process proceeds to step S133.

If it is determined in step S119 or S127 in FIG. 12 that the pupil region is not detected, focusing is not performed, and the process proceeds to step S133 in FIG. After step S129, the process proceeds to step S133.

In step S133, the system controller 131 determines whether or not the shutter button has been fully pressed based on a signal corresponding to an operation input from the operation unit 146. If it is determined in step S133 that the shutter button has been fully pressed, the process proceeds to step S134.

In step S134, the image sensor 103 photoelectrically converts the light from the subject condensed through the optical system such as the lens 101 and the diaphragm 102 in pixel units to acquire the electric signal of each pixel of the image. The image signal, which is an electric signal of each pixel of the image, is output to the memory 211 of the digital signal processing unit 106 via the analog signal processing unit 104 and the A/D conversion unit 105.

In step S135, the display control unit 214 causes the display unit 141 to display an image based on the image data stored in the memory 211 as a captured image.

In step S136, the codec processing unit 215 encodes the image data stored in the memory 211. The codec processing unit 215 supplies the encoded image data to the storage unit 142.

In step S137, the codec processing unit 215 causes the storage unit 142 to record the encoded image data. After that, the process returns to step S111, and the subsequent processes are repeated.

Also, if it is determined in step S133 that the shutter button has not been fully pressed, the process returns to step S111, and the subsequent processing is repeated.

In addition, in the above-mentioned processing, when the pupil area is not detected, the focusing processing is not executed, or the focusing processing is executed under other conditions such as the focus frame. The operation when the pupil region is not detected may be selectively used depending on the setting by the user. At that time, the operation when not detected may be changed according to the target subject or the specific part which is not detected.

For example, by setting the focus process so that it will not be performed if it is not detected, it is possible to prevent further deterioration of the focus accuracy when the target subject or a specific part moves little. On the contrary, by setting the focus process at the position designated by the user when it is not detected, the focus process is performed in advance when the movement of the target subject or the specific part is large. be able to.

FIG. 14 is a flowchart illustrating the pupil area selection processing in step S114 of FIG.

In step S151, the display control unit 214 determines whether or not there are two or more pupil regions in the focus frame. For example, when two animals are in a position in front of the imaging device 100, any number of pupils from 1 to 4 can be detected.

If it is determined in step S151 that there are two or more pupil regions in the focus frame, the process proceeds to step S52.

In step S152, the display control unit 214 determines whether or not there are two or more front pupil regions.

If it is determined in step S152 that there are two or more front pupil regions, the process proceeds to step S153.

In step S153, the display control unit 214 calculates the distance between each pupil region and the center of the focus frame.

In step S154, the display control unit 214 selects the pupil area having the shortest distance from the center of the focus frame.

On the other hand, if it is determined in step S151 that there are two or more detected pupil regions in the focus frame, the process proceeds to step S155.

In step S155, the display control unit 214 selects one pupil area.

If it is determined in step S152 that there are no more than two front pupil regions, the process proceeds to step S156.

In step S156, the display control unit 214 selects the front pupil region.

After steps S154 to S156, the pupil area selection process ends, and the process returns to step S114 in FIG.

As described above, in the present technology, a notice frame for giving notice of a specific area to be focused is displayed on the image acquired from the imaging unit according to the type of subject.

For example, if the user performs an operation for instructing to start focusing and automatic focusing and imaging are performed when the notice frame is not displayed, the result may be unintended by the user depending on the condition of the subject. ..

According to the present technology, by displaying the notice frame, it is possible to know in advance that an intended position or an unintended position has been detected. Therefore, the user can select not to perform automatic focusing. It is possible.

With the above, the user can easily focus on a specific part such as a pupil region or the animal itself, depending on the animal that is the type of subject.

In the above description, the processing of detecting the pupils of animals such as dogs and cats has been described, but the present technology is the eyes, faces, parts of the face, neck, and neck of all living things such as birds, fish, reptiles, and amphibians. It can be applied to a specific part of the subject such as the head or the whole body (subject). The present technology can also be applied to a specific part of these subjects or a combination of subjects.

Furthermore, the present technology can be applied not only to living things, but also to specific parts of a subject such as a vehicle headlight, a front emblem, a windshield, or a driver's seat, or a motorcycle headlight or a helmet.

In these cases, a detection mode for detecting a specific part of the subject is preset and used. By doing so, the user's intention such as which detection result or detection method among a plurality of detection results or detection methods is prioritized or which of the plurality of objects is prioritized is captured. Can tell the device.

Note that, unlike the case of a person, when trying to focus on the eyes of an animal with long hair, the hair on the eyes may be focused instead of the eyes. In this case, the focus position may be adjusted backward, or the subject that is likely to be in focus on the hair may be preset in the imaging device, and the imaging device may control based on the setting. It is possible to obtain a suitable imaging result.

However, if a subject with short hair is similarly controlled, the result will be out of focus. Therefore, it is possible to further improve the convenience by adjusting in advance what kind of adjustment is reflected by using the detection target object or the specific part, and other detection results.

In the above description, the human eyes and the animal eyes are detected by different detection processes, but the human eyes may be detected by the same detection process as the animal eyes.

Also, the series of processes described above can be executed by hardware or software. When the series of processes described above is executed by software, a program forming the software is installed from a network or a recording medium.

As shown in FIG. 5, for example, this recording medium is composed of a removable recording medium 148 in which the program is recorded, which is distributed in order to distribute the program to the user, separately from the apparatus main body. The removable recording medium 148 includes a magnetic disk (including a flexible disk) and an optical disk (including a CD-ROM and DVD). It also includes magneto-optical disks (including MD (Mini Disc)) and semiconductor memory.

In this case, the program can be installed in the storage unit 142 by mounting the removable recording medium 148 in the drive 147.

Also, this program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting. In this case, the program can be received via the communication unit 145 and installed in the storage unit 142.

In addition, this program can be installed in advance in a storage unit 142 or a ROM (Read Only Memory) in the system controller 131.

Note that the program executed by the computer may be a program in which processing is performed in time series in the order described in this specification, or in parallel, or at a necessary timing such as when a call is made. It may be a program in which processing is performed.

In this specification, the system means a set of a plurality of constituent elements (devices, modules (parts), etc.), and it does not matter whether or not all constituent elements are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device housing a plurality of modules in one housing are all systems. ..

The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

For example, the present technology can have a configuration of cloud computing in which a single function is shared by a plurality of devices via a network and jointly processes.

Also, each step described in the above flow chart can be executed by one device or shared by a plurality of devices.

Further, when one step includes a plurality of processes, the plurality of processes included in one step can be executed by one device or shared by a plurality of devices.

Note that the effects described in this specification are merely examples and are not limited, and other effects may be present.

<Combination example of configuration>
The present technology may also be configured as below.
(1) An image pickup apparatus including a display control unit that displays a notice frame that gives a notice of a specific area to be focused on an image acquired from the image pickup unit according to the type of subject.
(2) The imaging device according to (1), wherein the subject is a person or an animal.
(3) The imaging device according to (1), in which the subject can be set in advance.
(4) The imaging device according to any one of (1) to (3), wherein the specific area is an area of a specific part of a subject.
(5) The imaging device according to (4), wherein the specific part is a pupil.
(6) The imaging device according to (1), wherein the specific region is a subject region.
(7) An area detection unit for detecting the specific area is further provided,
The display control unit controls the display of the notice frame in accordance with a focus setting frame for setting a focus region when a plurality of the specific regions are detected. (1) to (6) The imaging device described.
(8) The image pickup apparatus according to (7), wherein the display control unit controls the display of the notice frame according to the specific area closer to the center position of the focus setting frame.
(9) A focusing instruction unit for instructing the start of focusing for each image pickup unit of the image,
The image pickup apparatus according to (7) or (8), further including: an area setting unit that sets the detected specific area as a focus area of the image when the start of the focus is instructed.
(10) The imaging device according to (9), wherein the area setting unit sets the specific area detected within a predetermined range as the focus area.
(11) The imaging device according to (9), wherein the area setting unit sets the specific area detected within a predetermined range indicated by the focus setting frame as the focus area.
(12) When the focusing area is set by the area setting unit, the display control unit controls display of a focusing area frame indicating the focusing area, instead of displaying the notice frame. The imaging device according to any one of 9) to (11).
(13) The imaging device according to (9), wherein the display control unit controls the display of the focusing area frame by a display method different from the display of the notice frame.
(14) The imaging device according to (9), wherein the area setting unit sets the specific area detected within a predetermined range indicated by the focus setting frame as the focusing area.
(15) An image capturing instruction unit that instructs image capturing,
A focus control unit that controls the image capturing unit so as to perform the focus in the focus region set by the region setting unit to acquire the image when the image capturing is instructed. The imaging device according to any one of (6) to (10).
(16) The imaging device is
An imaging method that displays a notice frame that gives an advance notice of a specific area to be focused on the image acquired by the imaging unit according to the type of subject.
(16)
A program that causes a computer to function as a display control unit that displays a notice frame that gives a notice of a specific area to be focused on an image acquired from the imaging unit according to the type of subject.

100 image pickup device, 101 lens, 102 diaphragm, 103 image pickup element, 104 analog signal processing unit, 105 A/D conversion unit, 106 digital signal processing unit, 121 lens driver, 131 system controller, 141 display unit, 142 storage unit, 146 Operation unit, 211 memory, 212 subject detection unit, 212-1 person detection unit, 212-2 animal detection unit, 212-3 animal detection unit, 213 area setting unit, 214 display control unit, 215 codec processing unit

Claims

An imaging device that includes a display control unit that displays a notice frame that gives a notice of a specific area to be focused on an image acquired from the imaging unit according to the type of subject.
The imaging device according to claim 1, wherein the subject is a person or an animal.
The imaging device according to claim 1, wherein the subject can be set in advance.
The imaging device according to claim 1, wherein the specific region is a region of a specific portion of a subject.
The imaging device according to claim 4, wherein the specific portion is a pupil.
The imaging device according to claim 1, wherein the specific region is a region of a subject.
Further comprising an area detection unit for detecting the specific area,
The imaging device according to claim 1, wherein the display control unit controls the display of the notice frame according to a focus setting frame for setting a focus region when a plurality of the specific regions are detected.
The image pickup apparatus according to claim 7, wherein the display control unit controls display of the notice frame according to the specific region closer to the center position of the focus setting frame.
A focus instruction unit for instructing the start of focusing for each image pickup unit of the image;
The image pickup apparatus according to claim 7, further comprising: an area setting unit that sets the detected specific area as a focus area of the image when the start of the focus is instructed.
The image pickup apparatus according to claim 9, wherein the area setting unit sets the specific area detected within a predetermined range as the focus area.
The image pickup apparatus according to claim 9, wherein the area setting unit sets the specific area detected within a predetermined range indicated by the focus setting frame as the focus area.
The display control unit controls the display of a focus area frame indicating the focus area, instead of the display of the notice frame, when the focus area is set by the area setting unit. Imaging device.
The image pickup apparatus according to claim 12, wherein the display control unit controls the display of the focus area frame by a display method different from the display of the notice frame.
An imaging instruction unit for instructing imaging,
A focus control unit that controls the image capturing unit so as to perform the focus in the focus region set by the region setting unit and acquire the image when the image capturing is instructed. Item 9. The imaging device according to item 9.
The imaging device
An imaging method that displays a notice frame that gives an advance notice of a specific area to be focused on the image acquired by the imaging unit according to the type of subject.
A program that causes a computer to function as a display control unit that displays a notice frame that gives a notice of a specific area to be focused on an image acquired from the imaging unit according to the type of subject.