WO2019082775A1

WO2019082775A1 - Image processing device, image processing method, and imaging device

Info

Publication number: WO2019082775A1
Application number: PCT/JP2018/038742
Authority: WO
Inventors: 亮輔荒木
Original assignee: ソニー株式会社
Priority date: 2017-10-24
Filing date: 2018-10-17
Publication date: 2019-05-02

Abstract

The present invention improves the accuracy of AF control, AE control, etc., on a moving subject. An image signal inputted from the imaging area of an imaging unit is detected in a detection area of which a plurality are provided in the imaging area, and a detection value corresponding to each of all detection areas is generated. A subject is detected on the basis of the image signal. A detection area that corresponds to the subject is set from all detection areas on the basis of the result of subject detection, and a detection value that corresponds to the detection area set from the detection value that corresponds to each of all detection areas is specified.

Description

Image processing apparatus, image processing method and imaging apparatus

The present technology relates to an image processing apparatus, an image processing method, and an imaging apparatus, and more particularly to an image processing apparatus and the like that can improve the accuracy of AF control and AE control for a moving subject.

For example, Patent Document 1 describes an automatic focusing (AF) system established by setting a detection area at an AF area or subject recognition position set by the user and detecting the contrast in the detection area. . In this case, if the subject moves when AF is started, it leaves the subject from the AF area set by the subject recognition result immediately before the start of AF. As a result, the contrast of the subject can not be accurately detected, and the AF accuracy is deteriorated due to focusing on the background or false focusing. Furthermore, for a subject that is moving, subject recognition is performed first, and if the AF area is set using the result, the position of the subject and the AF area will always shift, and focusing during continuous shooting The rate drops.

Also, in the conventional Automatic Exposure (AE) system, there are similar disadvantages. For example, when the AE area is set according to the subject recognition result and AE is performed using the detected value, if the subject moves when AE is started, the subject comes out from the AE area set by the subject recognition result immediately before AE start. It will As a result, the brightness of the subject can not be accurately detected, and accurate automatic exposure can not be performed. Furthermore, in the case of a subject that is moving, this state continuously occurs continuously, so the state in which automatic exposure can not be accurately performed continues.

Unexamined-Japanese-Patent No. 2010-160297

The purpose of the present technology is to improve the accuracy of AF control and AE control for a moving subject.

The concept of this technology is
A detection processing unit that detects image signals input from an imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area, and generates detection values corresponding to all detection areas;
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An image processing apparatus is provided with a control unit that specifies a detection value corresponding to an area.

In the present technology, the detection processing unit detects an image signal input from the imaging area of the imaging unit in a detection area in which a plurality of imaging signals are arranged in the imaging area, and generates detection values corresponding to all detection areas. The subject detection unit also generates detection values corresponding to each of the entire detection areas. For example, the detection area may be spread over the entire area of the imaging area. Further, for example, the process of generating the detection value in the detection processing unit and the process of detecting the object in the object detection unit may be performed in parallel.

The control unit sets the detection area corresponding to the subject from all detection areas based on the detection result of the object detection unit, and the detection area set from the detection value corresponding to each of all the detection areas generated by the detection processing unit The detected value corresponding to is identified. For example, in the setting of the detection area corresponding to the subject, the area where the subject is detected is set as the detection area, or the area excluding the area where the subject is detected is set as the detection area. It is also good.

In addition, for example, a detection value storage unit that stores the detection value generated by the detection processing unit may be further provided. Then, in this case, the control unit causes the detection value of the detection area set corresponding to the subject detected based on the image signal of the predetermined frame by the subject detection unit to be stored in the detection value storage unit. The image signal may be detected and identified from detected values corresponding to all detection areas generated.

Also, for example, the detection area may be a ranging area. Then, in this case, the control unit may perform focus control based on the identified detection value. In this case, for example, a plurality of phase difference detection areas may be further disposed in the imaging area, and the number of ranging areas may be greater than the number of phase difference detection areas. Also, for example, the detection area may be a photometric area. Then, in this case, the control unit may perform exposure control based on the identified detection value.

As described above, in the present technology, the detection area corresponding to the subject is set from all detection areas of a plurality of detection areas arranged in the imaging area based on the subject detection result, and each detection area generated by the detection processing unit The detection value corresponding to the set detection area is specified from the detection value corresponding to. Therefore, the detection value for the moving subject can be specified with high accuracy, and the accuracy of AF control or AE control for the moving subject can be improved by using the specified detection value.

Also, the other concept of this technology is
An imaging unit,
A detection processing unit that detects image signals input from an imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area, and generates detection values corresponding to all detection areas;
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An imaging apparatus is provided with a control unit that specifies a detected value corresponding to an area.

According to the present technology, it is possible to improve the accuracy of AF control and AE control for a moving subject. The effects described in the present specification are merely examples and are not limited, and additional effects may be present.

FIG. 2 is a block diagram illustrating an exemplary configuration of a digital still camera. 5 is a flowchart showing an example of the procedure of contrast AF processing in the digital still camera of FIG. 1; FIG. 1 is a block diagram showing a configuration example of a digital still camera as a first embodiment. It is a figure for demonstrating the example of arrangement | positioning of several AF detection area in an imaging area. FIG. 6 is a diagram for describing an example of a captured image, the arrangement of an AF detection area corresponding thereto, and an AF detection area (AF detection frame) specified by object position information. 5 is a flowchart showing an example of the procedure of contrast AF processing in the digital still camera of FIG. 3; It is a figure which shows an example of a captured image (monitoring camera image) etc. when working with a robot arm. It is a block diagram showing an example of composition of a surveillance camera at the time of working with a robot arm as a 2nd embodiment. FIG. 2 is a block diagram illustrating an exemplary configuration of a digital still camera. 10 is a flowchart showing an example of the procedure of contrast AE processing in the digital still camera of FIG. 9; FIG. 14 is a block diagram showing an example of configuration of a digital still camera as a third embodiment. 12 is a flowchart showing an example of the procedure of contrast AE processing in the digital still camera of FIG. 11; It is a block diagram showing an example of rough composition of a vehicle control system. It is explanatory drawing which shows an example of the installation position of a vehicle exterior information detection part and an imaging part. It is a figure which shows an example of a schematic structure of an endoscopic surgery system. It is a block diagram which shows an example of a function structure of a camera head and CCU. It is a figure which shows the example at the time of combining the frame of the detection frame (ranging frame) and phase difference detection type | formula which used this technique.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be made in the following order.
1. First Embodiment Second embodiment 3. Third embodiment 4. Application example to mobile object 5. Application example to endoscopic surgery system 6. Modified example

<1. First embodiment>
[Configuration Example of Digital Still Camera]
FIG. 1 shows a configuration example of a digital still camera 100. The digital still camera 100 includes a control unit 101, an operation unit 102, an imaging lens 103, an imaging device 104, a signal processing unit 105, a display control unit 106, a display unit 107, and an object recognition unit 108. An object tracking unit 109, an AF detection area setting unit 110, an AF detection processing unit 111, an AF control unit 112, an image recording processing unit 113, and a recording medium 114 are provided.

The control unit 101 controls the operation of each unit of the digital still camera 100 based on a control program. The operation unit 102 is connected to the control unit 101, configures a user interface that receives various operations by the user, and includes keys, dials, buttons, a touch panel, a remote controller, and the like.

The imaging lens 103 includes a plurality of lenses and the like, and condenses light incident from a subject and guides the light to the imaging surface (imaging area) of the imaging element 104. The imaging lens 103 has a focus lens, and by controlling the focus lens, focus control is enabled. Further, the imaging lens 103 has an aperture and a shutter, and exposure control can be performed by controlling the aperture position and the shutter speed.

The image sensor 104 is a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD), or the like, which constitutes an imaging unit and has an imaging surface (imaging area) in which a plurality of pixels are arranged in a matrix. Light incident from a subject through the lens 103 is received by the imaging surface.

The signal processing unit 105 applies analog signal processing such as amplification to the analog image signal from the imaging element 104, and further A / D (Analog / Digital) converts the image signal obtained as a result. Further, the signal processing unit 105 applies digital signal processing such as noise removal processing to image data represented by a digital signal obtained by A / D conversion, and the image data obtained as a result is displayed on the display control unit 106, It is supplied to the subject recognition unit 108 and the image recording processing unit 113.

The display control unit 106 causes the display unit 107 to display a captured image corresponding to the image data from the signal processing unit 105. The display unit 107 includes an LCD (Liquid Crystal Display), an organic EL display (OELD: Organic Electroluminescence Display), or the like.

The subject recognition unit 108 recognizes a subject to be tracked from the image data, and supplies information such as the feature amount to the subject tracking unit 109 as tracking target information. The designation of the tracking target object is performed by, for example, the touch panel operation of the user. The subject tracking unit 109 detects the subject position from the image data using the tracking target information from the subject recognition unit 108, and sends the subject position information (including the shape information) to the display control unit 106 and the AF detection area setting unit 110. Supply. The display control unit 106 superimposes and displays a tracking frame on the captured image based on the subject position information.

The AF detection area setting unit 110 sets an AF detection area (AF detection frame) as a ranging area at the subject position in the imaging area based on the subject position information supplied from the subject tracking unit 109, and the information is It is supplied to the AF detection processing unit 111. The AF detection processing unit 111 acquires an AF detection value (contrast detection value) of the AF detection area from the image data based on setting information of the AF detection area, and supplies the AF detection value to the AF control unit 112.

The AF control unit 112 determines the focus lens position at the time of showing the highest contrast based on the history of AF detection values obtained along with the movement operation of the focus lens, and notifies the imaging lens 103 of the result. The focus lens is driven to be the focus lens position when high contrast is shown.

Under control of the control unit 101, for example, when the shutter button is pressed, the image recording processing unit 113 compresses the image data from the signal processing unit 105 according to a predetermined compression method such as JPEG (Joint Photographic Experts Group) method. Image data compressed according to the method and compressed is recorded on the recording medium 114. The recording medium 114 is, for example, a recording medium such as a memory card, and can be easily attached to and detached from the digital still camera 100.

The flowchart of FIG. 2 shows an example of the procedure of contrast AF processing in the digital still camera 100 of FIG. In step ST1, the AF process is started by the user pressing the shutter button halfway. Next, in step ST2, the focus lens position of the imaging lens 103 is driven to a position desired to be detected.

Next, in step ST3, an image signal is read from the imaging element 104, and in step ST4, the signal processing unit 105 generates image data. Next, in step ST5, the subject recognition unit 108 recognizes, for example, the subject specified by the user's touch panel operation, and in step ST6, the subject tracking unit 109 performs tracking processing of the subject present in the captured image. The subject position information (including the shape information) indicating the position on the captured image is obtained.

Next, in step ST7, the AF detection area setting unit 110 sets an AF detection area (AF detection frame) at the object position in the imaging area. Next, in step ST8, the AF detection processing unit 111 acquires an AF detection value (contrast detection value) of the AF detection area from the image data.

Next, in step ST9, the AF control unit 112 determines the in-focus state from the history of AF detection values. Then, in step ST10, when the in-focus determination can not be made, the process returns to step ST2, the focus lens position of the imaging lens 103 is moved, and the same process as described above is repeated.

On the other hand, when the in-focus determination is made, driving of the focus lens to the focus lens position of the AF detection value for which the in-focus determination is made by the AF control unit 112 is instructed to the imaging lens 103 in step ST11. Next, in step ST12, the focusing lens is driven by the imaging lens 103 to a designated position. Then, in step ST13, a series of AF processing is ended.

In the above-described contrast AF processing (see FIG. 2) in the digital still camera 100 of FIG. 1, after subject recognition is performed, an AF detection area (AF detection frame) is set, and thereafter AF detection is started to The procedure is to obtain an AF detection value. Therefore, for a moving subject, the subject position when subject detection is performed deviates from the subject position when AF detection is started, and it is difficult to obtain an AF detection value corresponding to the subject position. , High precision AF control is impossible.

FIG. 3 shows an example of the configuration of a digital still camera 200 according to the first embodiment. In FIG. 3, parts corresponding to FIG. 1 are given the same reference numerals, and the detailed description thereof will be omitted as appropriate. The digital still camera 200 includes a control unit 101, an operation unit 102, an imaging lens 103, an imaging element 104, a signal processing unit 105, a display control unit 106, a display unit 107, and an object recognition unit 108. An object tracking unit 109, an AF control unit 112, an image recording processing unit 113, a recording medium 114, an AF detection processing unit 201, an AF detection storage unit 202, and an AF detection value generation unit 203 are provided.

The control unit 101 controls the operation of each unit of the digital still camera 200 based on a control program. The operation unit 102 is connected to the control unit 101, configures a user interface that receives various operations by the user, and includes keys, dials, buttons, a touch panel, a remote controller, and the like.

The imaging element 104 is a complementary metal oxide semiconductor (CMOS) image sensor, a charge coupled device (CCD), or the like having an imaging surface in which a plurality of pixels are arranged in a matrix, constituting an imaging unit. Light from the subject is received by the imaging surface.

The signal processing unit 105 applies analog signal processing such as amplification to the analog image signal from the imaging element 104, and further A / D (Analog / Digital) converts the image signal obtained as a result. Further, the signal processing unit 105 applies digital signal processing such as noise removal processing to image data represented by a digital signal obtained by A / D conversion, and the image data obtained as a result is displayed on the display control unit 106, The information is supplied to the subject recognition unit 108, the AF detection processing unit 201, and the image recording processing unit 113.

The subject recognition unit 108 recognizes a subject to be tracked from the image data, and supplies information such as the feature amount to the subject tracking unit 109 as tracking target information. The designation of the tracking target object is performed by, for example, the touch panel operation of the user. The subject tracking unit 109 detects the subject position from the image data using the tracking target information from the subject recognition unit 108, and obtains subject position information (including shape information).

The subject tracking unit 109 supplies subject position information to the display control unit 106. The display control unit 106 superimposes and displays a tracking frame on the captured image based on the subject position information. The subject tracking unit 109 also supplies subject position information to the AF detection value generation unit 203 together with an image data ID identifying a frame of image data used when the subject position is determined.

The AF detection processing unit 201 acquires, from the image data from the signal processing unit 105, AF detection values (contrast detection values) of the AF detection area as a plurality of ranging areas arranged in the imaging area. In this case, the AF detection area is spread over the entire area of the imaging area. The AF detection area may be spread over all other areas except the periphery of the imaging area.

FIGS. 4A and 4B show a state in which rectangular AF detection areas are spread over the entire area of the imaging area, and FIG. 4A is an example in which the size of the AF detection area is large. FIG. 4B shows an example in which the size of the AF detection area is small. In FIG. 4A, 9 × 3 AF detection areas of 3 × 3 are disposed in the imaging area, and in FIG. 4B, 400 AF detection areas of 20 × 20 are disposed in the imaging area. The number of AF detection areas is not limited to this example.

4C and 4D show a state in which rectangular AF detection areas are spread in the area excluding the periphery of the imaging area, and FIG. 4C shows the size of the AF detection area. FIG. 4D shows an example in which the size of the AF detection area is small. In FIG. 4C, 3 × 3 nine AF detection areas are arranged, and in FIG. 4D, 20 × 20 400 AF detection areas are arranged. However, the number of AF detection areas is Is not limited to this example.

The AF detection processing unit 201 is an image for identifying a frame of image data used when obtaining the detection value, that is, an AF detection value acquired in all AF detection areas arranged in a plurality of imaging areas, that is, an AF detection value group. The data is supplied to the AF detection storage unit 202 together with the data ID and temporarily stored. The AF detection value generation unit 203 generates subject position information from among detection value groups associated with the same image data ID based on subject position information (including shape information) supplied from the subject tracking unit 109 and the image data ID. The AF detection values of a predetermined number of AF detection areas corresponding to the subject position identified in step are taken out from the AF detection storage unit 202 and integrated (averaged) to generate an AF detection value (integral AF detection value) at the subject position Do.

Here, an example of the captured image, the arrangement of the AF detection area corresponding thereto, and the AF detection area (AF detection frame) specified by the subject position information will be described. FIG. 5A shows an example of a captured image, and the captured image includes the subject OB. FIG. 5B shows an example of arrangement of the captured image and the AF detection area DE spread in the imaging area. In FIG. 5C, when the AF detection area DE is disposed as shown in FIG. 5B, the AF detection area DE within the range indicated by the arrow P corresponds to the subject based on the subject position information. It indicates that it is identified.

FIG. 5D also shows an arrangement example of the captured image and the AF detection area DE spread in the imaging area. In the case of this arrangement example, the size of the AF detection area DE is smaller than that of FIG. 5B, and the number of AF detection areas DE is increased accordingly. In FIG. 5E, when the AF detection area DE is disposed as shown in FIG. 5D, the AF detection area DE within the range indicated by the arrow P corresponds to the subject based on the subject position information. It indicates that it is identified.

Although described above as an example of the method of setting the predetermined number of AF detection areas corresponding to the subject position specified by the subject position information, the present invention is not particularly limited thereto. For example, all of the AF detection areas in which the subject is detected based on the subject position information may be set as the predetermined number of AF detection areas to be identified. Further, among the AF detection areas in which the subject is detected, an area in which the ratio of the subject information occupies is an arbitrary value or more, for example, 50% or more may be set as a predetermined number of AF detection areas to be specified. Further, among the AF detection areas in which the subject is detected, an area including other than the subject information (such as background) may be set as the specified number of AF detection areas to be specified.

Returning to FIG. 3, the AF detection value generation unit 203 supplies the generated AF detection value to the AF control unit 112. The AF control unit 112 determines the focus lens position at the time of showing the highest contrast based on the history of AF detection values obtained along with the movement operation of the focus lens, and notifies the imaging lens 103 of the result. The focus lens is driven to be the focus lens position when high contrast is shown.

Under control of the control unit 101, the image recording processing unit 113 compresses the image data from the signal processing unit 105 according to a predetermined compression method such as JPEG (Joint Photographic Experts Group) method, and the compressed image data Recording is performed on the recording medium 114. The recording medium 114 is, for example, a recording medium such as a memory card, and can be easily attached to and detached from the digital still camera 200.

The flowchart of FIG. 6 shows an example of the procedure of contrast AF processing in the digital still camera 200 of FIG. In step ST21, the AF process is started by the user pressing the shutter button halfway. Next, in step ST22, the focus lens position of the imaging lens 103 is driven to a position desired to be detected.

Next, in step ST23, an image signal is read from the imaging element 104, and in step ST24, the signal processing unit 105 generates image data. Next, the processes of step ST25 and step ST26 are performed, and the processes of step ST27 and step ST28 are performed in parallel.

In step ST25, the subject recognition unit 108 recognizes, for example, a subject specified by the user's touch panel operation, and in step ST26, the subject tracking unit 109 determines the subject position and obtains subject position information (including shape information). Ru. On the other hand, in step ST27, AF detection values (contrast detection values) of the AF detection areas arranged in a plurality of imaging areas by the AF detection processing unit 201 are respectively acquired, and in step ST28, detection values of all detection areas, ie, detection values The group is stored in the AF detection storage unit 202 together with an image data ID identifying a frame of image data used to obtain it.

Next, in step ST29, the AF detection value generation unit 203 generates the same image data from the AF detection storage unit 202 based on the subject position information (including the shape information) supplied from the subject tracking unit 109 and the image data ID. AF detection values of a predetermined number of AF detection areas corresponding to the subject position specified by the subject position information among the detection value groups associated by the ID are acquired and integrated, and the AF detection values at the subject position (Integral AF Detection value) is generated.

Next, in step ST30, the AF control unit 112 determines the in-focus state from the history of the AF detection value. Then, in step ST31, when it can not be determined to be in focus, the processing returns to step S22, the focus lens position of the imaging lens 103 is moved, and the same processing as described above is repeated.

On the other hand, when it is determined that the image is in focus, in step ST32, the imaging lens 103 is instructed to drive the focus lens to the focus lens position of the AF detection value determined to be in focus by the AF control unit 112. Next, in step ST33, the focusing lens is driven by the imaging lens 103 to the designated position. Then, in step ST34, a series of AF processing is ended.

Here, the necessity of the AF detection storage unit 202 in the digital still camera 200 of FIG. 3 will be described. If the processing times of the AF detection process (step ST27) and the subject detection process (step ST26) in the flowchart of FIG. 6 coincide with each other, the detection value storage process (step ST28) becomes unnecessary. The AF detection storage unit 202 in FIG.

However, in many cases, the processing time for the subject detection process (step ST26) is longer than that for the AF detection process (step ST27). Therefore, when performing processing for generating an AF detection value in the AF detection value generation unit 203, in order to use an AF detection value whose timing matches the subject detection timing, the AF detection value of each acquired AF detection area The AF detection storage unit 202 is required to temporarily store

In the digital still camera 200 shown in FIG. 3, the subject position detection process and the AF detection process are performed in parallel, and the AF detection value is stored in the AF detection storage unit 202 to detect the subject position. It becomes possible to refer to the AF detection value generated by the image data of the same frame as the used frame. Therefore, the AF detection area does not always shift with respect to a moving subject, the AF detection value for the subject can be correctly obtained, and high-accuracy AF control can be performed.

Further, in the digital still camera 200 shown in FIG. 3, since the AF detection areas are spread over all or almost all areas of the imaging area (see FIG. 4), a predetermined number of AF detection areas corresponding to the object position The AF detection value for the subject can be accurately obtained by appropriately specifying it, and the AF control can be performed with high accuracy.

<2. Second embodiment>
[Configuration example of surveillance camera]
In the digital still camera 200 shown in FIG. 3 described above, a predetermined number of AF detection areas corresponding to a subject position are appropriately specified based on subject position information obtained by subject detection, and a predetermined number of AF detection areas corresponding to the subject position are detected. The AF control is performed by generating an AF detection value (integral AF detection value) using the AF detection value. However, based on subject position information obtained by subject detection, a predetermined number of AF detection areas excluding the subject position are specified, and an AF detection value (integral AF detection using a predetermined number of AF detection values excluding the subject position) It is also conceivable to perform AF control by generating a value).

For example, consider a surveillance camera image when working with a robot arm. FIG. 7A shows a captured image (monitoring camera image) when working with the robot arm. In the captured image, robot arms S1 and S3 and a work target S2 to be worked by the robot arms S1 and S3 exist. Generally, the contrast AF is easy to focus on the robot arms S1 and S3 in order to focus on the closest one.

Therefore, as shown in FIG. 7B, the AF detection area (AF detection frame) DE is arranged at high density in the imaging area. Then, the robot arms S1 and S3 are recognized by image recognition, and as shown in FIG. 7C, the detected values of the predetermined number of AF detection areas DE excluding the recognized robot arms S1 and S3 are integrated. Thus, an AF detection value (integral AF detection value) is generated, and AF control is performed using this AF detection value. As a result, it becomes possible to focus on the work target S2 without being influenced by the robot arms S1 and S3 present on the near side.

FIG. 8 shows a configuration example of a monitoring camera 300 when working with a robot arm according to the second embodiment. In this surveillance camera 300, parts corresponding to the digital still camera 200 in FIG. 3 are assigned the same reference numerals, and the detailed description thereof will be omitted as appropriate. The monitoring camera 300 includes a control unit 101, an operation unit 102, an imaging lens 103, an imaging element 104, a signal processing unit 105, a display control unit 106, a display unit 107, and an AF detection processing unit 201. An AF detection storage unit 202, an AF control unit 112, an image recording processing unit 113, a recording medium 114, an object recognition unit 301, an object tracking unit 302, and an AF detection value generation unit 303 are included.

The control unit 101 controls the operation of each unit of the monitoring camera 300 based on the control program. The operation unit 102 is connected to the control unit 101, and constitutes a user interface that receives various operations by the user. The imaging lens 103 includes a plurality of lenses and the like, and condenses light incident from a subject and guides the light to the imaging surface (imaging area) of the imaging element 104. The imaging lens 103 has a focus lens, and by controlling the focus lens, focus control is enabled. Further, the imaging lens 103 has an aperture and a shutter, and exposure control can be performed by controlling the aperture position and the shutter speed.

The signal processing unit 105 applies analog signal processing such as amplification to the analog image signal from the imaging element 104, and further A / D (Analog / Digital) converts the image signal obtained as a result. Further, the signal processing unit 105 applies digital signal processing such as noise removal processing to image data represented by a digital signal obtained by A / D conversion, and the image data obtained as a result is displayed on the display control unit 106, The information is supplied to the object recognition unit 301, the AF detection processing unit 201, and the image recording processing unit 113. The display control unit 106 causes the display unit 107 to display a captured image corresponding to the image data from the signal processing unit 105.

The object recognition unit 301 recognizes a robot arm to be excluded from the image data from the image data, and supplies information such as the feature amount to the object tracking unit 302 as tracking target information. The object tracking unit 302 detects an object position, in this case, a robot arm position from image data using tracking target information from the object recognition unit 301, and obtains object position information (including shape information).

The object tracking unit 302 supplies object position information to the display control unit 106. The display control unit 106 superimposes and displays a tracking frame on the captured image based on the object position information. Further, the object tracking unit 302 supplies the object position information to the AF detection value generation unit 303 together with the image data ID identifying the frame of the image data used when the object position is determined.

A plurality of AF detection processing units 201 are arranged in the imaging area from the image data from the signal processing unit 105. For example, AF detection values (contrast detection values) of a plurality of AF detection areas (see FIG. 4) Get each one. The AF detection processing unit 201 supplies an AF detection value group to the AF detection storage unit 202 together with an image data ID identifying a frame of image data used when obtaining the detection value, and temporarily stores the AF detection value group.

The AF detection value generation unit 303 detects the detection data associated with the same image data ID from the AF detection storage unit 202 based on the object position information (including the shape information) supplied from the object tracking unit 302 and the image data ID. The AF detection values of a predetermined number of AF detection areas excluding the portion of the object position specified by the object position information in the value group are taken out from the AF detection storage unit 202 and integrated (adding average). An AF detection value (integral AF detection value) other than the arm is generated.

The method of setting a predetermined number of AF detection areas excluding the portion of the object position specified by the object position information is not particularly limited, but specifies the portion excluding all of the AF detection areas in which the object position is detected. The predetermined number of AF detection areas may be set. Further, among the AF detection areas in which the object position is detected, it is set as a predetermined number of AF detection areas to be identified except an area where the ratio occupied by the object position information is an arbitrary value or more, for example 50% or more. It is also good.

The AF detection value generation unit 303 supplies the generated AF detection value to the AF control unit 112. The AF control unit 112 determines the focus lens position at the time of showing the highest contrast based on the history of AF detection values obtained along with the movement operation of the focus lens, and notifies the imaging lens 103 of the result. The focus lens is driven to be the focus lens position when high contrast is shown.

Under the control of the control unit 101, the image recording processing unit 113 compresses the image data from the signal processing unit 105 according to a predetermined compression method such as the Moving Picture Experts Group (MPEG) method, and the compressed image data It is recorded on the recording medium 114 as surveillance image data.

In the digital still camera 300 shown in FIG. 8, when the object position detection process and the AF detection process are performed in parallel, and the AF detection value is stored in the AF detection storage unit 202, the object position is detected. It becomes possible to refer to the AF detection value generated by the image data of the same frame as the used frame. Therefore, even if the object moves, it is possible to correctly obtain AF detection values for areas other than the object position, that is, areas other than the robot arm, and AF control for focusing on areas other than the robot arm can be performed with high accuracy.

<3. Third embodiment>
[Configuration Example of Digital Still Camera]
FIG. 9 shows a configuration example of the digital still camera 400. In the digital still camera 400, parts corresponding to those of the digital still camera 100 in FIG. 1 are given the same reference numerals, and the detailed description thereof will be omitted as appropriate. The digital still camera 400 includes a control unit 101, an operation unit 102, an imaging lens 103, an imaging device 104, a signal processing unit 105, a display control unit 106, a display unit 107, and an object recognition unit 108. An object tracking unit 109, an AE detection area setting unit 401, an AE detection processing unit 402, an AE control unit 403, an image recording processing unit 113, and a recording medium 114 are included.

The control unit 101 controls the operation of each unit of the digital still camera 400 based on a control program. The operation unit 102 is connected to the control unit 101, and constitutes a user interface that receives various operations by the user. The imaging lens 103 includes a plurality of lenses and the like, and condenses light incident from a subject and guides the light to the imaging surface (imaging area) of the imaging element 104. The imaging lens 103 has a focus lens, and by controlling the focus lens, focus control is enabled. Further, the imaging lens 103 has an aperture and a shutter, and exposure control can be performed by controlling the aperture position and the shutter speed.

The signal processing unit 105 applies analog signal processing such as amplification to the analog image signal from the imaging element 104, and further A / D (Analog / Digital) converts the image signal obtained as a result. Further, the signal processing unit 105 applies digital signal processing such as noise removal processing to image data represented by a digital signal obtained by A / D conversion, and the image data obtained as a result is displayed on the display control unit 106, The information is supplied to the subject recognition unit 108, the AE detection processing unit 402, and the image recording processing unit 113. The display control unit 106 causes the display unit 107 to display a captured image corresponding to the image data from the signal processing unit 105.

The subject recognition unit 108 recognizes a subject to be tracked from the image data, and supplies information such as the feature amount to the subject tracking unit 109 as tracking target information. The subject tracking unit 109 detects the subject position from the image data using the tracking target information from the subject recognition unit 108, and causes the display control unit 106 and the AE detection area setting unit 401 to provide subject position information (including shape information). Supply. The display control unit 106 superimposes and displays a tracking frame on the captured image based on the subject position information.

The AE detection area setting unit 401 sets an AE detection area (AE detection frame) as a photometric area at the subject position in the imaging area based on the subject position information supplied from the subject tracking unit 109, and AE The signal is supplied to the detection processing unit 402. The AE detection processing unit 402 acquires an AE detection value (brightness level value) of the AE detection area from the image data based on setting information of the AE detection area, and supplies the AE detection value to the AE control unit 403.

The AE control unit 403 determines the aperture position and shutter speed of the imaging lens 103 and the gain of the signal processing unit 105 so that an appropriate exposure can be obtained from the AE detection value. Then, the AE control unit 403 notifies the imaging lens 103 of the determined aperture position and shutter speed, and controls the aperture position and shutter speed of the imaging lens 103. Further, the AE control unit 403 notifies the signal processing unit 105 of the determined gain, and controls the gain in the signal processing unit 105.

Under control of the control unit 101, for example, when the shutter button is pressed, the image recording processing unit 113 compresses the image data from the signal processing unit 105 according to a predetermined compression method such as JPEG (Joint Photographic Experts Group) method. Image data compressed according to the method and compressed is recorded on the recording medium 114. The recording medium 114 is, for example, a recording medium such as a memory card, and can be easily attached to and detached from the digital still camera 400.

The flowchart of FIG. 10 shows an example of the procedure of the AE processing in the digital still camera 400 of FIG. In step ST41, for example, the AE process is started by turning on the power. Note that, thereafter, the processing of this flowchart is repeated for each frame. After step ST41, in step ST42, an image signal is read from the imaging element 104, and in step ST43, the signal processing unit 105 generates image data.

Next, in step ST44, the subject recognition unit 108 recognizes, for example, the subject specified by the user's touch panel operation, and in step ST45, the subject tracking unit 109 detects the subject position from the image data. Position information (including shape information) indicating the position is obtained. Next, in step ST46, the AE detection area setting unit 401 sets an AE detection area (AE detection frame) at the object position in the imaging area. Next, in step ST47, the AE detection processing unit 402 acquires an AE detection value of the AE detection area from the image data.

Next, in step ST48, the AE control unit 403 determines the aperture position, shutter speed, and gain so as to obtain an appropriate exposure from the AE detection value. Then, in step ST49, the AE control unit 403 instructs the imaging lens 103 on the determined diaphragm position and shutter speed, and instructs the signal processing unit 105 on the determined gain.

Next, in step ST50, driving to the instructed aperture position and setting of the instructed shutter speed are performed by the imaging lens 103. Next, in step ST51, the signal processing unit 105 performs setting of the instructed gain. Then, in step ST52, a series of AE processing is ended.

In the above-described contrast AE processing (see FIG. 10) in the digital still camera 400 of FIG. 9, after subject recognition is performed, an AE detection area (AE detection frame) is set, and thereafter AE detection is started to The procedure is to obtain an AE detection value. Therefore, for a moving subject, the subject position at the time of subject detection and the subject position at the start of AE detection are shifted, and it is difficult to obtain an AE detection value corresponding to the subject position. , High precision AE control is impossible.

FIG. 11 shows a configuration example of a digital still camera 500 according to the third embodiment. In this digital still camera 500, parts corresponding to those of the

digital still cameras

200 and 400 in FIG. 3 and FIG. 9 are assigned the same reference numerals, and the detailed description thereof is omitted as appropriate. The digital still camera 500 includes a control unit 101, an operation unit 102, an imaging lens 103, an imaging device 104, a signal processing unit 105, a display control unit 106, a display unit 107, and an object recognition unit 108. An object tracking unit 109, an AE detection processing unit 501, an AE detection storage unit 502, an AE detection value generation unit 503, an AE control unit 403, an image recording processing unit 113, and a recording medium 114 are included.

The control unit 101 controls the operation of each unit of the digital still camera 500 based on a control program. The operation unit 102 is connected to the control unit 101, and constitutes a user interface that receives various operations by the user. The imaging lens 103 includes a plurality of lenses and the like, and condenses light incident from a subject and guides the light to the imaging surface (imaging area) of the imaging element 104. The imaging lens 103 has a focus lens, and by controlling the focus lens, focus control is enabled. Further, the imaging lens 103 has an aperture and a shutter, and exposure control can be performed by controlling the aperture position and the shutter speed.

The signal processing unit 105 applies analog signal processing such as amplification to the analog image signal from the imaging element 104, and further A / D (Analog / Digital) converts the image signal obtained as a result. Further, the signal processing unit 105 applies digital signal processing such as noise removal processing to image data represented by a digital signal obtained by A / D conversion, and the image data obtained as a result is displayed on the display control unit 106, The information is supplied to the subject recognition unit 108, the AE detection processing unit 501, and the image recording processing unit 113. The display control unit 106 causes the display unit 107 to display a captured image corresponding to the image data from the signal processing unit 105.

The subject recognition unit 108 recognizes a subject from the image data, and supplies information such as its feature amount to the subject tracking unit 109 as tracking target information. The subject tracking unit 109 detects the subject position from the image data using the tracking target information from the subject recognition unit 108, and obtains subject position information (including shape information).

The subject tracking unit 109 supplies subject position information to the display control unit 106. The display control unit 106 superimposes and displays a tracking frame on the captured image based on the subject position information. The subject tracking unit 109 also supplies subject position information to the AE detection value generation unit 503 together with an image data ID identifying a frame of image data used when the subject position is determined.

A plurality of AE detection processing units 501 are arranged in the imaging area from the image data from the signal processing unit 105, for example, AE detection areas as a plurality of photometric areas spread in the imaging area (similar to the AF detection area shown in FIG. Acquisition of the AE detection values (brightness level values) of The AE detection processing unit 501 supplies an AF detection value group to the AE detection storage unit 502 together with an image data ID identifying a frame of image data used when obtaining the detection value, and temporarily stores the AF detection value group.

The AF detection value generation unit 503 detects the detection data associated with the same image data ID from the AE detection storage unit 502 based on the subject position information (including the shape information) supplied from the subject tracking unit 109 and the image data ID. AE detection values of a predetermined number of AE detection areas corresponding to the subject position specified by the subject position information among the value group are taken out from the AE detection storage unit 502 and integrated (averaged) to obtain AE detection values at the subject position ( Integral AE detection value) is generated.

The method of setting the predetermined number of AE detection areas corresponding to the subject position specified by the subject position information is not particularly limited. For example, all AE detection areas in which the subject is detected based on the subject position information May be set as a predetermined number of AE detection areas to be identified. Further, among AE detection areas in which a subject is detected, an area in which the ratio occupied by subject information is an arbitrary value or more, for example, 50% or more may be set as a predetermined number of AE detection areas to be identified. Further, among the AE detection areas in which the subject is detected, an area including areas other than the subject information (such as background) may be set as the specified number of AE detection areas to be identified.

The AE detection value generation unit 503 supplies the generated AE detection value to the AE control unit 403. The AE control unit 403 determines the aperture position and shutter speed of the imaging lens 103 and the gain of the signal processing unit 105 so that an appropriate exposure can be obtained from the AE detection value. Then, the AE control unit 403 notifies the imaging lens 103 of the determined aperture position and shutter speed, and controls the aperture position and shutter speed of the imaging lens 103. Further, the AE control unit 403 notifies the signal processing unit 105 of the determined gain, and controls the gain in the signal processing unit 105.

Under control of the control unit 101, for example, when the shutter button is pressed, the image recording processing unit 113 compresses the image data from the signal processing unit 105 according to a predetermined compression method such as JPEG (Joint Photographic Experts Group) method. Image data compressed according to the method and compressed is recorded on the recording medium 114. The recording medium 114 is, for example, a recording medium such as a memory card, and can be easily attached to and detached from the digital still camera 500.

The flowchart of FIG. 12 shows an example of the procedure of the AE processing in the digital still camera 500 of FIG. In step ST61, for example, the AE process is started by turning on the power. Note that, thereafter, the processing of this flowchart is repeated for each frame. After the process of step ST61, in step ST62, an image signal is read from the imaging element 104, and in step ST63, the signal processing unit 105 generates image data. Next, the processes of steps ST64 and ST65 are performed, and the processes of steps ST66 and ST67 are performed in parallel.

In step S64, the subject recognition unit 108 recognizes, for example, the subject specified by the user's touch panel operation, and in step ST65, the subject tracking unit 109 detects the subject position from the image data, and indicates the position of the subject on the captured image. Position information (including shape information) is obtained. On the other hand, in step ST66, the AE detection processing unit 501 acquires AE detection values of a plurality of AE detection areas arranged in the imaging area, and in step ST67 the detection values of all detection areas, that is, the detection value group obtains them. Stored in the AE detection storage unit 502 together with an image data ID identifying a frame of image data used for

Next, in step ST68, the AE detection value generation unit 503 generates the same image data from the AE detection storage unit 502 based on the object position information (including the shape information) supplied from the object tracking unit 109 and the image data ID. AE detection values of a predetermined number of AE detection areas corresponding to the subject position specified by the subject position information among the detected value groups associated with the ID are acquired and integrated (summation average), and AE detection at the subject position A value (integral AE detection value) is generated.

Next, in step ST69, the AE control unit 403 determines, from the AE detection value, the aperture position, the shutter speed, and the gain such that the appropriate exposure can be obtained. Then, in step ST70, the AE control unit 403 instructs the imaging lens 103 of the determined diaphragm position and shutter speed, and instructs the signal processing unit 105 of the determined gain.

Next, in step ST71, drive to the instructed aperture position and setting of the instructed shutter speed are performed by the imaging lens 103. Next, in step ST72, the signal processing unit 105 performs setting of the instructed gain. Then, in step ST73, a series of AE processing is ended.

In the digital still camera 500 shown in FIG. 11, the subject position detection process and the AE detection process are performed in parallel, and when the AE detection value is stored in the AE detection storage unit 502, the subject position is detected. It becomes possible to refer to the AE detection value generated by the image data of the same frame as the used frame. Therefore, the AE detection area does not always shift with respect to a moving subject, the AE detection value for the subject can be correctly obtained, and highly accurate AE control becomes possible.

Further, in the digital still camera 500 shown in FIG. 11, the AE detection areas are spread over the entire area or almost the entire area of the imaging area (the same arrangement as the AF detection area shown in FIG. 4). Thus, the predetermined number of AF detection areas can be properly specified to obtain the AE detection value for the subject with high accuracy, and the AE control can be performed with high accuracy.

< 4 . Applications to mobiles>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure is realized as a device mounted on any type of mobile object such as a car, an electric car, a hybrid electric car, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, a robot May be

FIG. 13 is a block diagram showing a schematic configuration example of a vehicle control system that is an example of a mobile control system to which the technology according to the present disclosure can be applied.

Vehicle control system 12000 includes a plurality of electronic control units connected via communication network 12001. In the example illustrated in FIG. 13 , the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an in-vehicle information detection unit 12040, and an integrated control unit 12050. Further, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, an audio image output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated.

The driveline control unit 12010 controls the operation of devices related to the driveline of the vehicle according to various programs. For example, the drive system control unit 12010 includes a drive force generation device for generating a drive force of a vehicle such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, and a steering angle of the vehicle. It functions as a control mechanism such as a steering mechanism that adjusts and a braking device that generates a braking force of the vehicle.

Body system control unit 12020 controls the operation of various devices equipped on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a keyless entry system, a smart key system, a power window device, or a control device of various lamps such as a headlamp, a back lamp, a brake lamp, a blinker or a fog lamp. In this case, the body system control unit 12020 may receive radio waves or signals of various switches transmitted from a portable device substituting a key. Body system control unit 12020 receives the input of these radio waves or signals, and controls a door lock device, a power window device, a lamp and the like of the vehicle.

Outside vehicle information detection unit 12030 detects information outside the vehicle equipped with vehicle control system 12000. For example, an imaging unit 12031 is connected to the external information detection unit 12030. The out-of-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle, and receives the captured image. The external information detection unit 12030 may perform object detection processing or distance detection processing of a person, a vehicle, an obstacle, a sign, characters on a road surface, or the like based on the received image.

The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output an electric signal as an image or can output it as distance measurement information. The light received by the imaging unit 12031 may be visible light or non-visible light such as infrared light.

In-vehicle information detection unit 12040 detects in-vehicle information. For example, a driver state detection unit 12041 that detects a state of a driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera for imaging the driver, and the in-vehicle information detection unit 12040 determines the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041. It may be calculated or it may be determined whether the driver does not go to sleep.

The microcomputer 12051 calculates a control target value of the driving force generation device, the steering mechanism or the braking device based on the information inside and outside the vehicle acquired by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040, and a drive system control unit A control command can be output to 12010. For example, the microcomputer 12051 realizes functions of an advanced driver assistance system (ADAS) including collision avoidance or shock mitigation of a vehicle, follow-up traveling based on an inter-vehicle distance, vehicle speed maintenance traveling, vehicle collision warning, vehicle lane departure warning, etc. It is possible to perform coordinated control aiming at

Further, the microcomputer 12051 controls the driving force generating device, the steering mechanism, the braking device, and the like based on the information around the vehicle acquired by the outside information detecting unit 12030 or the in-vehicle information detecting unit 12040 so that the driver can Coordinated control can be performed for the purpose of automatic driving that travels autonomously without depending on the operation.

Further, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the external information detection unit 12030. For example, the microcomputer 12051 controls the headlamp according to the position of the preceding vehicle or oncoming vehicle detected by the external information detection unit 12030, and performs cooperative control for the purpose of antiglare such as switching the high beam to the low beam. It can be carried out.

The audio image output unit 12052 transmits an output signal of at least one of audio and image to an output device capable of visually or aurally notifying information to a passenger or the outside of a vehicle. In the example of FIG. 13 , an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are illustrated as the output device. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

FIG. 14 is a diagram illustrating an example of the installation position of the imaging unit 12031.

In FIG. 14 ,

imaging units

12101, 12102, 12103, 12104, and 12105 are provided as the imaging unit 12031.

The

imaging units

12101, 12102, 12103, 12104, and 12105 are provided, for example, on the front nose of the vehicle 12100, a side mirror, a rear bumper, a back door, an upper portion of a windshield of a vehicle interior, and the like. The imaging unit 12101 provided in the front nose and the imaging unit 12105 provided in the upper part of the windshield in the vehicle cabin mainly acquire an image in front of the vehicle 12100. The

imaging units

12102 and 12103 included in the side mirror mainly acquire an image of the side of the vehicle 12100. The imaging unit 12104 provided in the rear bumper or the back door mainly acquires an image of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield in the passenger compartment is mainly used to detect a leading vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

Note that FIG. 14 shows an example of the imaging range of the imaging units 12101 to 12104. The imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, the imaging ranges 12112 and 12113 indicate the imaging ranges of the

imaging units

12102 and 12103 provided on the side mirrors, and the imaging range 12114 indicates The imaging range of the imaging part 12104 provided in the rear bumper or the back door is shown. For example, by overlaying the image data captured by the imaging units 12101 to 12104, a bird's eye view of the vehicle 12100 viewed from above can be obtained.

At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera including a plurality of imaging devices, or an imaging device having pixels for phase difference detection.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 measures the distance to each three-dimensional object in the imaging ranges 12111 to 12114, and the temporal change of this distance (relative velocity with respect to the vehicle 12100). In particular, it is possible to extract a three-dimensional object traveling at a predetermined speed (for example, 0 km / h or more) in substantially the same direction as the vehicle 12100 as a leading vehicle, in particular by finding the it can. Further, the microcomputer 12051 can set an inter-vehicle distance to be secured in advance before the preceding vehicle, and can perform automatic brake control (including follow-up stop control), automatic acceleration control (including follow-up start control), and the like. As described above, it is possible to perform coordinated control for the purpose of automatic driving or the like that travels autonomously without depending on the driver's operation.

For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 converts three-dimensional object data relating to three-dimensional objects into two-dimensional vehicles such as two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, telephone poles, and other three-dimensional objects. It can be classified, extracted and used for automatic avoidance of obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 into obstacles visible to the driver of the vehicle 12100 and obstacles difficult to see. Then, the microcomputer 12051 determines the collision risk indicating the degree of risk of collision with each obstacle, and when the collision risk is a setting value or more and there is a possibility of a collision, through the audio speaker 12061 or the display unit 12062 By outputting an alarm to the driver or performing forcible deceleration or avoidance steering via the drive system control unit 12010, driving support for collision avoidance can be performed.

At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the imaging units 12101 to 12104. Such pedestrian recognition is, for example, a procedure for extracting feature points in images captured by the imaging units 12101 to 12104 as an infrared camera, and pattern matching processing on a series of feature points indicating the outline of an object to determine whether it is a pedestrian or not The procedure is to determine When the microcomputer 12051 determines that a pedestrian is present in the captured image of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio image output unit 12052 generates a square outline for highlighting the recognized pedestrian. The display unit 12062 is controlled so as to display a superimposed image. Further, the audio image output unit 12052 may control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

The example of the vehicle control system to which the technology according to the present disclosure can be applied has been described above. Among the configurations described above, the technology according to the present disclosure can also obtain distance information with a subject (a leading vehicle or a pedestrian, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like) from the imaging units 12101 to 12104. Used.

By performing full-frame detection on the ranging area, it is possible to perform more accurate distance measurement even when measuring the distance from the moving object. Here, the method of measuring the distance to the subject based on the detection value detected from the distance measurement area is not particularly limited, but the following known method may be used. That is, the imaging unit has a distance measurement window, and can measure the distance to the subject based on the detection value detected from the distance measurement area by the method of triangulation.

< 5 . Application example to endoscopic surgery system>
The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be applied to an endoscopic surgery system.

FIG. 15 is a diagram showing an example of a schematic configuration of an endoscopic surgery system to which the technology (the present technology) according to the present disclosure can be applied.

FIG. 15 illustrates a surgeon (doctor) 11131 performing surgery on a patient 11132 on a patient bed 11133 using the endoscopic surgery system 11000. As shown, the endoscopic surgery system 11000 includes an endoscope 11100, other surgical instruments 11110 such as an insufflation tube 11111 and an energy treatment instrument 11112, and a support arm device 11120 for supporting the endoscope 11100. , A cart 11200 on which various devices for endoscopic surgery are mounted.

The endoscope 11100 includes a lens barrel 11101 whose region of a predetermined length from the tip is inserted into a body cavity of a patient 11132, and a camera head 11102 connected to a proximal end of the lens barrel 11101. In the illustrated example, the endoscope 11100 configured as a so-called rigid endoscope having a rigid barrel 11101 is illustrated, but even if the endoscope 11100 is configured as a so-called flexible mirror having a flexible barrel Good.

At the tip of the lens barrel 11101, an opening into which an objective lens is fitted is provided. A light source device 11203 is connected to the endoscope 11100, and light generated by the light source device 11203 is guided to the tip of the lens barrel by a light guide extended inside the lens barrel 11101, and an objective The light is emitted toward the observation target in the body cavity of the patient 11132 through the lens. In addition, the endoscope 11100 may be a straight endoscope, or may be a oblique endoscope or a side endoscope.

An optical system and an imaging device are provided inside the camera head 11102, and the reflected light (observation light) from the observation target is condensed on the imaging device by the optical system. The observation light is photoelectrically converted by the imaging element to generate an electric signal corresponding to the observation light, that is, an image signal corresponding to the observation image. The image signal is transmitted as RAW data to a camera control unit (CCU: Camera Control Unit) 11201.

The CCU 11201 is configured by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and centrally controls the operations of the endoscope 11100 and the display device 11202. Furthermore, the CCU 11201 receives an image signal from the camera head 11102 and performs various image processing for displaying an image based on the image signal, such as development processing (demosaicing processing), on the image signal.

The display device 11202 displays an image based on an image signal subjected to image processing by the CCU 11201 under control of the CCU 11201.

The light source device 11203 includes, for example, a light source such as an LED (light emitting diode), and supplies the endoscope 11100 with irradiation light at the time of imaging an operation part or the like.

The input device 11204 is an input interface to the endoscopic surgery system 11000. The user can input various information and input instructions to the endoscopic surgery system 11000 via the input device 11204. For example, the user inputs an instruction to change the imaging condition (type of irradiated light, magnification, focal length, and the like) by the endoscope 11100, and the like.

The treatment tool control device 11205 controls the drive of the energy treatment tool 11112 for ablation of tissue, incision, sealing of a blood vessel, and the like. The insufflation apparatus 11206 is a gas within the body cavity via the insufflation tube 11111 in order to expand the body cavity of the patient 11132 for the purpose of securing a visual field by the endoscope 11100 and securing a working space of the operator. Send The recorder 11207 is a device capable of recording various types of information regarding surgery. The printer 11208 is an apparatus capable of printing various types of information regarding surgery in various types such as text, images, and graphs.

The light source device 11203 that supplies the irradiation light when imaging the surgical site to the endoscope 11100 can be configured of, for example, an LED, a laser light source, or a white light source configured by a combination of these. When a white light source is configured by a combination of RGB laser light sources, the output intensity and output timing of each color (each wavelength) can be controlled with high precision. It can be carried out. Further, in this case, the laser light from each of the RGB laser light sources is irradiated to the observation target in time division, and the drive of the image pickup element of the camera head 11102 is controlled in synchronization with the irradiation timing to cope with each of RGB. It is also possible to capture a shot image in time division. According to the method, a color image can be obtained without providing a color filter in the imaging device.

In addition, the drive of the light source device 11203 may be controlled so as to change the intensity of the light to be output every predetermined time. The drive of the imaging device of the camera head 11102 is controlled in synchronization with the timing of the change of the light intensity to acquire images in time division, and by combining the images, high dynamic without so-called blackout and whiteout is obtained. An image of the range can be generated.

The light source device 11203 may be configured to be able to supply light of a predetermined wavelength band corresponding to special light observation. In special light observation, for example, the mucous membrane surface layer is irradiated by irradiating narrow band light as compared with irradiation light (that is, white light) at the time of normal observation using the wavelength dependency of light absorption in body tissue. The so-called narrow band imaging (Narrow Band Imaging) is performed to image a predetermined tissue such as a blood vessel with high contrast. Alternatively, in special light observation, fluorescence observation may be performed in which an image is obtained by fluorescence generated by irradiation with excitation light. In fluorescence observation, body tissue is irradiated with excitation light and fluorescence from the body tissue is observed (autofluorescence observation), or a reagent such as indocyanine green (ICG) is locally injected into body tissue and the body tissue is Excitation light corresponding to the fluorescence wavelength of the reagent can be irradiated to obtain a fluorescence image or the like. The light source device 11203 can be configured to be able to supply narrow band light and / or excitation light corresponding to such special light observation.

Figure 16 is a block diagram showing an example of a functional configuration of the camera head 11102 and CCU11201 shown in FIG. 15.

The camera head 11102 includes a lens unit 11401, an imaging unit 11402, a drive unit 11403, a communication unit 11404, and a camera head control unit 11405. The CCU 11201 includes a communication unit 11411, an image processing unit 11412, and a control unit 11413. The camera head 11102 and the CCU 11201 are communicably connected to each other by a transmission cable 11400.

The lens unit 11401 is an optical system provided at a connection portion with the lens barrel 11101. The observation light taken in from the tip of the lens barrel 11101 is guided to the camera head 11102 and is incident on the lens unit 11401. The lens unit 11401 is configured by combining a plurality of lenses including a zoom lens and a focus lens.

The imaging device constituting the imaging unit 11402 may be one (a so-called single-plate type) or a plurality (a so-called multi-plate type). When the imaging unit 11402 is configured as a multi-plate type, for example, an image signal corresponding to each of RGB may be generated by each imaging element, and a color image may be obtained by combining them. Alternatively, the imaging unit 11402 may be configured to have a pair of imaging devices for acquiring image signals for right eye and left eye corresponding to 3D (dimensional) display. By performing 3D display, the operator 11131 can more accurately grasp the depth of the living tissue in the operation site. When the imaging unit 11402 is configured as a multi-plate type, a plurality of lens units 11401 may be provided corresponding to each imaging element.

In addition, the imaging unit 11402 may not necessarily be provided in the camera head 11102. For example, the imaging unit 11402 may be provided inside the lens barrel 11101 immediately after the objective lens.

The driving unit 11403 is configured by an actuator, and moves the zoom lens and the focusing lens of the lens unit 11401 by a predetermined distance along the optical axis under the control of the camera head control unit 11405. Thereby, the magnification and the focus of the captured image by the imaging unit 11402 can be appropriately adjusted.

The communication unit 11404 is configured of a communication device for transmitting and receiving various types of information to and from the CCU 11201. The communication unit 11404 transmits the image signal obtained from the imaging unit 11402 to the CCU 11201 as RAW data via the transmission cable 11400.

The communication unit 11404 also receives a control signal for controlling the drive of the camera head 11102 from the CCU 11201 and supplies the control signal to the camera head control unit 11405. The control signal includes, for example, information indicating that the frame rate of the captured image is designated, information indicating that the exposure value at the time of imaging is designated, and / or information indicating that the magnification and focus of the captured image are designated, etc. Contains information about the condition.

Note that the imaging conditions such as the frame rate, exposure value, magnification, and focus described above may be appropriately designated by the user, or may be automatically set by the control unit 11413 of the CCU 11201 based on the acquired image signal. Good. In the latter case, the so-called AE (Auto Exposure) function, AF (Auto Focus) function, and AWB (Auto White Balance) function are incorporated in the endoscope 11100.

The camera head control unit 11405 controls the drive of the camera head 11102 based on the control signal from the CCU 11201 received via the communication unit 11404.

The communication unit 11411 is configured by a communication device for transmitting and receiving various types of information to and from the camera head 11102. The communication unit 11411 receives an image signal transmitted from the camera head 11102 via the transmission cable 11400.

Further, the communication unit 11411 transmits a control signal for controlling driving of the camera head 11102 to the camera head 11102. The image signal and the control signal can be transmitted by telecommunication or optical communication.

An image processing unit 11412 performs various types of image processing on an image signal that is RAW data transmitted from the camera head 11102.

The control unit 11413 performs various types of control regarding imaging of a surgical site and the like by the endoscope 11100 and display of a captured image obtained by imaging of the surgical site and the like. For example, the control unit 11413 generates a control signal for controlling the drive of the camera head 11102.

Further, the control unit 11413 causes the display device 11202 to display a captured image in which a surgical site or the like is captured, based on the image signal subjected to the image processing by the image processing unit 11412. At this time, the control unit 11413 may recognize various objects in the captured image using various image recognition techniques. For example, the control unit 11413 detects a shape, a color, and the like of an edge of an object included in a captured image, thereby enabling a surgical tool such as forceps, a specific biological site, bleeding, mist when using the energy treatment tool 11112, and the like. It can be recognized. When displaying the captured image on the display device 11202, the control unit 11413 may superimpose various surgical support information on the image of the surgery section using the recognition result. The operation support information is superimposed and presented to the operator 11131, whereby the burden on the operator 11131 can be reduced and the operator 11131 can reliably proceed with the operation.

A transmission cable 11400 connecting the camera head 11102 and the CCU 11201 is an electric signal cable corresponding to communication of an electric signal, an optical fiber corresponding to optical communication, or a composite cable of these.

Here, in the illustrated example, communication is performed by wire communication using the transmission cable 11400, but communication between the camera head 11102 and the CCU 11201 may be performed wirelessly.

Heretofore, an example of the endoscopic surgery system to which the technology according to the present disclosure can be applied has been described. The technology according to the present disclosure can also be suitably used for a camera head 11102 used in endoscopic surgery. For example, a blood vessel or body tissue is recognized as a subject, a detection frame is specified based on the result, and focus control is performed based on the detection value, thereby improving AF accuracy at the time of surgery. In addition, in the case of automatically detecting a subject, there are cases where it is easy to recognize an artificial tool or the like such as the surgical tool 11110 or waste, as the subject, instead of the blood vessels or body tissue to be originally focused on. In this case. By performing control to specify a detection frame other than the frame in which the subject is detected, it is possible to improve the AF accuracy at the time of surgery. In addition, although the endoscopic surgery system was demonstrated as an example here, the technique which concerns on this indication may be applied to others, for example, a microscopic surgery system etc.

<6. Modified example>
In addition, in the above-mentioned embodiment, although the example of AF by contrast detection type was shown, this art can be suitably used also with combined use with AF by phase difference detection type. FIG. 17 shows an example in which a detection frame (ranging frame) using the present technology and a frame of a phase difference detection formula are combined. Although the number and area of the distance measurement frame are both larger than the number and area of the phase difference detection type frame in FIG. 17, the number of the range detection frame and the number of areas of the phase difference detection type is not particularly limited thereto. The area of the distance measurement frame may be large (small) and the number of frames may be small (large). Further, the number and area of the distance measurement frames may be the same as the number and area of the frames of the phase difference detection frame.

Here, in the case of FIG. 17, the effect of further improving the AF accuracy particularly in the hybrid AF can also be obtained. In the phase difference detection type AF, it is known that the accuracy of the AF is deteriorated in a high luminance portion or a repetitive pattern. As shown in FIG. 17, it is obtained from the distance measurement frame by combining the contrast AF detection frame (distance measurement frame) and the frame of the phase difference detection type, and arranging more distance measurement frames than the number of frames of the phase difference detection type. The detected contrast AF detection value enables high luminance determination and repetitive pattern detection within the frame of the phase difference detection formula. By this processing, in addition to high luminance and repetitive pattern detection by the phase difference detection type AF conventionally performed, high density detection by the contrast AF detection frame (ranging frame) can be performed, and the final AF Accuracy can be improved.

Further, the above-described embodiment discloses the present technology in the form of exemplification, and it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the scope of the present technology. That is, in order to determine the gist of the present technology, the claims should be taken into consideration.

Furthermore, the present technology can also be configured as follows.
(1) A detection processing unit that detects image signals input from an imaging area of an imaging unit in a plurality of detection areas arranged in the imaging area, and generates detection values corresponding to all detection areas,
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An image processing apparatus comprising: a control unit that specifies a detected value corresponding to an area.
(2) The image processing apparatus according to (1), wherein the detection area is a ranging area.
(3) The image processing apparatus according to (2), wherein the control unit performs focus control based on the identified detection value.
(4) The image processing apparatus according to (1), wherein the detection area is a photometric area.
(5) The image processing apparatus according to (4), wherein the control unit performs exposure control based on the identified detection value.
(6) The image processing apparatus according to any one of (1) to (5), further including: a detection value storage unit that stores the detection value generated by the detection processing unit.
(7) The control unit stores the detection value of the detection area set corresponding to the object detected based on the image signal of the predetermined frame by the object detection unit in the detection value storage unit. The image processing apparatus according to (6), wherein the image processing apparatus according to (6) above is specified from the detection value corresponding to the entire detection area generated by detecting an image signal of a frame.
(8) The image processing apparatus according to any one of (1) to (7), wherein the detection area is spread over the entire area of the imaging area.
(9) The process of generating the detection value in the detection processing unit and the process of detecting the object in the object detection unit are performed in parallel. The image processing according to any one of (1) to (8) apparatus.
(10) A plurality of phase difference detection areas are further arranged in the imaging area,
The image processing apparatus according to (2) or (3), wherein the number of the ranging areas is larger than the number of the phase difference detection areas.
(11) The image processing apparatus according to any one of (1) to (10), wherein in the setting of the detection area corresponding to the subject, the area where the subject is detected is set as a detection area.
(12) In the setting of the detection area corresponding to the subject, an area excluding the area where the subject is detected is set as a detection area. The image processing apparatus according to any one of (1) to (10).
(13) The detection processing unit detects an image signal input from the imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area, and generates a detection value corresponding to each of the entire detection areas,
A subject detection unit detects a subject based on the image signal,
The control unit sets a detection area corresponding to the subject from the entire detection area based on the detection result of the subject detection unit, and based on the detection value corresponding to each of the full detection areas generated by the detection processing unit An image processing method for specifying a detection value corresponding to the set detection area.
(14) an imaging unit,
A detection processing unit that detects image signals input from an imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area and generates detection values corresponding to each of the entire detection areas;
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An imaging apparatus comprising a control unit that specifies a detection value corresponding to an area.

100 ... digital still camera 101 ... control unit 102 ... operation unit 103 ... imaging lens 104 ... imaging device 105 ... signal processing unit 106 ... display control unit 107 ... display Unit 108 ... object recognition unit 109 ... object tracking unit 110 ... AF detection area setting unit 111 ... AF detection processing unit 112 ... AF control unit 113 ... image recording processing unit 114 · · · Recording media 200: digital still camera 201: AF detection processing unit 202: AF detection storage unit 203: AF detection value generation unit 300: monitoring camera 301: object recognition unit 302 Object tracking unit 303 AF detection value generation unit 400 Digital still camera 401 AE detection area setting unit 402 AE detection processing unit 4 3 ... AE control unit 500 ... digital still camera 501 ... AE detection processing unit 502 ... AE detector storage unit 503 ... AE detection value generation unit

Claims

A detection processing unit that detects image signals input from an imaging area of an imaging unit in a plurality of detection areas arranged in the imaging area and generates detection values corresponding to all detection areas;
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An image processing apparatus comprising: a control unit that specifies a detected value corresponding to an area.
The image processing apparatus according to claim 1, wherein the detection area is a ranging area.
The image processing apparatus according to claim 2, wherein the control unit performs focus control based on the identified detection value.
The image processing apparatus according to claim 1, wherein the detection area is a photometric area.
The image processing apparatus according to claim 4, wherein the control unit performs exposure control based on the identified detection value.
The image processing apparatus according to claim 1, further comprising a detection value storage unit that stores the detection value generated by the detection processing unit.
The control unit controls the detection value of the detection area set corresponding to the object detected based on the image signal of the predetermined frame by the object detection unit, the image of the predetermined frame stored in the detection value storage unit. The image processing apparatus according to claim 6, wherein a signal is detected to specify from detection values corresponding to the entire detection area generated.
The image processing apparatus according to claim 1, wherein the detection area is spread over the entire area of the imaging area.
The image processing apparatus according to claim 1, wherein the process of generating the detection value in the detection processing unit and the process of detecting the object in the object detection unit are performed in parallel.
A plurality of phase difference detection areas are further arranged in the imaging area,
The image processing apparatus according to claim 2, wherein the number of the ranging areas is larger than the number of the phase difference detection areas.
The image processing apparatus according to claim 1, wherein in the setting of the detection area corresponding to the subject, an area in which the subject is detected is set as a detection area.
The image processing apparatus according to claim 1, wherein in the setting of the detection area corresponding to the subject, an area excluding the area where the subject is detected is set as a detection area.
A detection processing unit detects image signals input from an imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area, and generates detection values corresponding to all the detection areas,
A subject detection unit detects a subject based on the image signal,
The control unit sets a detection area corresponding to the subject from the entire detection area based on the detection result of the subject detection unit, and based on the detection value corresponding to each of the full detection areas generated by the detection processing unit An image processing method for specifying a detection value corresponding to the set detection area.
An imaging unit,
A detection processing unit that detects image signals input from an imaging area of the imaging unit in a plurality of detection areas arranged in the imaging area, and generates detection values corresponding to all detection areas;
A subject detection unit that detects a subject based on the image signal;
The detection area corresponding to the subject is set from the full detection area based on the detection result of the subject detection unit, and the detection set from the detection values corresponding to the full detection area generated by the detection processing unit An imaging apparatus comprising a control unit that specifies a detection value corresponding to an area.