JP2014147062A - Image pickup device, and control method and program therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image pickup device capable of reducing an amount of pixel data to cut down a memory capacity while maintaining a refocus range during refocusing.SOLUTION: The image pickup device comprises: image pickup means having a two-dimensional array of a plurality of pixels each converting an optical image into a pixel signal by photoelectric conversion, and a photographing optical system having a focus lens; and a microlens array comprising a plurality of microlenses arranged between the image pickup means and the focus lens, each microlens corresponding to one divided block when the two-dimensional array of pixels is divided in block units, each block including a prescribed number of pixels. By detecting a subject included in a photographing screen on the basis of a read-out pixel signal, the image pickup device controls an amount of reduction for a pixel signal read out from a divided block corresponding to a detection area for the subject detected on the photographing screen, and an amount of reduction for a pixel signal read out from a divided block corresponding to an area outside the detection area.

Description

  The present invention relates to an imaging apparatus, and more particularly to an imaging apparatus that captures an optical image of a subject to acquire light field data and a control method thereof.

  For example, by arranging a microlens array arranged at a ratio of one pixel to a plurality of pixels on the front surface of the image sensor, not only the two-dimensional intensity distribution of light but also information on the incident direction of light incident on the image sensor can be obtained. It is possible to obtain three-dimensional information of space. A camera (imaging device) capable of obtaining such three-dimensional information of the subject space is called a light field camera. The three-dimensional information of the subject space is called light field data. By acquiring the light field data and image reconstruction processing after shooting, the focus position of the image, the shooting viewpoint, the depth of field, Image processing called refocusing such as adjustment becomes possible.

  FIG. 11 is a diagram illustrating a configuration example for realizing a focus function for obtaining three-dimensional information of a subject space in a light field camera.

  In the figure, a microlens 1104 is disposed on the image plane 1103, and an image sensor 1105 is disposed on the back surface of the microlens 1104. The light condensed on the focus lens 1102 is projected onto the microlens 1104 disposed on the image plane 1103.

  In a light field camera that is an imaging apparatus having such a structure, a light beam from a subject 1101 projected onto a microlens 1104 is divided by the microlens 1104 and then projected onto an imaging element 1105. Thereby, the information imaged by the image sensor 1105 includes information on a plurality of light beam directions. As described above, the pixel data (light field data) obtained as a result of image formation on the image sensor by the focus function of the light field camera has a plurality of different parallaxes, thereby enabling refocusing of the recorded image.

The number LFDATA (number of pixels) of the imaging pixel data by the light field camera described above is given by assuming that the number of pixels of the imaging device is N and the number of divisions of the image divided by one microlens is M.
LFDATA = N x M
It becomes. In FIG. 11, the number M of pixels divided at least in the direction parallel to the imaging surface of the imaging element 1105 is four.

  As described above, the amount of pixel data captured by the light field camera increases in accordance with the number of divisions, and the imaging device having the focus function that enables acquisition of the light field data performs various signal processing on the image pixel data. In this case, the processing load increases. Furthermore, when capturing image data into various image signal processing means, a large capacity data storage means is required.

  As a result, power consumption increases as the data processing load increases, and the cost increases due to an increase in the capacity of the data storage means accompanying an increase in the amount of processing data.

As a method for reducing the amount of data, a method of reducing the amount of pixel data to be processed by adding pixel outputs in an image sensor has been considered.
(For example, see Patent Document 1)

JP 2009-3122 A

  However, in an imaging apparatus having a focus function of a light field camera, when the amount of pixel data to be processed is reduced by pixel addition in the imaging device, there are the following problems. Since information necessary for refocusing is lost due to the reduction in the amount of pixel data, when refocusing a captured image during reproduction, the refocus range becomes shallow.

  In view of the above problems, the present invention can reduce the amount of data processed by various image signal processing means while maintaining the refocus range at the time of refocus after shooting, and reduce the capacity of the means for storing the data. The purpose is to provide a light field camera that can also be used.

  In order to achieve the above object, an imaging unit having a two-dimensional array of a plurality of pixels for converting an optical image into a pixel signal by photoelectric conversion, and a focus lens, and being disposed between the imaging unit and the focus lens. An imaging optical system having a microlens array composed of a plurality of microlenses, and each microlens is divided into one divided block when the two-dimensional array of pixels is divided into blocks each including a predetermined number of pixels. A corresponding imaging device includes a reading unit that reads out the pixel signal from a plurality of pixels, a subject detection unit that detects a subject included in the shooting screen based on the pixel signal read out by the reading unit, and a subject detection The reading means is controlled based on the detection result of the object by the means, and corresponds to the detected object detection area on the shooting screen. A control means for variably controlling the reduction of the pixel signals read out from the divided blocks corresponding to the area other than the reduction and the detection region of the pixel signals read out from the divided blocks.

  According to the present invention, it is possible to provide an imaging apparatus capable of reducing the amount of light field data processed by the image signal processing unit while maintaining the refocus range at the time of refocusing a recorded image as much as possible.

1 is a block configuration diagram of an imaging apparatus according to a first embodiment of the present invention. FIG. 2 is a diagram illustrating a divided pixel configuration and a captured image of the imaging apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating a flowchart of an imaging operation of the imaging apparatus according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating addition of divided pixel data in the imaging apparatus according to the first embodiment of the present invention. The figure which shows the to-be-photographed object detection area | region in the imaged frame image. FIG. 5 is a diagram illustrating a flowchart of another imaging operation of the imaging apparatus according to the first embodiment of the present invention. The figure which shows the some subject detection area in the imaged frame image. The figure which shows the flowchart of the imaging operation which concerns on 2nd Example of this invention. The figure which shows the division | segmentation of the frequency domain in the frame image which concerns on 2nd Example of this invention. The figure which shows addition of the division | segmentation pixel data in the imaging device which concerns on 2nd Example of this invention. The figure which shows the structural example for implement | achieving the focus function of a light field camera. The figure for demonstrating the pixel data based on a modification in the 1st or 2nd Example of this invention. The figure for demonstrating the read-out area | region of the pixel data which concerns on the modification of the 1st or 2nd Example of this invention.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of the overall configuration of a light field camera as an example of an imaging apparatus according to the first embodiment of the present invention.

  In the figure, reference numeral 101 denotes a zoom lens, and reference numeral 102 denotes a focus lens, which constitutes a photographing optical system. Reference numeral 103 denotes a microlens array, 104 denotes an imaging device such as a CMOS, and 105 denotes an interface unit between the imaging device 104 and various signal processing units. 106 is an image signal processing means for processing an image pickup signal (pixel signal) from the image pickup device, 107 is a subject (object) detection means, 108 is a main memory such as a DRAM that temporarily stores data processed by each signal processing section. It is memory. Each functional block shares and uses the main memory 108 and transfers each signal processing data through the main memory 108. Reference numeral 109 denotes a memory controller which is a control unit of the main memory 108, and 110 denotes feature detection means. Pixels in an area detected by the subject detection means 107 from video data (shooting screen) temporarily stored in the memory 108 from the image signal processing means 106 Data is read to extract subject characteristics. For example, when the detection target is a human, a part in the face (ear, eye, nose, mouth, etc.) is extracted. Reference numeral 111 denotes a recognition / authentication unit that analyzes the feature of the subject detected by the feature detection unit 110 and identifies, for example, the presence or absence of a smile or an individual. The subject may be other than a person, and the feature detection unit 110 extracts features (color, shape, etc.) of the detected subject, and the recognition / authentication unit 111 performs subject recognition.

  An image data display / recording processing unit 112 reads image data stored in the main memory 108, records it on a recording medium (not shown), and displays an image on a display unit (not shown). A frame for indicating a subject in a captured image is displayed. A system control unit 113 controls the entire imaging apparatus, and controls each unit by loading and executing a program stored in a memory (not shown). This program also includes a control program for the imaging operation of the imaging apparatus of the present invention described below. Unless otherwise specified, the imaging apparatus of the present embodiment operates under the control of this program. Reference numeral 114 denotes an image sensor control unit (sensor control unit) that controls reading of pixel signals obtained by photoelectrically converting an optical image by the image sensor 104 such as a CMOS.

  The photographing optical system includes a zoom lens 101 and a focus lens 102, and collects light from a subject and enters the microlens array 103. The focus lens 102 projects light from each point of the subject onto one of the microlens arrays 103. That is, the imaging apparatus of the present embodiment is an imaging apparatus having the function of the light field camera shown in FIG. Note that the configurations of the microlens array 103 and the image sensor 104 in the light field camera are already well known, and thus the description thereof is omitted here.

  FIG. 2 is a diagram illustrating divided pixels of an imaging element included in the imaging apparatus having the function of the light field camera illustrated in FIG. 1 and an example of an image captured by the imaging element. A plurality of pixels in a two-dimensional array arranged in the image sensor 104 are divided in units of 2 × 2 (four divisions) division blocks corresponding to the microlens array 103, and an image pickup signal of the pixels included in the division block is a light field. Process as data. In this case, there are four types of light rays incident on one divided block of the image sensor 104, and four types (A to D) of images with different parallax are obtained. As a result, when the recorded image is reproduced, there are four possible depths of field, and the refocus range can be adjusted in four stages. Note that the divided blocks are not limited to 2 × 2 divisions, and may be, for example, 4 × 4 divisions (16 divisions) as described later, or division blocks including other predetermined number of division pixels. These are design choices for the image sensor 104 and the microlens array 103.

  FIG. 3 is a diagram illustrating a flowchart of the imaging operation of the imaging apparatus according to the present embodiment. This operation is performed by the system control unit 113 executing a control program.

  After the power is turned on (S301), the system control unit 113 detects the synchronization signal of the imaging system (S302) and starts the operation of each process (for example, detects the vertical synchronization signal and starts the imaging operation process). First, the image sensor control unit 114 reads out a pixel signal from the image sensor 104 (S03), and the image signal processing means 106 performs signal processing on the pixel signal (S312). In parallel with the image signal processing, the system control unit 113 determines ON / OFF of the subject detection function (S304). When the subject detection function is OFF, the synchronization of the imaging system is detected to start the next video frame. (S302).

  If the subject detection function is ON in step S304, the feature detection unit 110 detects a detection area where the subject in the shooting frame is detected based on the subject detection result, and obtains pixel data corresponding to the detection area from the memory 108. Read (S305). In the present embodiment, the detection area is a rectangular area, but is not limited to this.

  If the subject detection function is ON, the system control unit 113 further determines ON / OFF of the function for recognizing the detected subject state (S306). When the subject recognition function is ON, the feature detection unit extracts the feature of the subject from the pixel data of the rectangular area read from the memory 108 (S307), and performs authentication and recognition of the subject in the rectangular area (S308).

  Here, FIG. 5 shows an example of the detection area A where the subject detection means detects the subject. The system control unit 113 instructs the detection area A) where the subject has been authenticated and recognized to read out pixel signals of 4 × 4 divided (16 divided) pixels of the image sensor 104 shown in FIG. (S309), the image sensor control unit 114 is controlled (S311). The read pixel signal is converted into light field data by the image signal processing means 106.

  The read pixel signal is subjected to various signal processing by the image signal processing unit 106 (S312), and is written in the memory 108 as light field data (S313).

  For the region B other than the detection region A where the subject is authenticated and recognized, the adjacent pixels of the 4 × 4 divided pixels of the image sensor 104 are added to each other as shown in FIG. Read out from the image sensor 104 as pixel signals of divided pixels (four divisions). The system control unit 113 instructs the image sensor control unit 114 to do so (S309), and controls the image sensor control unit 114 (S311). The read pixel signal is subjected to various signal processing by the signal processing means 106 (S312), and is written in the memory 108 as light field data (S313).

  The display / recording processing unit 112 reads the image data written in the memory 108 at a desired timing (S314), records it on a recording medium (not shown), and displays the image on the display unit (not shown). (S315).

  When the recorded image is reproduced in this way, in the detection area A in which the subject is authenticated and recognized, in the present embodiment, there are 16 depths of field, and the refocus range has 16 levels. Can be adjusted.

  Further, in the area (area B) other than the detection area A where the subject is authenticated and recognized, in this embodiment, there are four depths of field, and the refocus range can be adjusted in four stages.

  As described above, in the detection region where the subject is authenticated and recognized, the signal of the divided pixel obtained by dividing one pixel of the image sensor 104 corresponding to the microlens into 4 × 4 (16 divisions) is read from the image sensor 104. . On the other hand, in an area other than the detection area where the subject is authenticated and recognized, adjacent pixels of the divided pixels obtained by dividing one pixel of the image sensor 104 corresponding to the micro lens by 4 × 4 (16 divisions) are added. Read out from the image sensor 104 as a pixel signal of 2 × 2 divided pixels (4 divided). Thus, when the recorded image is reproduced, the refocus range when the recorded image of the desired subject (the subject in the area A in the present embodiment) is reproduced is retained, and the recorded pixel data other than the subject is processed. Can be reduced. Therefore, the memory capacity for storing the pixel data can be reduced. The addition reading instruction is not limited to switching from 4 × 4 to 2 × 2, but may be switching from 1 × 1 (no division), for example, according to the reduction amount of pixel data. What is necessary is just to set suitably.

  FIG. 6 is a diagram illustrating a flowchart of the imaging operation of the imaging apparatus according to the present embodiment when a plurality of subjects are detected during shooting. This operation is also performed by the system control unit 113 executing a control program.

  In this embodiment, an example will be described in which there are two subjects as shown in FIG.

  After the power is turned on (S601), the system control unit 113 detects the synchronization signal of the imaging system (S602), and starts the operation of each process (for example, detects the vertical synchronization signal and starts the imaging processing operation). In the various processes of the imaging system, first, the image sensor control unit 114 reads out a pixel signal from the image sensor 104 (S603), and the image signal processing means 106 performs signal processing on the pixel signal (612). In parallel with the image signal processing, the system control unit 113 determines ON / OFF of the subject detection function (S604). When the subject detection function is OFF, the synchronization of the imaging system is detected to start the next video frame. It waits until it does (S602).

  If the subject detection function is ON in step S604, the feature detection unit 110 detects a detection region where the subject in the shooting frame is detected based on the subject detection result, and reads out pixel data of the detection region from the memory 108 ( S605). In the present embodiment, the detection area is a rectangular area, but is not limited to this.

  In the present embodiment, pixel data corresponding to the rectangular areas (A, B) in which two subjects in one photographing frame are detected is read from the memory 108 (S605).

  The feature detection unit 110 extracts the feature of the subject from the two read rectangular regions and performs authentication and recognition of the subject (S606). Based on the result of the authentication and recognition of the subject, the feature detection unit 110 evaluates the detection region ( For example, the reliability of recognition) is calculated (S607).

  Thereafter, the calculated evaluation values of the respective detection areas are compared (S608). Here, it is assumed that the evaluation value in the rectangular area A (for example, the recognition reliability Ath) is larger than the evaluation value in the rectangular area B (for example, the recognition reliability Bth). This suggests that the subject in the rectangular area A is more likely to be a desired subject.

  In this case, the system control unit 113 instructs the pixel signal in the rectangular area A to read the pixel signal of the 4 × 4 divided (16 divided) pixels of the image sensor 104 shown in FIG. 4A (S609). ), The image sensor control unit 114 is controlled (S611). The read pixel signal is subjected to various signal processing by the image signal processing means 106 (S612), and is written in the memory 108 as light field data (S613). Then, the display / recording processing unit 112 reads the pixel data written in the memory at a desired timing (S614), records the data on a recording medium (not shown), and displays the image on the display means (not shown). It performs (S615).

  For the pixel signal in the rectangular area B, as shown in FIG. 4B, adjacent pixels of 4 × 4 divided (16 divided) pixels of one pixel of the image sensor 104 are added to each other to obtain 2 × 2 divided ( It is read out from the image sensor 104 as a pixel signal of 4 divisions. The system control unit 113 instructs the image sensor control unit 114 to do so (S610), and controls the image sensor control unit 114 (S611). The read pixel signal is subjected to various signal processing by the image signal processing means 106 (S612), and is written in the memory 108 (S613). Then, the display / recording processing unit 112 reads the pixel data written in the memory 108 at a desired timing (S614), records it on a recording medium (not shown), and displays an image on the display unit (not shown). It performs (S615).

  Further, as shown in FIG. 4C, the areas other than the rectangular areas A and B are obtained by adding adjacent pixels of 4 × 4 divided (16 divided) pixels of one pixel of the image sensor 104 to 1 × 1. The data is divided (no division) and read from the image sensor 104 as light field data. The system control unit 113 controls the image sensor control unit 114 as described above. The read pixel signal is subjected to various signal processing by the image signal processing means 106 (S612), and is written in the memory 108 (S613). The display / recording processing unit 112 reads the pixel data written in the memory 108 at a desired timing (S614), records it on a recording medium (not shown), and displays an image on the display unit (not shown). (S615).

  As described above, when there are a plurality of detection areas in which the subject is authenticated and recognized, the pixel signal amount read from the image sensor 104 of the 4 × 4 (16-division) divided pixel of one pixel of the image sensor 104 is determined as Change according to the evaluation value of the detection result. For example, according to the evaluation value, adjacent pixels of 4 × 4 divided pixels are added and read out as a 2 × 2 divided pixel signal, or further, adjacent pixels are added to obtain 1 × 1 divided pixel (no division). Switches whether to read out as a pixel signal. As a result, the refocus range when reproducing the recorded image is maintained for the desired subject, and the pixel data amount is reduced for subjects other than the desired subject. As a result, the capacity of the memory for storing the pixel data can be reduced.

  Further, in the present embodiment, when a plurality of subjects are detected, instead of the determination based on the evaluation value, for example, the photographer looks at the photographed image from a display panel configured with a touch panel, etc. A desired subject may be selected from the inside. Also in this case, the pixel signal of the subject selected as described above is read out as a pixel signal of 4 × 4 divided pixels, and the region not selected is read out as a 2 × 2 divided pixel or 1 × 1 divided pixel.

  When a plurality of subjects are detected, the addition configuration may be switched according to the position of the detection area. For example, the detection area closest to the center in one frame image is read out as a pixel signal of 4 × 4 divided pixels. For other detection regions, adjacent pixels of the 4 × 4 divided pixels may be added and read as pixel signals of 2 × 2 divided pixels or 1 × 1 divided pixels (no division).

  In addition, for the detection regions having the same subject detection evaluation value, the addition configuration (reduction amount of pixel data) may be switched according to whether or not the position is closer to the center in one frame image.

  Next, a second embodiment of the present invention will be described using the flowchart of FIG.

  FIG. 8 is a diagram illustrating a flowchart of the photographing operation according to the present embodiment. This operation is also performed by the system control unit 113 executing a control program. Since the configuration of the image pickup apparatus is the same as that of the first embodiment, description thereof is omitted here.

  After the power is turned on (S801), the system control unit 113 detects a synchronization signal of the imaging system (S802) and starts each processing operation (for example, detects a vertical synchronization signal and starts the processing operation). First, a pixel signal is read from an image sensor such as a CMOS (S803), a frequency component of the read pixel signal for one frame is calculated (S804), and the calculated frequency component is compared with a threshold value (fth) (S805). ). According to the comparison result of the frequency components, for example, as shown in FIG. 9, the pixel signal for one frame is divided into two pixel signals of region A and region B. In this embodiment, the subject detection unit 107 has this function, but the image signal processing unit 106 may have this function.

  For a region where the calculated frequency component is larger than the threshold value (fth) (region B in FIG. 9, ie, the frequency component in the pixel data for one frame is high), one pixel of the image sensor 104 is divided into 4 × 4. Read out as pixel signals of pixels (16 divisions). The system control unit 113 instructs such reading (S806), and controls the image sensor control unit 114 (S808). The read pixel signal is subjected to various signal processing by the image signal processing means 106 (S809), and written in the memory 108 (S810).

  For a region where the calculated frequency component is smaller than the threshold value (fth) (region A in FIG. 9, for example, sky in a landscape), one pixel of the image sensor 104 is 4 × as shown in FIG. The adjacent pixels of the four divided pixels are added and read as 2 × 2 divided pixels (four divided). The system control unit 113 instructs such reading (S807), and controls the image sensor control unit 114 (S808). The read pixel signal is subjected to various signal processing by the image signal processing means 106 (S809), and written in the memory 108 (S810).

  The display / recording processing unit 112 reads out pixel data written in the memory 108 at a desired timing, performs recording on a recording medium (not shown), and displays an image on the display unit (not shown).

By recording pixel data according to the present embodiment, when an image is reproduced, there are 16 different depths of field in regions with high frequency components in pixel data for one frame, and the refocus range is 16 Can be adjusted in stages.
Also, in the pixel data for one frame, a region having a low frequency component is read out as a 2 × 2 divided pixel by adding pixel signals of 4 × 4 divided pixels of one pixel of the image sensor 104 to adjacent pixels. As a result, the amount of pixel data to be processed can be reduced and the memory capacity for storing the pixel data can be reduced in the region where the frequency component in the pixel data for one frame is low.

  Further, when calculating the frequency component in the pixel data for one frame, the horizontal frequency component and the vertical frequency component may be calculated, and the horizontal frequency component and the vertical frequency component may be compared. For example, when the frequency component in the vertical direction is high, as shown in FIG. 10A, in the 4 × 4 divided pixels of one pixel of the image sensor 104, two pixels are added in the horizontal direction (divided into two), and the vertical direction May be read without adding. That is, the frequency components in the horizontal and vertical directions may be compared, and the pixel addition amount may be increased in the direction where the frequency components are low. Further, when the frequency components in the horizontal and vertical directions are lower than the predetermined threshold value, the pixel addition amount may be increased for each component.

  As described above, according to the present invention, it is possible to provide an imaging apparatus capable of reducing the amount of light field data processed by the image signal processing means while maintaining the refocus range at the time of refocusing a recorded image as much as possible. It becomes possible.

[Modification]
Next, a modification of the above-described first or second embodiment of the present invention will be described.
The subject area detected in the first embodiment and the subject in the area divided in the second embodiment are light field data outside the area (peripheral area including the boundary of the area) in the refocus image processing. It may be composed of In this case, refocus image processing may be impossible depending on the amount of reduction when reading pixel data. In this modified example, in order to solve such a problem, in the region near the boundary of the subject region, instead of addition reading, the data of the divided pixels existing at the long base line position in the unit pixel cell are read from the memory 108. Refocusing is possible by reading.

  FIG. 13 is a diagram for explaining an area for reading data of divided pixels existing at a position having a long base line length. In the figure, with respect to the area where the subject is detected (black rectangular area in the figure), the gray portion of the dotted line frame is the peripheral area from which pixel data is read according to this modification. This peripheral area is set with a width X. The width X can be variably controlled as will be described later.

  Next, referring to FIG. 12, a configuration for reading pixel data from the set peripheral region will be described. In the peripheral area, for example, as shown in the figure, in the unit pixel cell existing under the microlens, the shaded pixel data (A1, A2) in the light field data divided into 4 × 4 (16 divisions), Alternatively, gray pixel data (B1, B2) is read from the memory 108. That is, the pixel data of each divided pixel of the divided pixel pair having the longest baseline length among the light field data divided into unit pixel cells is read from the memory 108.

  In the case of this modification, the amount of pixel data read from the memory 108 in the peripheral region is variable according to the reliability of subject detection in the case of the modification of the first embodiment. This is because when the pixel data existing at the pixel position having a long base line length at the boundary between the subject region and the outside of the region is read from the memory 108, the width X of the peripheral region is made variable to control the range of reading the pixel data. It is. For example, when the object detection reliability is low, the distance X is increased, and when the object detection reliability is high, the distance X is decreased and light field data around the rectangular area used for refocusing is controlled. It should be noted that the peripheral region setting function with reference to the reliability may be provided to the subject detection means 107 or the image signal processing means 106. In the case of the modification of the second embodiment, the horizontal frequency component is compared with the vertical frequency component according to the comparison of the horizontal frequency component and the vertical frequency component in the same manner as the pixel data reduction amount control described with reference to FIG. Control may be performed so that the size of the peripheral region is variably determined in the direction perpendicular to the direction.

  As described above, according to this modification, near the boundary between the subject region and the outside of the subject region, the pixel data existing at the pixel position having the long base line length in the divided block is read from the memory 108, thereby The possibility of refocusing on the subject can be further increased.

  In the above-described embodiments and modifications of the present invention, the control of the entire apparatus performed by the system control unit 113 may be performed by one piece of hardware, or by a plurality of pieces of hardware sharing the processing. May be.

  Each means constituting the imaging apparatus and each step of the recording method in the embodiment of the present invention described above can be realized by operating a program stored in a RAM or ROM of a computer. This program and a computer-readable storage medium storing the program are included in the present invention.

  In addition, the present invention can be implemented as, for example, a system, apparatus, method, program, storage medium, or the like. Specifically, the present invention may be applied to a system including a plurality of devices. The present invention may be applied to an apparatus composed of a single device.

  In the present invention, a software program (in the embodiment, a program corresponding to the flowcharts shown in FIGS. 3, 6 and 9) for realizing the functions of the above-described embodiments is supplied directly or remotely to a system or apparatus. Including. This includes the case where the system or the computer of the apparatus is also achieved by reading and executing the supplied program code.

  Accordingly, since the functions of the present invention are implemented by computer, the program code installed in the computer also implements the present invention. In other words, the present invention includes a computer program itself for realizing the functional processing of the present invention. In that case, as long as it has the function of a program, it may be in the form of object code, a program executed by an interpreter, script data supplied to the OS, and the like.

  Examples of the storage medium for supplying the program include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, there are MO, CD-ROM, CD-R, CD-RW, magnetic tape, nonvolatile memory card, ROM, DVD (DVD-ROM, DVD-R) and the like.

  As another program supply method, there is a method of connecting to a homepage on the Internet using a browser of a client computer. It can also be supplied by downloading the computer program itself of the present invention or a compressed file including an automatic installation function from a homepage to a storage medium such as a hard disk.

  It can also be realized by dividing the program code constituting the program of the present invention into a plurality of files and downloading each file from a different homepage. That is, a WWW server that allows a plurality of users to download a program file for realizing the functional processing of the present invention on a computer is also included in the present invention.

  As another method, the program of the present invention is encrypted, stored in a storage medium such as a CD-ROM, distributed to users, and encrypted from a homepage via the Internet to users who have cleared predetermined conditions. Download the key information to be solved. It is also possible to execute the encrypted program by using the key information and install the program on a computer.

  Further, the functions of the above-described embodiments are realized by the computer executing the read program. Furthermore, based on the instructions of the program, an OS or the like running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments can be realized by the processing.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

Claims (22)

A micro that includes an imaging unit having a two-dimensional array of a plurality of pixels, each of which converts an optical image into a pixel signal by photoelectric conversion, and a plurality of microlenses arranged between the focus lens, the imaging unit, and the focus lens In an imaging apparatus having a photographing optical system having a lens array, each microlens corresponding to one divided block when the two-dimensional array of pixels is divided into blocks each including a predetermined number of pixels,
Reading means for reading out the pixel signals from the plurality of pixels;
Subject detection means for detecting a subject included in the shooting screen based on the pixel signal read by the reading means;
Based on the detection result of the subject by the subject detection means, the reading means is controlled, and the reduction amount of pixel signals read from the divided blocks corresponding to the detected subject detection area on the photographing screen and other than the detection area An image pickup apparatus comprising control means for controlling a reduction amount of a pixel signal read from a divided block corresponding to the region.   2. The control unit according to claim 1, wherein a reduction amount of the pixel signal is increased in a divided block corresponding to an area other than the detection area than in a divided block corresponding to the detected area of the subject. The imaging device described in 1.   When the subject detection means detects a plurality of subjects, it calculates the reliability of detection of each detected subject, and the control means controls the readout means to divide the divided block corresponding to the detection area of each subject. The imaging apparatus according to claim 1, wherein a reduction amount of a pixel signal read from the image is controlled in accordance with the reliability.   4. The control unit according to claim 3, wherein the control unit controls the reading unit to increase the reduction amount of the pixel signal in the detection area of the subject having the low reliability than the detection area of the subject having the high reliability. The imaging device described in 1.   When there is a detection region having the same calculated reliability, the control unit controls the reading unit, and the reduction amount of the pixel signal read from the divided block corresponding to each detection region having the same reliability, The imaging apparatus according to claim 3 or 4, wherein the reliability is controlled in accordance with a position of each detection region.   When the subject detection unit detects a plurality of subjects, the control unit controls the readout unit, and determines the reduction amount of the pixel signal read from the divided block corresponding to the detection region of each subject. The imaging device according to claim 1, wherein the imaging device is controlled in accordance with.   The image pickup apparatus according to claim 6, wherein the control unit controls the reading unit to increase the reduction amount of the pixel signal as the position of the detection region is farther from the center of the shooting screen.   The control unit includes a selection unit that selects a detection area of the subject detected by the subject detection unit, and the control unit controls the readout unit to reduce the amount of pixel signal read from the divided block corresponding to the detection region of each subject, The imaging apparatus according to claim 1, wherein control is performed according to whether the detection area is selected by the selection unit.   The control means controls the reading means to read out the reduction amount of the pixel signal read from the divided block corresponding to the detection area selected by the selection means from the divided block corresponding to the detection area not selected. The image pickup apparatus according to claim 8, wherein the image pickup device is smaller than the reduction amount of the pixel signal.   The control unit controls the readout unit to set a peripheral region including the boundary in the vicinity of the boundary of the detection region, and to output a pixel signal read from the divided block corresponding to the peripheral region in the divided block. The imaging apparatus according to any one of claims 1 to 9, wherein the imaging device is determined according to a baseline length given by the pixel pair.   The imaging apparatus according to claim 10, wherein the control unit controls the reading unit to determine the size of the peripheral region according to the reliability.   Based on the frequency component of the pixel signal, the image capturing screen includes a dividing unit that divides the photographing screen into a plurality of regions corresponding to different frequency components, and the control unit controls the reading unit and corresponds to each of the divided regions. The imaging apparatus according to claim 1, wherein a reduction amount of pixel signals read from the divided blocks to be controlled is controlled according to the different frequency components.   The dividing means calculates frequency components in different directions in the divided areas, and the control means controls the reading means to reduce pixel signals read from the divided blocks corresponding to the divided areas. The imaging apparatus according to claim 12, wherein the amount is controlled according to a direction in which a high frequency component is calculated.   The control unit controls the reading unit to set a peripheral region including the boundary in the vicinity of the boundary of the region, and to output a pixel signal read from the divided block corresponding to the peripheral region to a pixel in the divided block The imaging apparatus according to claim 12, wherein the imaging device is determined according to a baseline length given by the pair.   15. The imaging apparatus according to claim 14, wherein the control unit controls the reading unit to determine the size of the peripheral area according to the frequency component in the different direction.   The control means controls the reading means to add and read pixel signals of a predetermined number of pixels included in the divided block, and controls the reduction amount of the pixel signals by changing the number of pixels to be added. The imaging device according to any one of claims 1 to 15, wherein   Image signal processing means for obtaining light field data by processing the pixel signal read by the reading means under control of the reduction amount by the control means, and recording means for recording the light field data on a recording medium The imaging apparatus according to claim 1, further comprising: A micro that includes an imaging unit having a two-dimensional array of a plurality of pixels, each of which converts an optical image into a pixel signal by photoelectric conversion, and a plurality of microlenses arranged between the focus lens, the imaging unit, and the focus lens An imaging device control method having a photographing optical system having a lens array, and each microlens corresponding to one divided block when the two-dimensional array of pixels is divided into blocks each including a predetermined number of pixels In
A reading step of reading out the pixel signal from the plurality of pixels;
A subject detection step of detecting a subject included in the shooting screen based on the pixel signal read in the readout step;
Based on the detection result of the subject in the subject detection step, the readout step is controlled, and the reduction amount of the pixel signal read from the divided block corresponding to the detection region of the detected subject on the photographing screen and the detection region A control method comprising a control step of controlling a reduction amount of a pixel signal read from a divided block corresponding to a region other than. A micro that includes an imaging unit having a two-dimensional array of a plurality of pixels, each of which converts an optical image into a pixel signal by photoelectric conversion, and a plurality of microlenses arranged between the focus lens, the imaging unit, and the focus lens A photographing optical system having a lens array, and a reading unit that reads out the pixel signal from the plurality of pixels, and each microlens divides the two-dimensional array of pixels into blocks each including a predetermined number of pixels. It is a program for controlling an imaging device corresponding to one divided block,
Computer
Subject detection means for detecting a subject included in the shooting screen based on the pixel signal read by the reading means;
Based on the detection result of the subject by the subject detection means, the reading means is controlled, and the reduction amount of pixel signals read from the divided blocks corresponding to the detected subject detection area on the photographing screen and other than the detection area A program that functions as a control unit that controls a reduction amount of a pixel signal read from each divided block corresponding to the area.   A computer-readable storage medium storing the program according to claim 19.   A program that causes a computer to function as each unit of the imaging device according to any one of claims 1 to 17.   A storage medium storing a program that causes a computer to function as each unit of the imaging apparatus according to any one of claims 1 to 17.