JP2016025511A - Image pickup device, its control method, computer program, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress the increase in electric power consumed by communication between an image pickup device and an image processing engine.SOLUTION: An image pickup device comprises: image pickup means for reading pixel signals from pixels arrayed in two dimensions in units of array lines; signal processing means for performing a signal processing on pixel signals read out from the image pickup means; and output means for outputting pixel signals output from the image pickup means and the signal processing means. The image pickup means, the signal processing means and the output means follow control signals from the outside; in a first processing mode, the output means outputs pixel signals from the image pickup means; and in a second processing mode, the signal processing means performs the signal processing on at least part of pixel signals read out from the image pickup means, and the output means outputs the pixel signals at least partially subjected to the signal processing. The image pickup means, the signal processing means and the output means are formed on a common semiconductor chip.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image pickup device, a control method thereof, and an image pickup apparatus, and more particularly to an image pickup device and an image pickup apparatus that have technical effects in improving a frame rate and reducing power consumption in image signal processing during shooting.

Patent Document 1 describes a technique for obtaining a control value earlier than in the prior art by individually providing a pixel for obtaining a control value for each pixel of an image sensor and a pixel used for producing a display image. . Specifically, an image sensor having the following configurations (1) and (2) has been proposed.
(1) A control information generation unit that generates control information used when imaging a first image signal corresponding to a voltage signal obtained from the first pixel group.
(2) Output means for outputting a second image signal corresponding to a voltage signal obtained from the second pixel group as an image display signal for image display.

Japanese Patent Application No. 2012-288036

The image sensor described in Patent Document 1 is intended to increase the speed of focusing processing, and has a mechanism for directly outputting a control signal from the image sensor to the control circuit of the actuator. On the other hand, the image display signal converts RGB data into image data for display via an image processing engine.
However, with high frame rate driving including focusing processing, it is necessary to transmit a large amount of data when transmitting signals from the image sensor to the image processing engine, and there is concern about the effect on power consumption.

  The present invention has been made paying attention to such a problem, and an object of the present invention is to provide an imaging apparatus capable of selecting whether recording is performed with emphasis on recording image quality or frame rate in accordance with the application. .

  In order to achieve the above object, the imaging device of the present invention includes an imaging unit that reads out a pixel signal in units of rows of an array from a plurality of pixels that are two-dimensionally arranged, and a pixel signal that is read out from the imaging unit. Signal processing means for performing signal processing, an output means for outputting a pixel signal output from the imaging means and the signal processing means, and the imaging means, the signal processing means and the output means, according to a control signal from the outside, In the processing mode, the output means outputs the pixel signal from the imaging means, and in the second processing mode, the signal processing means performs signal processing on at least a part of the pixel signal read from the imaging means. The output means outputs a pixel signal that has been subjected to signal processing at least in part, and the imaging means, signal processing means, and output means are formed on the same semiconductor chip.

  In addition, the imaging device of the present invention performs first signal processing on an imaging unit that reads out a pixel signal in units of rows of an array from a plurality of pixels arranged in two dimensions, and a pixel signal read out from the imaging unit. A first signal processing unit; a second signal processing unit that performs a second signal processing on the pixel signal output from at least one of the imaging unit and the first signal processing unit; and the imaging unit reads out the pixel signal. Correspondingly, a control means for controlling the first signal processing means and the second signal processing means is provided, and the control means has a plurality of processing modes of pixel signals read from the imaging means, and a plurality of processing The mode includes a first processing mode in which the second signal processing means receives a pixel signal from the imaging means and performs second signal processing, and a pixel signal read out from the imaging means by the first signal processing means. At least part of the first signal processing The second signal processing means includes a second processing mode in which the pixel signal is received from the first processing means and the second signal processing is performed. The imaging means, the signal processing means, and the output means are the same semiconductor chip. Formed on top.

  According to the present invention, it is possible to provide an imaging apparatus capable of taking a picture with priority given to either the image quality or the frame rate in consideration of power consumption.

It is a figure which shows the structure of the imaging device concerning 1st Example of this invention. It is a figure which shows the pixel arrangement | sequence of the image pick-up element used with the imaging device concerning the 1st Example of this invention. It is a figure which shows the timing chart of imaging | photography operation | movement of the imaging device concerning 1st Example of this invention. It is a figure which shows the flowchart of the imaging | photography process operation | movement of the imaging device concerning 1st Example of this invention. It is a figure which shows the flowchart of the imaging | photography process operation | movement of the imaging device concerning 1st Example of this invention. It is a figure for demonstrating the outline | summary of the signal processing performed with the imaging device concerning 1st Example of this invention. It is a figure which shows the timing chart of imaging | photography operation | movement of the imaging device concerning 2nd Example of this invention. It is a figure which shows the flowchart of imaging | photography process operation | movement of the imaging device concerning 2nd Example of this invention. It is a figure which shows the flowchart of imaging | photography process operation | movement of the imaging device concerning 2nd Example of this invention. It is a figure which shows the flowchart of imaging | photography process operation | movement of the imaging device concerning 2nd Example of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

  FIG. 1 is a block diagram showing a configuration of a main part of an image pickup apparatus having an image pickup element according to the first embodiment of the present invention. The imaging apparatus includes an optical system 101, an imaging element unit 102, a drive circuit unit 103, a signal processing unit A104, an imaging unit 105, a signal processing unit B106, a memory unit 107, an image display unit 108, an image recording unit 109, and an operation unit 110. The control unit 111 is configured. The function of each unit of the imaging apparatus according to the present embodiment is realized by the CPU of the control unit 111 executing and controlling a control program stored in a memory (not shown). In this case, the control unit may be configured as a single unit that controls the entire imaging apparatus including the illustrated units, or may be configured by a plurality of CPUs according to the functions of the imaging apparatus.

  The optical system 101 includes a focusing lens that forms a subject image on the image sensor unit 102, a zoom lens that performs optical zoom, a diaphragm that adjusts the brightness of the subject image, and a shutter that controls exposure. It is driven by 103.

  The image sensor unit 102 includes a plurality of pixels arranged two-dimensionally in the horizontal and vertical directions, and a circuit that outputs image signals read from these pixels in a predetermined order in units of rows. In the present embodiment, as shown in FIG. 2, the imaging signals of the pixel rows are selectively output as the main stream and the substream, and the output imaging signals are RAW data.

  The drive circuit unit 103 operates the optical system 101 and the image sensor unit 102 in accordance with a control signal from the control unit 111. The signal processing unit A104 is a signal processing block that performs signal processing on a part of the pixel signal output from the image sensor unit 102, and outputs the signal with a data amount reduced from that of the input pixel signal. Therefore, by improving the data rate between the imaging unit 105 and the signal processing unit B106, the frame rate at which the imaging apparatus can perform image processing is improved.

  The imaging unit 105 is a device that includes the imaging element unit 102 and the signal processing unit A104. A pixel signal converted into a synchronization signal and a differential signal is transmitted from the imaging unit 105 to the signal processing unit B106.

  The signal processing unit B106 is controlled by the control signal from the control unit 111, and performs appropriate signal processing on the output signal converted into the digital signal sent from the imaging unit 105 to convert it into image data. Further, output signals and image data converted into digital signals are output to the memory unit 107 and the image recording unit 109, or signal processing is performed by receiving image data from the memory unit 107 and the image recording unit 109. Further, it has a function of detecting photometric data such as an in-focus state and an exposure amount from a signal from the image sensor unit 102 and transmitting it to the control unit 111.

  The memory unit 107 is temporarily controlled by the control signal from the control unit 111, and temporarily stores the output signal of the image sensor unit 102 converted into a digital signal and the signal processed image data. Further, the image data for display is output to the image display unit 108. The image display unit 108 is controlled by a control signal from the control unit 111, and displays the display image data stored in the memory unit 107 in order to determine the composition before shooting and to check the image after shooting. It consists of a viewfinder (EVF) and a liquid crystal display (LCD).

  The image recording unit 109 includes a detachable memory and the like. The image recording unit 109 is controlled by a control signal from the control unit 111 and can record and detach an output signal and image data converted into a digital signal sent from the signal processing unit B106. Reading from a simple memory.

  The operation unit 110 transmits an external instruction using an operation member such as a switch or a push button to the control unit 111, and a display member such as an LCD or a photodiode or an image display unit 108 according to a control signal from the control unit 111. Is used to display the state of the imaging device. The instruction from the outside is, for example, the state of the power switch of the imaging device, the instruction for displaying an image before photographing, various instructions for photographing, the display of the photographed image, or the menu operation for instructing the operation of the imaging device in advance. . Further, an on-screen operation may be performed using the image display unit 108 as a display member and a touch panel attached to the image display unit 108 as an operation member.

  The control unit 111 controls the entire imaging apparatus according to an instruction from the operation unit 110. Further, the optical system 101 is controlled in accordance with photometric data such as the in-focus state and exposure amount sent from the signal processing unit B106, and an optimal subject image is formed on the image sensor unit 102.

  Next, the pixel arrangement of the image sensor unit 102 used in the present imaging device will be described. The pixels of the image sensor unit 102 are divided into two pixels, a main stream and a substream.

  FIG. 2 shows the pixel arrangement of the main stream and the substream. Here, consider a case where four color filters are arranged in a Bayer array. Here, as an example, a white portion is a main stream and a gray portion is a substream. As shown in FIG. 2, the main stream has more pixels than the substream.

  Next, the use of each pixel will be described.

  First, the main stream will be described. The main stream pixel signals are used for recording and display. When the image quality is important, the main stream pixel signal is transmitted to the signal processing unit B106. When importance is attached to the frame rate, the pixel signal of the main stream is subjected to predetermined processing by the signal processing unit A104 to reduce the data amount of the pixel signal and then transmitted to the signal processing unit B106.

  Next, substreams will be described. The sub-stream pixel signal is used for both image processing parameter creation and recording. When importance is attached to the image quality, the substream pixel signal is transmitted to the signal processing unit B106 as in the main stream. When importance is attached to the frame rate, the pixel signal of the substream is transmitted to the signal processing unit A104 as in the main stream, but the processing is not performed there and is transmitted to the signal processing unit B106 as it is. The substream pixel signal transmitted to the signal processing unit B106 is input when the signal processing unit B106 performs image processing on the main stream pixel signal subjected to predetermined signal processing in the signal processing unit A104. Used as a parameter calculation signal.

  Next, the operation of the imaging apparatus according to the present embodiment will be described. The imaging apparatus has a still image shooting mode and a moving image shooting mode, and in each shooting mode, a processing mode that is an image quality priority mode and a frame rate priority mode can be set. The setting of each mode is performed by the control unit 111 according to the operation of the operation unit 110, and various functions are realized by the control unit 111 controlling each unit of the imaging apparatus according to the information of the set mode.

First, the operation in the still image shooting mode will be described. This operation is achieved by the control unit 111 controlling each unit of the imaging apparatus according to the information of the still image shooting mode set according to the operation of the operation unit 110 to realize the following operation.
(1) In response to an instruction from the photographing switch of the operation unit 110, the still image photographing operation is started.
(2) The signal processing unit B106 detects photometric data from the signal from the imaging unit 105, and transmits the result to the control unit 111.
(3) The control unit 111 controls the optical system 101 based on the photometric data.
(4) The image sensor unit 102 performs exposure for recording a still image and outputs an image signal (pixel signal).
(5) The signal processing unit B106 converts the pixel signal from the image sensor unit 102 into image data for recording and sends it to the image recording unit 109, and also converts it into image data for display and converts it into the image display unit 108. send. As a result, the image recording unit 109 records the image data in a removable memory, and the image display unit 108 displays the image data.
(6) Return to display image control.

  Next, the operation of the imaging apparatus when the image quality priority mode and the frame rate priority mode according to the present embodiment are applied to moving image shooting will be described.

  FIG. 3 is a diagram illustrating a timing chart of the operation of the imaging unit 105. 3A) is an image quality priority mode, b) is a frame rate priority mode under the frame rate in a), and c) is a timing chart of a frame rate priority mode for optimizing the frame rate. In the figure, VD is a vertical synchronization signal input from the drive circuit unit 103 to the imaging unit 105 for each frame of an image, and Sensor Read is required from reading of the pixel signal of the imaging element unit 102 to generation of digital data. This is the time for one frame. Further, Output Data is a period in which the pixel signal that has become digital data is transmitted to the signal processing unit B106, and Signal Processor is a period in which signal processing is performed on the pixel signal that has become digital data.

  First, the image quality priority mode will be described. The image quality priority mode is a mode in which processing is performed with emphasis on the image quality of an image used for recording / display. Specifically, the captured image data is transmitted from the imaging unit 105 to the signal processing unit B106 by LVDS or the like without being processed in the sensor. In FIG. 3a), each block performs signal processing of one frame in synchronization with the rise of VD (t01). The time Sensor Read for one frame required from the reading of the pixel signal of the image sensor unit 102 to the generation of digital data is t01 to t04. The period Output Data during which the pixel signal (RAW data) that has become digital data is transmitted to the signal processing unit B106 is t03 to t05. The timing of the vertical synchronization signal VD of the next frame is t06, and the period t01 to t06 is one frame period. In this mode, since the pixel signal from the image sensor unit 102 is sent to the signal processing unit B106 without reducing the data amount, there is no deterioration in image quality due to the data amount reduction.

  Next, the frame rate priority mode will be described. The frame rate priority mode is a mode for performing processing with an emphasis on the frame rate of an image used for recording / display. Specifically, the captured image data is subjected to image processing by the signal processing unit A104 built in the sensor, the amount of image data per frame is reduced, and transmitted to the signal processing unit B106. By this processing, more frame data can be transmitted per unit time to the signal processing unit B106 than in the image quality priority mode, and the frame rate can be improved.

  Thus, the reason why the signal processing unit A104 is incorporated in the imaging unit 105 on the same chip as the imaging device unit 102 in this embodiment is for power saving, high-speed processing, and low-cost design. The imaging unit 105 and the signal processing unit AB (104, 106) or the control unit 106 are often arranged on separate substrates (chips), and there are many resistance components and capacitance components of wiring for communication between chips. Become. For this reason, driving power must be increased, for example, it is slower than communication using wiring within the same chip, or in order to send a high-speed signal, it is necessary to drive with an amplifier to maintain signal waveform quality.

  On the other hand, in this embodiment, since the signal processing unit A104 is formed on the same semiconductor chip, the output wiring for the image data can be shortened, and the arrangement of the amplifier can be omitted. In addition, since the amount of image data per frame in the frame rate priority mode is small, the communication time between the image sensor unit 102 and the control unit 106 is shortened, so that power consumption can be reduced.

  The frame rate priority mode in FIG. 3B shows a timing chart of four items. A different part from FIG. 3 a) is demonstrated. In this frame rate priority mode, digital data read from the imaging unit during the Sensor Read period is output to the signal processing unit A104. The pixel signal for one frame output from the image sensor section tends to reduce the amount of data during image processing in the process of becoming a final image such as JPG. Therefore, the signal processing unit A104 performs predetermined image processing to reduce the data amount. In the present embodiment, predetermined signal processing is performed on the main stream pixel signals during the period from t10 to t13, and the signal processing unit A104 outputs the substream pixel signals as they are without performing any processing.

  The period Output Data during which the pixel signal which is digital data is transmitted from the signal processing unit A104 to the signal processing unit B106 is t12 to t15. In this embodiment, periods t12 to t14 in which the signal-processed main stream pixel signals are output from the signal processing unit A104 and periods t14 to t15 in which the sub-stream pixel signals are output from the signal processing unit A104 are provided separately. It has been. In this mode, since the data amount of the pixel signal of the main stream is reduced by the signal processing unit A104, the transmission period Output Data to the signal processing unit B106 is shorter than a). As a result, the margin period from data transmission to the next frame was t05 to t06 in a), but in t) to t15 to t16 in b). By shortening the margin period in b) to the same level as a), It is possible to increase the frame rate. The result is shown in FIG. The period required for one frame in b) was reduced from t08 to t16 to t17 to t25 in c). As a result, the period of one frame became t17 to t25, and in the case of a) and b) Short compared to.

  Next, processing operations in the image quality priority mode and the frame rate priority mode shown in FIG. 3 will be described with reference to the flowchart of FIG. 4A shows a flowchart of the processing operation in the image quality priority mode, and FIG. 4B shows a flowchart of the processing operation in the frame rate priority mode.

  First, the processing operation in the image quality priority mode will be described with reference to FIG. In the image quality priority mode, signal processing of the main stream and substream is performed only by the signal processing unit B106. 4A, in S401, the image sensor unit 102 acquires pixel signals, and in S402, the acquired main stream and substream pixel signals are transmitted from the imaging unit 105 to the signal processing unit B106. In S403, the signal processing unit B106 receives the main stream and sub stream pixel signals.

  In S404, image processing parameters are acquired from the main stream and substream pixel signals received by the signal processing unit B106, and then in S405, images of the main stream and substream pixel signals based on the image processing parameters obtained in S404. Process. In step S406, the main stream and sub stream pixel signals are converted into a format suitable for display and recording. This makes it possible to perform shooting with priority on image quality in moving image shooting.

  Next, the processing operation in the frame rate priority mode will be described with reference to the flowchart of FIG. In the frame rate priority mode, the pixel signals of the main stream and the substream are transmitted from the imaging element unit 102 to the signal processing unit A104.

  In S407, a pixel signal is acquired from the image sensor unit 102, and in S408, signal processing of the main stream pixel signal acquired by the signal processing unit A104 is performed. In step S409, the signal processing unit A104 transmits the signal-processed main stream pixel signal to the signal processing unit B106.

  In S410, when the signal processing unit A104 transmits the substream pixel signal to the signal processing unit B106, in S411, the signal processing unit B106 receives the main stream pixel signal transmitted from the signal processing unit A104. In S412, the signal processing unit B106 receives the pixel signal of the substream sent from the signal processing unit A104.

  In step S413, the signal processing unit B106 obtains an image processing parameter from the received pixel signal of the substream. Here, the parameters required for the pixel signal, such as the white balance adjustment coefficient and the strength of noise reduction, are obtained. In S414, the signal processing unit B106 performs image processing of the main stream image based on the image processing parameters obtained in S413. Thereafter, in S415, the signal processing unit B106 converts the processed image signal into a format corresponding to display or recording. This makes it possible to perform shooting with priority on the frame rate in moving image shooting.

  Next, a predetermined process performed by the signal processing unit A104 on the pixel signal of the main stream in the frame rate priority mode of the embodiment will be described. Here, a processing operation for reducing the amount of transfer data by performing YUV conversion signal processing on the R, Gr, Gb, and B pixel signals output from the image sensor unit 102 will be described with reference to FIG.

FIG. 5 a) is a diagram schematically illustrating pixel signals arranged corresponding to the pixel arrangement of the image sensor unit 102. Here, if R, Gr, Gb, and B constituting 2 × 2 are considered as one group of the Bayer array, in FIG. 5 a), since each pixel has an 8-bit pixel signal, one group is expressed in 32 bits. Is done. The configuration of the pixel signal obtained by converting this according to the YUV422 format is shown in FIG. The conversion formula in this case is shown below. Gr and Gb are substituted for G.
Y = 0.299 × R + 0.587 × G + 0.114 × B
Cb = -0.172 × R -0.339 × G + 0.511 × B
Cr = 0.511 × R -0.428 × G -0.083 × B
Here, Y is a luminance signal, and Cr and Cb are color difference signals.

  Although the pixel signal format in FIG. 5b) is different from that in a), since one group Y, Y, Cr, and Cb in the Bayer array after conversion is 8 bits each, one group has a data amount of 32 bits. There is no change in the amount of data.

  The configuration of the pixel signal when converted according to the YUV411 format is shown in FIG. Here, Cr and Cb span two groups, and 48 bits in two groups. Cr and Cb across two groups are obtained from the average of the two groups. By doing so, it becomes 24 bits per group, and 1/4 of the data amount per group is reduced compared with the R, Gr, Gb, and B formats of FIG.

  As described above, in the present embodiment, the processing performed by the signal processing unit A104 on the main stream in the frame rate priority mode is the conversion processing to the YUV411 format, but other formats such as the YUV420 format may be used. Also, the conversion process may be selected according to the frame rate value required for the optimization of the frame rate.

  As described above, according to the first embodiment, it is possible to acquire an image in which the items selected for the image quality and the frame rate are emphasized in the moving image mode. Specifically, when priority is given to image quality, both the main stream image and the substream image are acquired by the image processing unit B106. On the other hand, when priority is given to the frame rate, it is possible to improve the frame rate without changing the data rate by reducing the data amount of the main stream signal and transmitting the pixel signal to the signal processing unit B106. is there.

  In the present embodiment, the image quality priority mode and the frame rate priority mode have been described taking moving image shooting as an example, but the present invention can be similarly applied to still image shooting. In still image shooting, the effect of improving the frame rate is small, but the effect of reducing power consumption can be expected.

  Next, a second embodiment of the present invention will be described. This embodiment is different from the first embodiment in that a three-stage mode that balances image quality priority, frame rate priority, and image quality and frame rate can be set. Since the configuration of the imaging apparatus is the same as that described with reference to FIGS. 1 and 2 in the first embodiment, description thereof is omitted here. In the following description, when the frame rate is prioritized, a mode in which the signal processing unit A104 performs signal processing of only the main stream pixel signals is referred to as a “balance” mode. A mode in which the signal processing unit A104 performs signal processing of both main stream and substream pixel signals, which is characterized by the present embodiment, is “frame rate priority mode” or “frame rate priority mode with optimized frame rate”. Called.

  An image quality priority mode, a balance mode, and a frame rate priority mode according to the present embodiment will be described with reference to an operation timing chart of the imaging unit 105 shown in FIG. In the figure, a) is an image quality priority mode, b) is a balance mode in the case of the frame rate in a), and c) is a balance mode timing chart for optimizing the frame rate. Further, d) is a frame rate priority mode in the case of the frame rate in a), and e) is a timing chart of the frame rate priority mode for optimizing the frame rate. In FIG. 6 as well, VD, Sensor Read, Signal Processor, and Output Data indicate the same period as in the first embodiment.

  First, since FIGS. 6a) to 6c are the same as FIGS. 3a) to 3c) of the first embodiment, description thereof will be omitted. However, for the modes of FIGS. 6a) to 6c), different names from those in the first embodiment are used.

  Next, the frame rate priority mode in FIGS. 6d) and e) will be described. In this mode, digital data read from the image sensor unit 102 during the Sensor Read period is output to the signal processing unit A104 and processed. The difference between FIGS. 6d) and e) is that the frame rate is optimized in FIG. 6e), and t39 to t45 in FIG. 6e) are the same as t29 to t35 in FIG. 6d).

  The signal processing unit A104 performs signal processing on the pixel signal that has become digital data. In the balance mode of FIG. 6b), the signal processing unit A104 does not perform processing on the pixel signal of the substream, but in the frame rate priority mode of d), the signal processing unit A104 also performs signal processing on the pixel signal of the substream. I do. In this case, the period Output Data during which the pixel signal that has become digital data is transmitted to the signal processing unit B106 is t33 to t35. In transmission to the signal processing unit B106, pixel signals are output separately for the main stream and the substream in the balanced mode of FIG. 6B) (serial communication). On the other hand, in the frame rate priority mode d), the main stream and the substream are handled in the same way (parallel communication).

  As described above, the transfer period of the pixel signal from the imaging unit 105 to the signal processing unit B106 is shorter in the balance mode period (t12 to t15) than in the image quality priority mode period (t03 to t05). This period (t33 to t35) is further shortened.

  In addition, the period of a series of processing performed in one frame is also shorter in the balance mode period (t08 to t15) than in the image quality priority mode period (t01 to t05), and the frame rate priority mode period (t29 to t35) is It becomes even shorter.

Next, the processing operation of the imaging apparatus in each mode shown in FIG. 6 will be described using the flowcharts shown in FIGS. 7a is an image quality priority mode according to the timing chart of FIG. 6a), FIG. 7c is a balance mode according to the timing chart of FIG. 6b) or c), and FIG. 7c is an operation flow of the frame rate priority mode according to the timing chart of FIG. It is. Since FIGS. 7a and 7b are the same flowcharts as FIGS. 4a and 4b, description thereof is omitted here. 7C includes a processing operation in which the signal processing unit A104 performs signal processing on the main stream and substream pixel signals and transmits the processed signal to the image processing unit B106. According to this processing operation, although the image quality is obtained by a simple signal processing system in the image sensor, the image quality is given priority, and compared with the case where the external signal processing system pixel signal data is transmitted from the imaging unit 105. The amount of data to be reduced can be reduced.
In the present embodiment, an example of a specific processing operation in FIG. 7c will be described.

  In S716, an image is taken by the control of the control unit 111 according to the operation of the operation unit 110. In S717, the signal processing unit A104 acquires the pixel signals of the main stream and the substream from the imaging unit 105 and performs signal processing thereof. At this time, the pixel signal after the signal processing is in a format such as JPG that can be reproduced and displayed by the imaging apparatus.

  Next, in S718, the signal processing unit A104 transmits the processed pixel signal to the signal processing unit B106, and in S719, the signal processing unit B106 receives the transmitted pixel signal. In S720, the pixel signal received by the signal processing unit B106 is processed to generate a pixel signal corresponding to the display and recording format.

  According to the second embodiment of the present invention described above, it is possible to perform shooting with priority on the selected item with respect to image quality and frame rate regardless of still images and moving images.

  Specifically, when priority is given to image quality, high-quality image processing can be performed by performing signal processing on the pixel signal of the imaging element unit by the signal processing unit B106 outside the imaging unit 105.

  When the image quality and the frame rate are balanced, the main stream pixel signal having a large amount of data is image-processed by the signal processing unit A104 to reduce the data amount, and the pixel signal is sent from the imaging unit 105 to the signal processing unit B106. Send. Thereafter, the signal processing unit B106 performs image processing on the pixel signal of the main stream and the pixel signal of the substream.

  In addition, when priority is given to the frame rate, the pixel signal whose data amount is reduced as much as possible by performing signal processing on all the pixel signals of the image sensor unit 102 in the signal processing unit A104 inside the image capturing unit 105 is used as the signal processing unit. Can be output to B106. As a result, the amount of data communicated between the image capturing unit 105 and the signal processing unit B106 can be reduced more than when the image quality is prioritized and when the image quality and the frame rate are balanced. It can contribute to the effect of power reduction.

  The object of the present invention can also be achieved by supplying a storage medium storing software program codes for realizing the functions of the above-described embodiments to a system or apparatus. That is, it goes without saying that the object of the present invention can also be achieved when the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing the program code constitute the present invention.

  As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  In addition, the functions of the above-described embodiments can also be realized when an OS (basic system or operating system) running on the computer performs part or all of the actual processing based on an instruction of the program code read by the computer. Realized. Needless to say, this case is also included in the present invention.

  Furthermore, after the program code read from the storage medium is written to the memory provided in the function expansion board inserted in the computer or the function expansion unit connected to the computer, the processing based on the instruction of the program code is also performed. Included in the invention. That is, it goes without saying that the present invention also includes the case where the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing based on the instruction of the program code to realize the functions of the above-described embodiment. Yes.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

Claims (15)

Imaging means for reading out pixel signals in units of rows of the array from a plurality of pixels arrayed two-dimensionally;
Signal processing means for performing signal processing on the pixel signal read from the imaging means;
With
Output means for outputting pixel signals output from the imaging means and the signal processing means;
The imaging means, the signal processing means, and the output means are configured to output a pixel signal from the imaging means in the first processing mode according to a control signal from the outside, and in the second processing mode, The signal processing unit performs the signal processing on at least a part of the pixel signal read from the imaging unit, and the output unit outputs the pixel signal on which the signal processing is performed on the at least part. Have
The image pickup device, wherein the image pickup means, the signal processing means and the output means are formed on the same semiconductor chip.   The image sensor according to claim 1, wherein the signal processing is processing for reducing a data amount of at least a part of the pixel signal.   The image pickup device according to claim 2, wherein the pixel signal read from the image pickup unit is RAW data, and the signal processing is conversion processing to a YUV411 format or a YUV420 format.   The pixel array includes a first array and a second array, each including a plurality of rows, and the first array includes more rows than the second array and is read from the imaging unit. 4. The imaging according to claim 1, wherein the at least part of the output pixel signal is a first pixel signal read from the pixels of the first array. 5. element.   The imaging means, the signal processing means, and the output means are further configured to perform a first signal processing on the first pixel signal in the third processing mode according to the control signal from the outside. Including a configuration in which the second pixel signal read from the pixels of the two arrays is not subjected to the signal processing, and the fourth processing mode performs the signal processing on the first and second pixel signals. The output means outputs the first and second pixel signals from the signal processing means by serial communication in the third processing mode, and the signal processing means in the fourth processing mode. The image sensor according to claim 4, wherein the first and second pixel signals from are output in parallel communication.   The image sensor according to claim 5, wherein a frame rate in the third or fourth processing mode is higher than a frame rate in the first processing mode. An imaging unit having a plurality of pixels arranged in two dimensions, a signal processing unit for performing signal processing on a pixel signal read from the imaging unit, and a pixel signal output from the imaging unit and the signal processing unit are output. In the control method of the image sensor in which the output means to be formed on the same semiconductor chip,
A reading step of reading out pixel signals in units of rows of the array from an imaging means having a plurality of pixels arrayed two-dimensionally;
In the first processing mode, the output means outputs the pixel signal from the imaging means in the first processing mode, and the output means outputs the pixel signal from the imaging means, the signal processing means, and the output means. The signal processing unit performs the signal processing on at least a part of the pixel signal read from the imaging unit, and the output unit operates to output the pixel signal subjected to the signal processing to the at least part. A control method comprising a step.   A program for causing a computer to execute the control method according to claim 7.   A computer-readable storage medium storing a program for causing a computer to execute the control method according to claim 7. Imaging means for reading out pixel signals in units of rows of the array from a plurality of pixels arrayed two-dimensionally;
First signal processing means for performing first signal processing on the pixel signal read from the imaging means;
Second signal processing means for performing second signal processing on a pixel signal output from at least one of the imaging means and the first signal processing means;
Control means for controlling the first signal processing means and the second signal processing means in response to the imaging means reading out the pixel signal;
The control means has a plurality of processing modes of pixel signals read from the imaging means, and the plurality of processing modes are obtained when the second signal processing means receives pixel signals from the imaging means and A first processing mode for performing second signal processing; and the first signal processing means performs the first signal processing on at least a part of the pixel signal read from the imaging means, and A second processing mode in which the signal processing means receives a pixel signal from the first signal processing means and performs the second signal processing;
The imaging apparatus, wherein the imaging means, the signal processing means, and the output means are formed on the same semiconductor chip.   The first signal processing is processing for reducing a data amount of at least a part of the pixel signal, and the second signal processing is image data from the pixel signal acquired by the second signal processing means. The imaging apparatus according to claim 10, further comprising a process of generating   The pixel array includes a first array and a second array, each including a plurality of rows, and the first array includes more rows than the second array and is read from the imaging unit. The image sensor according to claim 11, wherein the at least part of the output pixel signal is a first pixel signal read from the pixels of the first array.   The second signal processing means generates the image data from the first pixel signal, and generates an image processing parameter for generating the image data from the second pixel signal. The imaging device according to claim 12.   In the plurality of processing modes, the first signal processing unit further performs first signal processing on the first pixel signal, and outputs the second pixel signal read from the pixels in the second array. Includes a third processing mode in which the first signal processing is not performed and a fourth processing mode in which the first signal processing is performed on the first and second pixel signals, and the second signal processing means includes In the third processing mode, the first and second pixel signals are obtained from the first signal processing means by serial communication, and in the fourth processing mode, the first signal processing means. The imaging apparatus according to claim 12, wherein the first and second pixel signals are acquired by parallel communication.   The imaging apparatus has a moving image shooting mode and a still image shooting mode, the moving image shooting mode, the still image shooting mode, the first processing mode, the second processing mode, the third processing mode, and the Setting means for setting any one of the fourth processing modes, wherein the control means is configured to select the first processing mode, the second processing mode, the second processing mode in the moving image shooting mode or the still image shooting mode. The image pickup apparatus according to claim 14, wherein control is performed so as to set any one of the third processing mode and the fourth processing mode.

Images