JP2011055188A - Image capturing apparatus, and method of manufacturing the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem in an image capturing apparatus where an moving image is captured in interlaced mode, wherein a memory is likely to be crashed when recording a progressive mode image data. <P>SOLUTION: The image capturing apparatus includes an imaging means for capturing a subject and an interpolation means for interpolating the captured image data. The apparatus generates the progressive mode image data by performing field interpolation processing when the imaging means is driven. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a control method for preventing an imaging system from failing during shooting in an imaging apparatus capable of shooting moving images.

  When executing various signal processing on an imaging apparatus such as a digital camera or a digital video camera, in particular, a captured image, there is a system that temporarily buffers data in a main memory (for example, SDRAM) and executes the various processing. Here, for example, when reading image data from a sensor as an imaging means at high speed, the period during which image data is read out at high speed and the period during which image processing is executed overlap, and access to the main memory is concentrated. There is a case where the memory bandwidth is broken.

  In particular, a problem becomes apparent when a moving image is converted into a progressive signal and recorded, or when a still image that is a progressive signal is recorded during moving image recording. During execution of IP conversion processing for converting an interlaced signal to a progressive signal, the memory band is broken, the main memory cannot be accessed, and as a result, the imaging system is broken.

  Therefore, a method has been proposed for obtaining a high-definition still image even during moving image recording without causing the imaging system to fail. Detection of motion appearing in a captured image, switching between intra-field interpolation processing and inter-field interpolation processing during horizontal / vertical blanking according to the detection result, generating an image for a still image and recording it on a recording medium This is a technique (Patent Document 1).

JP 2007-228119 A

  However, when an image for a still image is generated by the above-described method, interpolation processing during a period in which image data is read from the image sensor at high speed is inter-field interpolation if the motion that appears in the captured image is small. That is, since the previous field is referred to in time, access to the main memory is increased, and further, the image data is read from the solid-state imaging device such as the CMOS and written in the main memory. There is a possibility that.

  In addition, in an imaging apparatus such as a digital camera or a digital video camera, there is a concern that the bandwidth of the SDRAM will be further compressed as the resolution of the imaging unit further increases. In order to avoid this, it is conceivable to increase the capacity of the SDRAM serving as the main memory or to have a plurality of SDRAMs serving as the main memory. However, any of these methods leads to an increase in the cost of the imaging apparatus.

  In view of the above problems, an object of the present invention is to provide an imaging apparatus that obtains high-definition moving images and still images and a control method thereof without increasing the cost and without causing various signal processing of the imaging apparatus to fail. To do.

  In order to achieve the above object, an image pickup apparatus according to the present invention includes an image pickup unit that stores image data obtained by picking up a subject image in an interlaced manner in a memory, and reads out the image data stored in the memory. Interpolating means for interpolating progressive image data, and the interpolating means performs intra-field interpolation during a period in which the imaging means is performing the imaging or storing the image data in a memory. The interpolation process is performed by the following.

  The image pickup apparatus control method according to the present invention includes an image pickup step in which an image pickup unit picks up a subject image in an interlaced manner and stores the obtained image data in a memory, and reads out the image data stored in the memory. An interpolation step for performing interpolation processing to progressive image data, wherein the interpolation processing is performed by intra-field interpolation during a period in which the imaging step is performed. .

  According to the present invention, it is possible to read high-definition image data from the imaging means at high speed without breaking down the bandwidth of the main memory.

Configuration diagram of imaging apparatus for explaining the present embodiment Timing chart for explaining the present embodiment Processing sequence explanatory diagram of this embodiment Flowchart diagram of processing of this embodiment

(First embodiment)
FIG. 1 shows a configuration diagram of an imaging apparatus such as a digital camera or a digital video camera that can be used in this embodiment.

  Reference numeral 101 denotes a lens, reference numeral 102 denotes a solid-state image pickup device such as a CMOS for picking up a subject image, and reference numeral 103 denotes a TG (Timing Generate) unit that performs read control of the solid-state image pickup device 102 such as a CMOS. Reference numeral 104 denotes an A / D converter (sensor interface) that converts an analog signal read from the solid-state image sensor 102 into a digital signal. Reference numeral 105 processes a signal from the solid-state image sensor that has been converted into a digital signal. It is a camera signal processing part which produces | generates. An interpolation unit 106 performs either inter-field interpolation processing or intra-field interpolation processing on the image data and converts the interlace signal into a progressive signal. Reference numeral 107 denotes a moving image resizing unit for setting the image data to a moving image recording angle of view. Reference numeral 108 denotes an NR (Noise Reduction) processing unit for the image resized by the moving image resizing unit 107, and reference numeral 109 denotes a moving image encoding unit that encodes the image data resized to the angle of view of the moving image. A recording signal processing unit 110 converts a video signal and a voice information signal from a microphone (not shown) serving as a voice information input unit into a recording format. The recording signal processing unit 110 further decodes the voice data and uses a speaker ( (Not shown). Reference numeral 111 denotes a recording medium for recording video information and audio information.

  112 is a still image encoding unit that performs encoding processing such as JPEG, and 113 is a still image recording unit such as an SD card that stores still image data encoded by still image encoding means such as JPEG. . Reference numeral 114 denotes a main memory such as an SDRAM for temporarily storing data processed by various signal processing units and functional blocks such as encoding means. Each functional block shares and uses the main memory 114, and passes signal processing data and image data through the main memory 114. Reference numeral 115 denotes an arbiter unit that controls access requests from each functional block to the main memory 114, and arbitrates access. 116 divides a field of view angle made up of image data read from the solid-state image sensor 102 such as a CMOS. This is a region control unit. Reference numeral 117 denotes a movement / subject detection unit that detects the subject being photographed and detects the movement of the subject, and 118 is a system control unit that controls these imaging systems. In this configuration, the shutter is a rolling shutter, and the shutter is sequentially turned off for each scanning line, charge accumulation is performed, and accumulated charge for each image is read.

  The movement / subject detection unit 117 detects a target subject and detects the amount of movement. Although detection methods for subject detection vary depending on the target, in this embodiment, human face detection is performed, the skin color portion is detected, and the subject (face) is determined from the characteristics of the human face (for example, the position of the eyes). . If no skin color portion exists, no subject is present. In addition, the motion amount detection is determined using a motion vector representing the direction and magnitude of the motion of the subject (a motion vector is calculated between frames). The block (for example, N × N pixels) of the previous frame (N−1 frame) and the target frame (N frame) is shifted, the best matching portion is detected, and the motion vector is calculated. Description of these detailed detection methods is a well-known technique and will be omitted.

  When the subject is panned and photographed, or when the subject moves greatly, the photographed image is distorted due to the scanning time difference within the angle of view due to the rolling shutter. Therefore, image data is read from the solid-state imaging device at high speed, the difference in scanning time within the angle of view is suppressed as much as possible, and the influence of image distortion due to the rolling shutter is suppressed. For example, assume that image data is read from the solid-state imaging device in synchronization with the vertical synchronization signal (V). FIG. 2 shows a processing load when image data is read from the solid-state imaging device in synchronization with the vertical synchronization signal (V).

  FIG. 2 (1) shows the occupancy ratio of the memory band with respect to the main memory 114 when image data is read from the solid-state imaging device 102 such as CMOS over a 1V period. The axis is the processing time required for image processing. FIG. 2B shows the memory bandwidth occupancy when image data is read out from the solid-state imaging device 102 such as a CMOS at high speed. Similarly, the vertical axis represents the memory bandwidth occupancy with respect to the main memory 114, and the horizontal axis represents image processing. Processing time required. The shaded portion is the sum of the occupancy ratios of the memory band with respect to the main memory 114 for signal processing in each functional block such as camera signal processing after image data is read from the solid-state imaging device 102 such as CMOS.

  In FIG. 2B, since image data is read out from the solid-state imaging device 102 such as a CMOS at high speed, reading processing from the solid-state imaging device 102 is concentrated in the first half of V. Among each functional block, the occupation ratio of the memory band with respect to the main memory 114 for signal processing of the interpolation unit 106 is particularly high when a moving image is interpolated and recorded as a progressive signal or when a still image in moving image is recorded.

  In the interpolation unit 106, the video signal necessary for the above processing is the N-1 field temporally preceding the N field, the former is a field signal currently obtained from the image sensor 103, and the latter is 1 This is a field signal before the field. Of the two field signals necessary for the processing in the interpolation unit 106, the N-1 field signal is stored in the main memory 114, and therefore is read from the main memory 114 when processing is performed. In addition, in the moving image encoding unit 109, when the moving image encoded frame is a B picture, it becomes a bi-predictive encoded picture, so two frames of past and future I pictures or P pictures are required (a plurality of reference frames are included). ). Therefore, access to the main memory 114 from the moving image encoding unit 109 increases.

  Therefore, in this imaging system, when image data is read out from the solid-state imaging device 102 at a high speed, access to the main memory 114 by signal processing of the interpolation unit 106 is most concentrated. At this time, the occupation ratio of the memory bandwidth exceeds 100%, and there is a possibility that the access request cannot be accepted. As a result, the imaging system fails. Therefore, this embodiment uses a technique that does not cause the system to fail as an imaging apparatus such as a digital camera or a digital video camera, which will be described below.

  In the imaging apparatus such as a digital camera or a digital video camera shown in FIG. 1, image data is read from the solid-state imaging device 102 at high speed by the system control unit 118 in accordance with an instructed timing pulse from the TG 103.

  This aspect is shown in FIG. In FIG. 3, the horizontal axis is a time axis, and (1) to (5) represent the timing of V, respectively. The column “Read” indicates the read timing of each divided area, and “Process” indicates which area of image data is processed at which timing. The colored portions in the column “Sensor Drive” indicate the period during which the solid-state image sensor 102 is driven. In (1), first, image data is read out from the solid-state image sensor 102 at high speed, the image data is converted into a digital signal by the A / D converter 104, and temporarily stored in the main memory 114 via the arbiter 115. . At this time, the image data converted into a digital signal by the A / D conversion unit 104 is also input to the motion / subject detection unit 117. The image data read from the solid-state imaging device 102 at high speed (1) stored in the main memory 114 is read from the main memory 114 at the next V timing (2), and the camera signal processing unit 105 is input.

  Further, the video signal (N field) from the camera signal processing unit 105 is input to the interpolation unit 106, and the video signal and the previous N−1 field read from the main memory 114 are read by the interpolation unit 106. Inter-field interpolation processing is performed by the video signal. After inter-field interpolation processing, the video signal is written back to the main memory 114. Thereafter, in the still image encoding unit 112 that performs encoding processing such as JPEG for recording a still image, the image data is read from the main memory 114, encoded as a still image, and the still image in the main memory 114 is recorded. It is stored in an area where encoded data used only for processing is stored. Thereafter, it is stored in a recording unit 113 such as an SD card for storing still image data.

  The video signal subjected to the inter-field interpolation processing is read from the main memory 114, and the moving image resizing unit 107 converts the image data into the angle of view for moving image recording. Thereafter, an NR (Noise Reduction) processing unit 108 and a moving image resizing unit 107 perform NR processing and resizing on the image data converted to the angle of view of moving image recording. Then, it is encoded by the moving image encoding unit 109, subjected to recording signal processing by the recording signal processing unit 110, and recorded on the recording medium 111. Similarly, at the timing of (2), image data is read from the solid-state image sensor 102 at high speed in real time even during various image signal processing. Similarly, the image data read out from the solid-state imaging device 102 at high speed is converted into a digital signal by the A / D converter 104 and stored in the main memory 114 via the arbiter 115.

  As described above, in the present embodiment, the interpolator 106 performs inter-field interpolation using the N field and N−1 field image data, and then the video signal is written back to the main memory 114. Therefore, accesses to the main memory 114 are concentrated during the interpolation process (write and read requests to the main memory 114). At this time, the timing of accessing the main memory 114 (Write / Read request to the main memory 114) accompanying the high-speed reading process of the image data from the solid-state imaging device 102 overlaps. As a result, the bandwidth for the main memory 114 may be compressed, and an access request to the main memory 114 may not be accepted, which may break down the system as the imaging device. Therefore, as a measure for avoiding system failure, the processing of this embodiment is to divide the one field angle being shot into a plurality of areas and switch the order of various processes for each area.

  A processing sequence according to the present embodiment will be described with reference to FIGS. 3 and 4. In the present embodiment, it is assumed that an imaging apparatus that records moving image data with an interlace signal in normal shooting sets a mode in which moving image data is recorded with a progressive signal. That is, it is assumed that the main memory 114 has a capacity necessary for processing interlaced image data, and that the main memory will be destroyed if the progressive moving image data is processed as it is. 4A is a main flow of the present embodiment, and FIG. 4B is a flow of “reading / interpolation processing”. In either case, the system control unit 118 sends and receives signals to instruct each unit to execute.

  When the image pickup apparatus is turned on, the process is started (S401), and a recording start instruction from the photographer is awaited (S402). After receiving the recording start instruction, the following processing is first executed at the timing shown in FIG. A recording start instruction is issued to a TG (Timing Generator) unit 103, and a pulse signal for reading image data is output to the solid-state imaging device 102 (S403). Image data read from the solid-state image sensor 102 is converted into a digital signal by the A / D conversion unit 104 (sensor interface) and input to the motion / subject detection unit 117. The image data is also output to the main memory 114 via the arbiter unit 115 and stored.

  In response to an instruction from the area control unit 116, the camera signal processing unit 105 divides one angle of view composed of the image data read from the solid-state imaging device 102 into a plurality of areas to obtain divided image data (S404). In the present embodiment, it is assumed that one angle of view is divided into four regions. The division of the area is realized by reading the image data from the main memory (DDR or the like), and the reading order is changed by controlling the read address.

  Further, the presence / absence of the subject being photographed is detected in each region divided into a plurality of regions by the motion / subject detection unit 117 (S405), and the detected motion amount of the subject is detected (S406). Thereafter, reading / interpolation processing is executed (S407).

  Read / interpolation processing will be described with reference to FIG. First, the reading order of each divided area is determined based on the presence / absence of the subject obtained in S405 and S406 and the result of motion amount detection. In the present embodiment, it is assumed that the subject exists in any of the areas A, B, C, and D, and the movement of the subject in the A area is larger than the movement of the subject that exists in the other three areas. At this time, image data corresponding to the area A is first read from the main memory 114 (S411). Therefore, the order of the image data read from the main memory 114 is A → B → C → D as shown in FIG. 3B (B → C → D is originally arbitrary).

  First, the area A is read from the main memory 114 (S411). Thereafter, in the flowchart, there is provided a step of determining whether or not the area read out in S412 is the area read out first in one image data. Here, the process proceeds to S414 as Yes. Since the solid-state image sensor 102 is operating (interacting with the sensor) during the A area interpolation processing, the interpolation processing is performed by intra-field interpolation with less burden on the main memory 114 (S414). Here, since reading of the entire area has not been completed, the process returns to S411 to start reading of the next area. This time, the process proceeds to branch No in S412, and the other three regions (B, C, D) are interpolated by inter-field interpolation (S413). The above is performed until the reading of all areas is completed (S415). When reading of all areas is completed, the process returns to the main flow (S416). Thereafter, it is determined whether or not a recording end instruction has been received. If an end instruction has been received, the process ends (S409). If not, the process returns to S403 to start the operation at the next V timing (S408). ). If reading / interpolation processing from the memory is performed at the timing of FIG. 3 (2), reading / interpolation processing from the memory is performed simultaneously at the timing of the next FIG. 3 (3) by returning to S403. Reading of the image to be performed from the image sensor is started.

  Similarly, subject detection and motion amount detection are also performed at the timing of FIG. This time, it is assumed that there are subjects in A and B, and the movement of the subject in the B area is larger than the movement of the subject in the A area. At this time, image data corresponding to the area B is first read from the main memory 114 (S411). Therefore, the order of the image data read from the main memory 114 is B → A → C → D as shown in FIG. 3B (A → C → D is originally arbitrary).

  First, the area B is read from the main memory 114 (S411). Thereafter, in the flowchart, there is provided a step of determining whether or not the area read out in S412 is the area read out first in one image data. Here, the process proceeds to S414 as Yes. Since the solid-state image sensor 102 is operating (while the sensor is being driven) during the B area interpolation processing, the interpolation processing is performed by intra-field interpolation with a small burden on the main memory 114 (S414). Here, since reading of the entire area has not been completed, the process returns to S411 to start reading of the next area. This time, the process proceeds to branch No in S412, and the other three regions (A, C, D) are interpolated by inter-field interpolation (S413). The above is performed until the reading of all areas is completed (S415). When reading of all areas is completed, the process returns to the main flow (S416).

  At the timings of FIGS. 3 (4) and 3 (5), the processing order is B → A → C → D and D → A → B → C, and the same processing is performed.

  As described above, in an imaging apparatus such as a digital camera or a digital video camera that reads image data from a solid-state imaging device at high speed, intra-field interpolation processing is performed and image data is recorded during the period when the imaging apparatus is driven. An image in the main memory 114 is divided so that the interpolation processing in the intra-field interpolation for the region where the movement of the subject being photographed is divided and the timing for reading out the image data from the solid-state imaging device 102 are divided into a plurality of regions. The data read timing is adaptively controlled. This makes it possible to reduce access to the main memory 114 without executing an access request (Read request) to the main memory 114 during a period in which image data is read from the solid-state imaging device 102. As a result, the bandwidth for the main memory 114 is compressed, and it is possible to prevent the access request for the main memory 114 from being rejected, and the failure of the imaging system can be avoided.

  Further, in the present embodiment, an example in which moving image data is recorded as a progressive signal in moving image shooting has been described. However, the present invention is not limited thereto, and still images that are also recorded as a progressive signal during moving image shooting. It can also be applied when shooting image data. In this case, at the timing of one of the VDs in FIG. 3, the intermediate interpolation process is executed using the image data read at the two previous timings.

  In the present embodiment, one field angle of each captured image data is divided into four divided image data, and the process is executed. The number of divisions is a balance between the resolution of the captured image and the drive performance of the image sensor. It was decided by. Therefore, for example, when an image sensor having a higher reading speed than the image sensor used in the present embodiment is used, it is conceivable that the drive of the image sensor is finished in a time for processing an area of 1/8 of the angle of view. In such a case, if one field angle is divided into eight divided image data and only the first divided image data of one region is subjected to interpolation processing by intra-field interpolation, the memory does not break down and more regions are used. A highly accurate interpolation process can be performed.

  As another embodiment of the present invention, the number of divisions is determined regardless of the performance of the image sensor, and it is determined whether or not the drive of the image sensor is finished every time interpolation processing of divided image data in one region is completed. Add steps to Then, interpolation processing may be performed by intra-field interpolation until driving of the image sensor is completed.

(Second Embodiment)
In the present embodiment, a process is considered when there is no movement of the subject in each of the divided areas as compared with the first embodiment. Note that the description of the same parts as those in the first embodiment is omitted. A processing sequence in the present embodiment will be described with reference to FIGS. 3 and 4 which are the same as those in the first embodiment.

  As in the first embodiment, one angle of view composed of image data read from the solid-state image sensor 102 is divided into four areas (A, B, C, and D in FIG. 3). The presence or absence of a subject (S405) and the amount of movement are detected (S406). Thereafter, the process proceeds to reading / interpolation processing (S407).

For example, at the timing of FIG. 3 (2), when the movement amount of the subject detected at the timing of FIG. 3 (1) is detected and there is no movement of the subject, priority is given to the area where the subject (particularly a person) does not exist. Thus, image data is read from the main memory 114. Here, it is assumed that no subject exists only in the A area. In this case, the order of reading the image data from the main memory 114 is the order of A → B → C → D (S410).
Hereinafter, as in the first embodiment, the first A area is interpolated by intra-field interpolation, and the other three areas (B, C, D) are interpolated by inter-field interpolation. The process is executed, and the process returns to the main process (S416).

  3 (3), (4), and (5), the processing order is determined as B → A → C → D, B → A → C → D, and D → A → B → C, respectively. Processing has been performed.

  As described above, in a system that records moving image data with a progressive signal, the shooting angle of view is divided into a plurality of areas, the movement of the subject in each area and the presence or absence of the subject are detected, and there is no movement of the subject. Intra-field interpolation processing is executed for an area where no subject exists. The reading of the image data in the main memory 114 is adaptively controlled so that the timing of the reading process for reading out the image data from the solid-state image sensor 102 at a high speed matches. Thereby, it is possible to reduce the access to the main memory 114 while performing the interpolation processing without reducing the resolution for the subject. As a result, the bandwidth for the main memory 114 is compressed, and it is possible to prevent the access request for the main memory 114 from being rejected, and the failure of the imaging system can be avoided.

  Needless to say, this embodiment can also be applied to still image shooting during moving image shooting, as in the first embodiment.

  Further, in the first and second embodiments (first embodiment), when the subject is confirmed in all the areas, and there is a magnitude relationship in the amount of movement (second embodiment), the subject is one area. When I couldn't confirm it, I mentioned. However, the present invention can also be applied to various other situations regarding the presence of an object and the amount of movement. For example, if there is an area where the movement of the subject is large and an area where the subject does not exist at the same time, processing may be performed with a priority indicating which one is to be used for intra-field interpolation.

(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

102 Image Sensor 103 TG Unit 105 Camera Signal Processing Unit 106 Interpolation Unit 114 Main Memory 116 Area Control Unit 117 Motion / Subject Detection Unit 118 System Control Unit

Claims (7)

Imaging means for storing image data obtained by imaging a subject image in an interlaced manner in a memory;
Interpolating means capable of reading out image data stored in the memory and performing interpolation processing into progressive image data by intra-field interpolation or inter-field interpolation;
The interpolating unit performs the interpolating process by intra-field interpolation during a period in which the imaging unit is performing the imaging or storing the image data in a memory. Dividing means for dividing the image data stored in the memory into a plurality of divided image data;
Subject detection means for performing subject detection on the divided image data,
The interpolating unit performs intra-field interpolation on the divided image data in which the subject is not detected by the subject detecting unit during the period in which the imaging unit performs the imaging or storing the image data in the memory. The imaging apparatus according to claim 1, wherein interpolation processing is performed. Dividing means for dividing the image data stored in the memory into a plurality of divided image data;
Motion detection means for detecting motion with respect to the divided image data;
In the period during which the imaging unit is performing the imaging or storing the image data in the memory, the interpolating unit is arranged in the field from the divided image data whose motion is detected by the motion detecting unit to the one in which the motion is large. The imaging apparatus according to claim 1, wherein the interpolation process is performed by interpolation.   The imaging apparatus according to claim 2 or 3, wherein the number of divisions of the image data by the dividing unit is determined by a resolution of the image data or a driving speed of the imaging unit. An imaging step of causing the imaging means to capture the subject image in an interlaced manner and storing the obtained image data in a memory;
An interpolation step of reading the image data stored in the memory and performing an interpolation process into progressive image data,
In the interpolation step, the interpolation process is performed by intra-field interpolation during a period in which the imaging step is performed.   A computer-executable program in which a procedure of a control method for an imaging apparatus according to claim 5 is described.   A computer-readable storage medium storing a program in which a procedure of a control method for an imaging apparatus according to claim 5 is described.

Images