JP2012239132A - Image processor, image processing method, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten a time till a processing start.SOLUTION: An imaging part 11 outputs division imaging data of multiple fields obtained by dividing imaging data of one frame. A sensor I/F 50 stores the division imaging data of the first field to be output by the imaging part 11 in a memory 13. Then, the sensor I/F 50 stores the division imaging data of the second field in the memory 13. A memory access part 53 reads the division imaging data of the first field from the memory 13. A calculation part 54 calculates the division imaging data of the second field and the division imaging data of the first field read by the memory access part 53 from the memory 13, thereby to generate division imaging data corresponding to a final field. The memory access part 52 stores the division imaging data corresponding to the final field in the memory 13.

Description

  The present invention relates to an image processing device, an image processing method, and an imaging device.

  Conventionally, an imaging apparatus such as a digital camera converts an imaging signal obtained by an imaging element such as a CCD image sensor into digital imaging data and stores the digital imaging data in a memory (for example, SDRAM). Then, the imaging device displays the image data read from the memory on a display unit (for example, LCD). Further, the imaging apparatus performs processing such as color processing and compression processing on the image data read from the memory, and stores the processed data in the memory.

  A CCD image sensor used as an image sensor employs a method of outputting an image signal (image data) corresponding to a captured image divided into a plurality of times (see, for example, Patent Documents 1 and 2). . An imaging signal (imaging data) output at a time is called a field.

  The imaging apparatus composes a plurality of fields of imaging data output from the imaging device to form one frame, and performs the above color processing and compression processing on the image data of the one frame.

JP 2006-174379 A JP 2009-159056 A

  Various processes for image data require imaging data for one frame. An increase in the number of pixels in the image sensor causes an increase in the number of fields output from the image sensor, thereby prolonging a period for outputting imaging data for one frame. As a result, the waiting time of the user is increased, for example, the update interval of the image displayed on the display unit becomes longer.

  According to an aspect of the present invention, the first field to the (N-1) th field among the divided imaging data of the first to Nth fields (N is an integer of 2 or more) obtained by dividing one frame of imaging data. Generation unit that generates divided imaging data of X fields (X is an integer of 1 or more) including the Nth field based on the divided imaging data of one or more fields, and generated by the generation unit A storage control unit that stores the divided imaging data of the X fields that have been recorded in a storage unit.

  According to one aspect of the present invention, the time until the start of processing can be shortened.

It is a schematic block diagram of an imaging device. It is a block circuit diagram of the sensor interface circuit of the first embodiment. It is explanatory drawing of imaging data. It is explanatory drawing of a multi-field reading system. It is operation | movement explanatory drawing of a sensor interface circuit. (A) and (b) are timing diagrams of the multi-frame readout method. It is a block circuit diagram of the sensor interface circuit of the second embodiment. It is explanatory drawing of a pixel interpolation process. It is a timing diagram of a multi-frame reading system. It is a block circuit diagram of a sensor interface circuit of a third embodiment. It is explanatory drawing of a pixel interpolation process. It is a timing diagram of a multi-frame reading system. (A) and (b) are timing diagrams of the multi-frame readout method. (A) and (b) are timing diagrams of the multi-frame readout method. It is explanatory drawing of a multi-field reading system. (A) and (b) are explanatory drawings of pixel interpolation processing.

Hereinafter, embodiments will be described with reference to the accompanying drawings.
[Overall configuration of imaging device]
As shown in FIG. 1, the imaging apparatus is, for example, a digital still camera, and includes an imaging unit 11, an image processing unit (ISP: Image Signal Processor) 12, a memory (storage unit) 13, and a display device 14.

The imaging unit 11 includes an imaging optical system 21, an imaging element unit 22, and an analog front end (AFE: Analog Front End) 23.
The imaging optical system 21 includes a plurality of lenses (such as a focus lens) that collect light from a subject, a diaphragm that adjusts the amount of light that has passed through these lenses, and the like, and an optical subject image is captured by an imaging device unit. Lead to 22.

  The imaging element unit 22 includes, for example, a Bayer array color filter and an imaging element. The imaging element is a CCD (Charge Coupled Device) image sensor. The imaging element outputs an imaging signal (analog signal) corresponding to the amount of light incident through the color filter.

  The image sensor employs a method of outputting an image signal of one frame corresponding to a captured image divided into a plurality of times. An imaging signal output at a time is called a field. The CCD includes a photoelectric conversion unit that converts light into a signal and a vertical transfer unit that transfers the signal. By increasing the number of fields to be output, the size of the vertical transfer unit for one pixel can be reduced. For this reason, the area of the photoelectric conversion unit in one pixel can be increased, and high sensitivity and a wide dynamic range can be achieved.

  The AFE 23 includes an A / D conversion circuit that converts an analog imaging signal output from the imaging element unit 22 into digital imaging data, and outputs imaging data. Further, the AFE 23 outputs a control signal supplied from the image processing unit (ISP) 12 to the image sensor unit 22 in accordance with the synchronization signal. The synchronization signal includes a vertical synchronization signal indicating one field break and a horizontal synchronization signal indicating one line break.

  The image processing unit 12 includes a sensor interface (I / F) 31, a color processing unit 32, an image processing unit 33, a still image codec unit 34, a memory card interface (I / F) 35, a display interface (I / F) 36, A DMA arbitration unit 37, a memory controller 38, and a CPU (control unit) 39 are provided.

  The sensor I / F 31, the color processing unit 32, the image processing unit 33, the still image codec unit 34, the memory card I / F 35, the display I / F 36, and the DMA arbitration unit 37 are connected to each other via the internal bus 40.

  The DMA arbitration unit 37 is connected to the memory 13 via the memory controller 38. The memory 13 shown in FIG. 1 is, for example, a synchronous semiconductor memory (SDRAM: Synchronous Dynamic Random Access Memory). In the memory 13, imaging data output from the imaging unit 11 is stored by the sensor I / F 31. Further, the memory 13 stores image data processed by the processing units 32 to 34.

The sensor I / F 31 receives imaging data (RGB format image data (Bayer data)) output from the imaging unit 11 and stores it in the memory 13.
The color processing unit 32 is a color space conversion unit, for example, which reads out RGB format image data (Bayer data) stored in the memory 13 and converts the image data into YCbCr format image data. The color processing unit 32 stores the converted image data in the memory 13.

  The image processing unit 33 is one or a plurality of processing units that perform processing other than the above processing. The processing other than the above includes, for example, resolution conversion processing for increasing / decreasing the number of pixels, color tone conversion processing for converting a color tone such as sepia, edge enhancement processing for enhancing the contour (edge) of an image, and noise included in image data. Noise removal processing to be removed is included.

  The still image codec unit 34 reads the image data stored in the memory 13, encodes the image data by a predetermined method (for example, JPEG (Joint Photographic Experts Group) method), and encodes the image data (encoded data). ) Is stored in the memory 13.

  The memory card I / F 35 is connected to the memory card 15 attached to the imaging device. The memory card I / F 35 stores data (for example, compressed image data) stored in the memory 13 in the memory card 15.

  The display device 14 is connected to the display I / F 36. The display device 14 is, for example, a liquid crystal display device (LCD: Liquid Crystal Display). The display device 14 is used for the remaining amount of a battery that is a driving source of the photographing apparatus, a photographing mode, a photographing frame, display of stored image data, and the like. For example, the display I / F 36 reads the image data stored in the memory 13 and outputs the image data to the display device 14.

  The sensor I / F 31, the color processing unit 32, the image processing unit 33, the still image codec unit 34, the memory card I / F 35, and the display I / F 36 each include a memory access controller (DMAC) 31a to 36a. ing. The memory access controllers 31a to 36a output access requests corresponding to the processes performed by the circuits 31 to 36. For example, the sensor I / F 31 stores the imaging data output from the imaging unit 11 in the memory 13. For this reason, the memory access controller 31a of the sensor I / F 31 outputs a write request (write request). The display I / F 36 reads image data to be displayed on the display device 14 from the memory 13. Therefore, the memory access controller 36a of the display I / F 36 outputs a read request (read request).

  The DMA arbitration unit 37 arbitrates the requests (requests) output from the memory access controllers 31a to 36a of the circuits 31 to 36 according to the priority set corresponding to the processing units 31 to 36, for example. Access to one circuit is permitted. The circuit permitted to access outputs a control signal for accessing the memory 13. In the case of a read request, the memory controller 38 reads data from the memory 13 in accordance with the control signal and outputs the read data to the request source. The memory controller 38 outputs a write request and data output from the request source to the memory 13, and the memory 13 stores the data.

  The CPU 39 controls the entire image processing unit 12. The CPU 39 controls the sensor I / F 31. As described above, the imaging device (CCD image sensor) included in the imaging device unit 22 divides one frame of imaging data into a plurality of fields, and sequentially outputs the divided imaging data of each field. The sensor I / F 31 receives the divided imaging data of the field that is sequentially output from the imaging unit 11. Then, the sensor I / F 31 generates divided imaging data corresponding to a field that has not been received from the divided imaging data of the received field, and stores the generated divided imaging data in the memory 13. As a result, the memory 13 stores image data having a data amount corresponding to the number of pixels for one frame. Therefore, the time (imaging data output period) required from when the image sensor unit 22 starts outputting one frame of imaging data to when one frame of imaging data is stored in the memory 13 is the imaging of all frames. It will be shorter than the time required to receive the data. Then, it is possible to start various processes for the imaging data stored in the memory 13 earlier than the timing at which the sensor I / F 31 receives the divided imaging data of all fields.

Next, configuration examples of the sensor I / F will be sequentially described.
[First Embodiment]
As shown in FIG. 2, the sensor I / F 50 includes three memory access units (denoted as DMA0, DMA1, and DMA2) 51, 52, and 53, and one arithmetic unit 54.

  Each of the memory access units 51 to 53 outputs a request according to the setting, and forms a transfer channel with the memory 13. The first memory access unit 51 forms a write channel for storing data in the memory 13. The first memory access unit 51 stores the imaging data output from the imaging unit 11 in the memory 13 via the write channel.

  The second memory access unit 52 forms a write channel for storing data in the memory 13. The second memory access unit 52 stores the data output from the calculation unit 54 in the memory 13 via the write channel. The memory access unit 51 is an example of a storage control unit (first storage control unit). The memory access unit 52 is an example of a storage control unit (second storage control unit).

  The third memory access unit 53 forms a read channel for reading data from the memory 13. The third memory access unit 53 outputs the data read from the memory 13 to the calculation unit 54. The memory access unit 53 is an example of a reading unit.

  Imaging data GD output from the imaging unit 11 is supplied to the calculation unit 54. The calculation unit 54 outputs data generated by calculating the imaging data GD and the data output from the third memory access unit 53 to the second memory access unit 52. In the calculation unit 54, calculation contents corresponding to the pixel position of the imaging data GD that each of the memory access units 51 to 53 accesses (reads or writes) the memory 13 is set.

  The imaging unit 11 divides one frame of imaging data into three fields of divided imaging data and sequentially outputs them. The sensor I / F 50 stores the divided imaging data of the first field in the memory 13 via the first memory access unit 51. Next, the sensor I / F 50 supplies the divided imaging data of the second field to the first memory access unit 51 and the calculation unit 54. The first memory access unit 51 stores the supplied divided image data of the second field in the memory 13. The third memory access unit 53 sequentially reads the imaging data stored in the memory 13, that is, the divided imaging data of the first field, and outputs it to the calculation unit 54. The computing unit 54 generates divided imaging data corresponding to the third field based on the input data, that is, the divided imaging data of the first field and the divided imaging data of the second field.

A relationship between one frame of image data and divided image data of each field will be described.
As shown in FIG. 3, the imaging data of one frame includes a plurality (12 in FIG. 3) of line data L1 to L12. Each of the line data L1 to L12 includes a value (pixel value) corresponding to the amount of light received by each of a plurality (4 in FIG. 3) of pixels arranged in one direction (for example, the horizontal direction) of the image sensor. Each pixel of the image sensor receives light that has passed through a color filter having a predetermined arrangement (for example, a Bayer arrangement). Accordingly, each pixel value included in each line data L1 to L12 includes color information corresponding to the arrangement and color of the color filter.

  For example, in FIG. 3, line data L1, L3, L5, L7, L9, and L11 corresponding to the odd-numbered columns of the image sensor alternately include pixel data of red pixels R and pixel data of green pixels Gr. The line data L2, L4, L6, L8, L10, and L12 corresponding to the stage column alternately include the pixel data of the green pixel Gb and the pixel data of the blue pixel B. The green pixel is indicated by Gr as a green pixel in a column in which red pixels R are arranged in the horizontal direction and Gb in a column in which blue pixels B are arranged in the horizontal direction.

  The line data L1 to L12 are sequentially output along the horizontal direction according to the configuration of the image sensor. For example, the line data L1 to L12 are sequentially output along the horizontal direction from the leftmost pixel to the rightmost pixel.

  This image sensor outputs image data of one frame divided into three fields. The divided imaging data of each field includes each line data for every three lines. Therefore, as shown in FIG. 4, the divided imaging data of the first field (first field) includes line data L1, L4, L7, and L10. Similarly, the divided imaging data of the second field (second field) includes line data L2, L5, L8, and L11, and the divided imaging data of the third field (third field) is line data L3 and L6. , L9, L12.

  As shown in FIG. 3, the line data L3 of the third field includes color information having the same arrangement as the line data L1 of the first field and the line data L5 of the second field. The line data L3 is between the line data L1 and the line data L5, and the distance between the pixel of the line data L3 and the pixel of the line data L1 in the vertical direction is the pixel of the line data L3 and the pixel of the line data L5. Is equal to the distance between The amount of light received by the pixel of line data L3 is often relatively related to the amount of light received by the pixel of line data L1 and the amount of light received by the pixel of line data L5. If the size of the image sensor does not change, the distance between pixels decreases as the number of pixels increases, and the amount of light received by each pixel often has a high correlation.

  For this reason, the sensor I / F 50 shown in FIG. 2 uses the divided imaging data of the first field and the divided imaging data of the second field, and sets the third field based on the relative relationship between the two fields. Corresponding divided imaging data is generated. Specifically, as shown in FIG. 5, the memory access unit (DMA0) 51 stores all line data L1, L4, L7, and L10 included in the first field in the memory 13.

  Next, when imaging data corresponding to the line data L5 of the second field is input, the memory access unit (DMA0) 51 stores the imaging data corresponding to the line data L5 in the memory 13. At this time, the memory access unit (DMA2) 53 sequentially reads the imaging data corresponding to the line data L1 from the memory 13. Then, the calculation unit 54 calculates imaging data corresponding to each pixel of the line data L3 based on the imaging data of each pixel of the line data L1 and the imaging data of each pixel of the line data L5. For example, the distance between the line data L3 and the line data L1, and the distance between the line data L3 and the line data L5 are equal to each other. Therefore, the calculation unit 54 calculates an average value of the imaging data of each pixel of the line data L5 and the imaging data of each pixel of the line data L1, and uses this calculation result as an imaging corresponding to each pixel of the line data L3. Output as data. The memory access unit (DMA1) 52 stores the output value of the calculation unit 54 in an area in the memory 13 corresponding to the line data L3 of the second field.

  Similarly, when the line data L8 of the second field is input, the memory access unit (DMA2) 53 sequentially reads the imaging data of the line data L4 corresponding to the line data L8. The arithmetic unit 54 sequentially calculates imaging data corresponding to each pixel of the line data L6 based on the imaging data of each pixel of the line data L8 and the imaging data of each pixel of the line data L4, and the memory access unit (DMA1 The image data corresponding to each pixel of the line data L6 is sequentially stored in the memory 13.

  Each of the memory access units 51 to 53 determines an area of the memory 13 to be accessed from time to time based on the number of fields output from the imaging unit 11 and the number of lines included in each field. The imaging unit 11 outputs the divided imaging data of each field in synchronization with the synchronization signal (horizontal synchronization signal Hs, vertical synchronization signal Vs). For example, as illustrated in FIG. 6A, the imaging unit 11 captures imaging data corresponding to the line data L1, L4, L7, and L10 included in the first field during one cycle of the vertical synchronization signal Vs. And output for each horizontal synchronizing signal Hs. During one period of the next vertical synchronizing signal Vs, imaging data corresponding to the line data L2, L5, L8, and L11 included in the second field is output for each horizontal synchronizing signal Hs. Further, during one cycle of the next vertical synchronizing signal Vs, imaging data corresponding to the line data L3, L6, L9, and L12 included in the third field is output for each horizontal synchronizing signal Hs.

  Accordingly, each of the memory access units 51 to 53 counts the vertical synchronization signal Vs and the horizontal synchronization signal Hs, respectively, and obtains an area for storing the imaging data of each pixel of each line based on the count value and the arithmetic expression. In addition, for example, a table (LUT: Look Up Table) may be used for acquiring the area. Further, a calculation unit that calculates an area accessed by each of the memory access units 51 to 53 may be provided, and the area may be set for each of the memory access units 51 to 53 from the calculation unit.

Next, the operation of the sensor I / F 50 will be described.
As shown in FIG. 6A, the imaging unit 11 shown in FIG. 2 has the first field line data L1, L4, L7, for each horizontal synchronization signal Hs during one cycle of the vertical synchronization signal Vs. The imaging data GD corresponding to L10 is sequentially output. The sensor I / F 50 stores the imaging data GD corresponding to each line data L1, L4, L7, and L10 in the memory 13 as indicated by hatching.

  Next, the imaging unit 11 sequentially outputs the imaging data GD corresponding to the line data L2, L5, L8, and L11 of the second field for each horizontal synchronization signal Hs during one cycle of the vertical synchronization signal Vs. The sensor I / F 50 stores imaging data GD corresponding to the line data L2 in the memory 13. Next, the sensor I / F 50 stores the imaging data GD corresponding to the line data L5 in the memory 13. The sensor I / F 50 calculates imaging data corresponding to the line data L3 based on the imaging data corresponding to the line data L1 read from the memory 13 and the imaging data corresponding to the line data L5, and stores the data in the memory 13. Store.

  Next, the sensor I / F 50 stores the imaging data GD corresponding to the line data L8 in the memory 13. The sensor I / F 50 calculates imaging data corresponding to the line data L6 based on the imaging data corresponding to the line data L4 read from the memory 13 and the imaging data corresponding to the line data L8, and stores the data in the memory 13. Store. Next, the sensor I / F 50 stores the imaging data GD corresponding to the line data L11 in the memory 13. The sensor I / F 50 calculates imaging data corresponding to the line data L9 based on the imaging data corresponding to the line data L7 read from the memory 13 and the imaging data corresponding to the line data L11, and stores the data in the memory 13. Store.

  As shown in FIG. 4, line data L12 is included in the divided imaging data of the third field (third field). The line data including the same color pixels as the line data L12 and including the color information in the same arrangement as the color information included in the line data L12 is the line data L10 on the upper side in the vertical direction in FIG. In FIG. 4, the line data on the lower side in the vertical direction is not included in the first field and the second field.

  However, the image data (line data L1 to L12) of one frame described above is a still image size (so-called image size). The number of effective pixels or the total number of pixels of the image sensor included in the imaging unit 11 illustrated in FIG. 1 is larger than the number of pixels of the still image, and the imaging data GD of the effective pixels around the pixel region used as the still image is Information necessary for image processing is output from the imaging unit 11 during the output period of each field. The imaging data GD (divided imaging data) of effective pixels in the periphery includes line data (line data two lines below the line data L12) vertically below the line data L12, and is stored in the memory 13. Is done. The sensor I / F 50 captures image data corresponding to the line data L10 read from the memory 13 and image data of peripheral effective pixels corresponding to the second field stored in the memory 13 (two lines below the line data L12). Image data corresponding to the line data L12 is calculated and stored in the memory 13.

  As described above, when the vertical synchronization signal Vs corresponding to the third field is input, the third field calculated based on the divided imaging data of the first field and the divided imaging data of the second field is input to the third field. Corresponding divided imaging data is stored in the memory 13. That is, all of the divided imaging data corresponding to the three fields constituting one frame are stored in the memory 13. Therefore, the processing units 32 to 36 illustrated in FIG. 1 start processing when the imaging unit 11 outputs the divided imaging data GD corresponding to the third field.

  As shown in FIG. 6B, when the divided imaging data corresponding to the three fields are sequentially stored in the memory 13, the processing can be started after the imaging data corresponding to the line data L12 is stored in the memory 13. It becomes. Compared to such a case, when the imaging data is stored in the memory 13 by the sensor I / F 50 shown in FIG. The time required for each process corresponds to the number of pixels. Accordingly, when the processing start timing is advanced, the processing end timing is also correspondingly advanced. Therefore, the time from the start of reading one frame to the end of the processing for the image data of that frame is shorter than the reading method shown in FIG.

As described above, according to the present embodiment, the following effects can be obtained.
(1) The sensor I / F 50 generates divided imaging data corresponding to the final (third) field from the divided imaging data of the first field output from the imaging unit 11 and the divided imaging data of the second field. And stored in the memory 13. Accordingly, each of the processing units 32 to 36 included in the image processing unit 12 performs processing on the imaging data stored in the memory 13 when the imaging unit 11 outputs the divided imaging data corresponding to the final field. Accordingly, the start timing of image processing is earlier than in the case where the divided imaging data of all fields are stored in the memory. If the process starts earlier, the timing at which the process ends is relatively earlier. Accordingly, the time required from the start of reading of imaging data of one frame to the end of processing for that frame, that is, the time required to process imaging data of one frame is shortened. It can be shortened.

  (2) The sensor I / F 50 generates divided imaging data corresponding to the final field output from the imaging unit 11 and stores the imaging data in the memory 13. Therefore, the sensor I / F 50 does not access the memory 13 when the imaging unit 11 outputs the divided imaging data corresponding to the final field. The priority of a request for the sensor I / F 50 to access the memory 13 is higher than the priority of a request made by another circuit. This is because all of the imaging data output by the imaging unit 11 is stored in the memory 13. This is because if the priority is low, the imaging data output by the imaging unit 11 cannot be stored in the memory 13 while waiting for the end of processing of other circuits.

  The sensor I / F that stores the divided imaging data of all fields in the memory 13 operates almost continuously when the imaging data of the frame is continuously output from the imaging unit. For example, when performing processing on imaging data of one frame stored in the memory, the sensor I / F stores imaging data of the next frame in the memory. That is, at the start of processing, the sensor I / F and the processing unit operate simultaneously. For this reason, the access request of the sensor I / F often competes with the access request of another processing unit.

The sensor I / F 50 of this embodiment does not access the memory 13 when outputting the divided imaging data corresponding to the final (third) field output by the imaging unit 11.
Since each processing unit 32 to 36 stores imaging data for one frame in the memory 13, when outputting the divided imaging data corresponding to the final (third) field output by the imaging unit 11, the memory 13 The process for the imaging data stored in the is executed.

  That is, only the processing units 32 to 36 access the memory 13 while outputting the divided imaging data corresponding to the final (third) field output by the imaging unit 11, so that access requests to the memory 13 are reduced. The rate (probability) of competing requests is low. For this reason, since the ratio which each process part 32-36 accesses increases, the average process speed of each process part 32-36 becomes high, ie, a process efficiency can be improved.

[Second Embodiment]
The second embodiment will be described below. In addition, the same code | symbol is attached | subjected about the same member as said each embodiment, and the description of all or one part is abbreviate | omitted.

  As shown in FIG. 7, the sensor I / F 60 includes two memory access units (DMA0, DMA1) 61, 62, two line memories (denoted as LMEM0, LMEM1) 63, 64, and a selection circuit (denoted as SEL) 65. , Multipliers 66 and 67, and an adder 68.

  The imaging data output from the imaging unit 11 is supplied to the first memory access unit 61. The first memory access unit 61 forms a transfer channel with the memory 13 according to the setting, and specifically forms a write channel for storing data in the memory 13. The first memory access unit 61 stores the imaging data output from the imaging unit 11 in the memory 13 via the write channel. The memory access unit 61 is an example of a storage control unit (first storage control unit).

Further, the imaging data output from the imaging unit 11 is supplied to the multiplier 66 and the two line memories 63 and 64.
The first multiplier 66 outputs the result obtained by multiplying the input data by the first coefficient α1 to the adder 68.

  The two line memories 63 and 64 are each set to a memory capacity capable of storing one line data. The two line memories 63 and 64 are controlled so as to store line data alternately. The selection circuit 65 reads the data stored alternately from the two line memories 63 and 64 and outputs the read data to the second multiplier 67.

The second multiplier 67 outputs the result obtained by multiplying the input data by the second coefficient α2 to the adder 68.
The adder 68 adds the data output from the second multiplier 67 to the data output from the first multiplier 66, and outputs the addition result data to the second memory access unit 62.

  The second memory access unit 62 outputs a request according to the setting, forms a transfer channel with the memory 13, and more specifically forms a write channel for storing data in the memory 13. The second memory access unit 62 stores the data output from the adder 68 in the memory 13 via the write channel. The memory access unit 62 is an example of a storage control unit (second storage control unit).

  The sensor I / F 60 generates divided imaging data corresponding to the third field from the divided imaging data of the first field, and stores the generated divided imaging data in the memory 13.

  That is, the first memory access unit 61, for example, based on the synchronization signal (vertical synchronization signal Vs, horizontal synchronization signal Hs), the divided imaging data of the first field and the divided imaging data of the second field, Store in the corresponding area of the memory 13.

  The line memories 63 and 64, the selection circuit 65, the multipliers 66 and 67, and the adder 68 are an example of a generation unit that generates divided imaging data corresponding to the third field from the divided imaging data of the first field. is there. The second memory access unit 62 designates an area corresponding to the third (final stage) field based on, for example, the synchronization signal (vertical synchronization signal Vs, horizontal synchronization signal Hs), and is output from the adder 68. Is stored in the memory 13.

Generation of divided imaging data corresponding to the third field will be described.
The line memories 63 and 64 sequentially store the imaging data of each line included in the first field. As shown in FIG. 9, the first line memory 63 stores imaging data corresponding to the line data L1, and the second line memory 64 stores imaging data corresponding to the line data L4.

  The coefficients α1 and α2 supplied to the multipliers 66 and 67 correspond to the positional relationship between the two lines included in the first field and the lines included in the third field between those lines. Is set to a value. Specifically, the coefficients α1 and α2 are included in the third field from the two lines (first and second reference lines) included in the first field, and have the same color as the first and second reference lines. Set to interpolate array lines.

  For example, as shown in FIG. 8, the divided imaging data of the first field includes line data L1 and line data L7, and the divided imaging data of the third field includes line data L3. The ratio of the distance between the line data L3 and the line data L1 (number between pixels) and the distance between the line data L3 and the line data L7 (number between pixels) is 1: 2. Corresponding to this ratio 1: 2, a pixel corresponding to the line data L3 is calculated by linear interpolation of the pixel value of the line data L1 and the pixel value of the line data L7. In this case, the ratio between the first coefficient α1 and the second coefficient α2 is 2: 1, and the first coefficient α1 and the second coefficient α2 are 0.66 and 0.33, respectively. The first coefficient α1 and the second coefficient α2 are weight values according to the pixel positional relationship between the line data L1 and L7 to be referenced and the line data L3 to be generated.

The adder 68 adds the output data of the second multiplier 67 to the output data of the first multiplier 66, and calculates the addition result.
Accordingly, as shown in FIG. 9, the sensor I / F 60 generates imaging data corresponding to the line data L3 based on the imaging data corresponding to the line data L7 and the imaging data corresponding to the line data L1. Similarly, the sensor I / F 60 generates imaging data corresponding to the line data L6 based on the imaging data corresponding to the line data L10 and the imaging data corresponding to the line data L4.

As described above, according to the present embodiment, the following effects can be obtained.
(1) The sensor I / F 60 generates divided imaging data corresponding to the final field from the divided imaging data of the first field output from the imaging unit 11 and stores the divided imaging data in the memory 13. Therefore, as in the first embodiment, the start timing of image processing is earlier than in the case where the divided imaging data of all fields are stored in the memory. If the process starts earlier, the timing at which the process ends is relatively earlier. Accordingly, the time required from the start of reading of imaging data of one frame to the end of processing for that frame, that is, the time required to process imaging data of one frame is shortened. It can be shortened.

  (2) The sensor I / F 60 sequentially stores the line data of the divided imaging data of the first field output from the imaging unit 11 in the line memories 63 and 64, and based on the line data stored in the line memories 63 and 64. Then, the divided imaging data corresponding to the final field is generated. Accordingly, access to the memory 13 is reduced as compared with the first embodiment, which is suitable for high-speed processing.

[Third embodiment]
Hereinafter, a third embodiment will be described. In addition, the same code | symbol is attached | subjected about the same member as said each embodiment, and the description of all or one part is abbreviate | omitted.

  As shown in FIG. 10, the sensor I / F 70 includes two memory access units (DMA0, DMA1) 61, 62, two line memories (denoted as LMEM0, LMEM1) 63, 64, and a selection circuit (denoted as SEL) 65. In addition to the multipliers 66 and 67 and the adder 68, a memory access unit 71, two multipliers 72 and 73, and an adder 74 are provided.

  Similarly to the first multiplier 66, the imaging data output from the imaging unit 11 and the third coefficient α3 are input to the third multiplier 72. The third multiplier 72 outputs the result obtained by multiplying the input data by the third coefficient α3 to the second adder 74.

  Similarly to the second multiplier 67, the fourth multiplier 73 receives the data alternately read from the two line memories 63 and 64 by the selection circuit 65 and the fourth coefficient α 4. . The fourth multiplier 73 outputs a result obtained by multiplying the data output from the selection circuit 65 by the fourth coefficient α4 to the second adder 74.

The second adder 74 adds the data output from the fourth multiplier 73 to the data output from the third multiplier 72, and outputs the addition result to the third memory access unit 71.
The third memory access unit 71 outputs a request according to the setting, forms a transfer channel with the memory 13, and more specifically forms a write channel for storing data in the memory 13. The third memory access unit 71 stores the data output from the adder 74 in the memory 13 via the write channel. The memory access unit 71 is an example of a storage control unit (second storage control unit).

  The third multiplier 72 is supplied with a third coefficient α3 that is equal to the second coefficient α2 for the second multiplier 67. The fourth multiplier 73 is supplied with a fourth coefficient α4 that is equal to the first coefficient α1 for the first multiplier 66.

  Therefore, as shown in FIG. 11, the third coefficient α3 and the fourth coefficient α4 are the distance between the line data L1 and the line data L5 (the number between pixels) and the line data L5 and the line data L7. Corresponding to the distance (number between pixels). The third coefficient α3 and the fourth coefficient α4 are weight values corresponding to the pixel positional relationship between the line data L1 and L7 to be referenced and the line data L5 to be generated. Therefore, the output data of the second adder 74 that adds the output data of the fourth multiplier 73 to the output data of the third multiplier 72 corresponds to the line data L5. Is included in the second field.

  Therefore, as shown in FIG. 12, the sensor I / F 70, based on the imaging data corresponding to the line data L1 and the imaging data corresponding to the line data L7, the imaging data corresponding to the line data L5 and the line data L3. The imaging data corresponding to is generated. Similarly, the sensor I / F 70 obtains imaging data corresponding to the line data L8 and imaging data corresponding to the line data L6 based on the imaging data corresponding to the line data L4 and the imaging data corresponding to the line data L10. Generate. That is, the sensor I / F 70 generates the divided imaging data corresponding to the second field and the divided imaging data corresponding to the third field from the divided imaging data of the first field.

Accordingly, the processing units 32 to 36 illustrated in FIG. 1 can perform the respective processes when the imaging unit 11 outputs the divided imaging data corresponding to the second field.
As described above, according to the present embodiment, the following effects can be obtained.

  (1) The sensor I / F 70 generates the divided imaging data corresponding to the second field and the divided imaging data corresponding to the last field from the divided imaging data of the first field output from the imaging unit 11, and the memory 13. To store. Accordingly, the start timing of image processing is earlier than in the case where the divided imaging data of all fields are stored in the memory. If the process starts earlier, the timing at which the process ends is relatively earlier. Accordingly, the time required from the start of reading of imaging data of one frame to the end of processing for that frame, that is, the time required to process imaging data of one frame is shortened. It can be shortened.

In addition, you may implement each said embodiment in the following aspects.
In each said embodiment, you may change the process of each process part 32-36 suitably.
For example, as shown in FIG. 13A, a processing time for one frame of imaging data may be set. In this case, for example, the frequency of the clock signal is lowered, and the speed at which the processing units 32 to 36 execute the processing is slowed down. When the operation speed is slowed down, the average power consumption is reduced, so that the power consumption in the imaging apparatus can be reduced.

  Further, as shown in FIG. 13B, the period for performing image processing on the imaging data of one frame is set to end during the transfer of the divided imaging data of two fields. This setting can be made, for example, by increasing the frequency of the clock signal. In this case, the divided imaging data of two fields is output from the imaging unit 11. This can be achieved, for example, by supplying a reset signal SR from the CPU 39 shown in FIG.

  The CPU 39 outputs a reset signal SR when the sensor I / F 31 stores image data having a data amount for one frame in the memory 13. The reset signal SR is sent to the image sensor unit 22 via the AFE 23. In response to the reset signal SR, the imaging element (image sensor) included in the imaging element unit 22 stops outputting the imaging data, and resets the photoelectric conversion unit and the vertical transfer unit (discharges accumulated charges). As a result, the imaging element unit 22 can generate imaging data of the next frame, that is, can accumulate charges according to incident light in the photoelectric conversion unit. In other words, the imaging element unit 22 can perform imaging at an interval shorter than the interval at which the divided imaging data of all fields constituting one frame is output, that is, increase the number of frames to be imaged per unit time. Can do.

  The CPU 39 outputs a reset signal SR according to the setting. The setting is performed by operating an operation button (not shown) or a touch panel provided together with the display device 14. The CPU 39 displays a menu screen for setting on the display device 14, and stores a setting value input or selected according to the operation in a register (for example, a nonvolatile memory included in the CPU 39). Then, the CPU 39 performs control for each processing unit including the sensor I / F 31 and outputs a reset signal SR to the imaging unit 11 according to the setting.

  In the second embodiment, the divided imaging data corresponding to the third field is generated based on the divided imaging data of the first field, but the third field is supported based on the divided imaging data of the second field. The divided imaging data to be generated may be generated. For example, in the sensor I / F 60 shown in FIG. 7, the values of the first coefficient α1 supplied to the first multiplier 66 and the second coefficient α2 supplied to the second multiplier 67 are switched. For example, the imaging data corresponding to the line data L6 included in the third field is generated based on the imaging data corresponding to the line data L2 and the imaging data corresponding to the line data L8.

  In the second embodiment, the referenced field may be configured to be changeable. For example, as shown in FIG. 14A, a mode for interpolating and generating divided imaging data corresponding to the third field from the divided imaging data of the first field, and the second field as shown in FIG. A mode for interpolating and generating divided imaging data corresponding to the third field from the divided imaging data can be switched.

  In each of the above forms, the number of fields may be two or four or more. The sensor I / F is configured according to the field output from the imaging unit 11 (imaging device), and among the input divided image data of a plurality of (two or more) fields, the division corresponding to at least the final stage field. Imaging data is generated.

  For example, in the case of an imaging unit that divides and outputs one frame into eight fields, each field (1F to 8F) includes a corresponding line as shown in FIG. FIG. 15 shows a case where the number of pixels (lines) in the vertical direction is 32. Further, the line data L33 and L34 illustrated in FIG. 15 are lines obtained by reading imaging data of pixels (for example, ring pixels) included in the periphery of a pixel region used as a still image.

  As described above, when the divided imaging data of eight fields is output from the imaging unit, for example, the first to fourth fields (1F to 4F) are stored in the memory 13 as illustrated in FIG. Then, the divided imaging data corresponding to the fifth field and the divided imaging data corresponding to the seventh field are generated from the divided imaging data of the first field (1F) and the third field (3F). Similarly, the divided imaging data corresponding to the sixth field and the divided imaging data corresponding to the seventh field are generated from the divided imaging data of the second field (2F) and the fourth field (4F). Then, the divided imaging data corresponding to the generated four fields is stored in the memory 13.

  Also, as shown in FIG. 16B, the divided imaging data corresponding to the third, fifth, and seventh fields is generated from the divided imaging data of the first field (1F), and the second field (2F) is divided. The divided imaging data corresponding to the fourth, sixth, and eighth fields is generated from the imaging data, and each divided imaging data is stored in the memory 13.

  Even with this configuration, it is possible to speed up the start of processing for imaging data of one frame, and the processing time related to the imaging data of one frame (from the start of output of divided imaging data to the end of image processing). Time required) can be shortened.

  The pixel area corresponding to the set still image size (image size) is different from the area (number of effective pixels: image area) including pixels that can be used as a still image set as a digital camera (image size). May be smaller than the image area). In such a case, divided image data corresponding to at least the final field is generated using pixels in the image area, and pixels that cannot be generated by interpolation (for example, line data included in the divided image data of the third field shown in FIG. 4). (L12 pixel) is not generated. In other words, it is only necessary to generate divided imaging data for pixels recorded as a still image or a moving image, and data may not be generated for pixels that are not recorded (processing is omitted).

  In the above embodiment, the image sensor unit 22 including a Bayer array color filter is used. However, an image sensor unit including another array color filter may be used. Further, the pixel arrangement direction is not limited to two orthogonal axes. Furthermore, an image sensor in which pixels of each color are formed in the depth direction of the chip may be used. Alternatively, an image sensor unit including filters of colors other than red (R), blue (B), and green (Gr, Gb) may be used. The filter is not limited to the primary color filter, and a complementary color filter may be used.

DESCRIPTION OF SYMBOLS 11 Image pick-up part 12 Image processing part 13 Memory 31 Sensor interface 50, 60, 70 Sensor interface 51-53 Memory access part 54 Calculation part 61,62 Memory access part 63,64 Line memory 65 Selection part 66,67 Multiplication part 68 Addition part 71 Memory Access Unit 72, 73 Multiplication Unit 74 Addition Unit GD Imaging Data L1-L12 Line Data

Claims (12)

Among the divided imaging data of the first to Nth fields (N is an integer of 2 or more) obtained by dividing the imaging data of one frame, the one or more of the fields from the first field to the (N−1) th field. A generation unit that generates divided imaging data of X fields (X is an integer of 1 or more) including the Nth field, based on the divided imaging data;
A storage control unit for storing the divided imaging data of the X fields generated by the generation unit in a storage unit;
An image processing apparatus comprising: Of the divided imaging data of the first to Nth fields (N is an integer of 2 or more) obtained by dividing the imaging data of one frame, the divided imaging data of the first field to the Mth field (M is an integer less than N). A first storage control unit for storing
A generating unit that generates the divided imaging data of the (M + 1) th to Nth fields based on the divided imaging data of one or more fields among the divided imaging data of the first to Mth fields;
A second storage control unit that stores the divided imaging data generated by the generation unit in the storage unit;
An image processing apparatus comprising: The generator is
One line data included in the divided imaging data of the field to be generated is generated based on a plurality of line data included in the divided imaging data of the field to be referenced.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The divided imaging data is image data including color information corresponding to the position of each pixel,
The generation unit generates the divided imaging data by calculating using pixel data of the same color included in a field to be referenced.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The generation unit generates the divided imaging data by assigning a weight according to a positional relationship between a plurality of lines to be referred to and a line to be generated to pixel values of the plurality of line data to be referred to.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The generator is
A readout unit for reading out the divided imaging data of the field stored in the storage unit;
The line included in the field generated by calculating pixel data included in the divided imaging data of the field output from the imaging unit that outputs the divided imaging data and pixel data read out by the reading unit. An arithmetic unit for generating pixel data;
The image processing apparatus according to claim 2, further comprising: The generator is
A plurality of line memories for storing line data included in one field to be referenced;
A selection unit for selecting output data of the line memory corresponding to the line data to be referenced;
A calculation unit that generates the line data of a field that is generated by calculating line data to be referenced and output data of the line memory selected by the selection unit;
The image processing apparatus according to claim 1, further comprising: The imaging unit that outputs the divided imaging data at a timing according to the field to be generated has a control unit that outputs a signal for the imaging unit to stop the divided imaging data and start processing for the next frame.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. Among the divided imaging data of the first to Nth fields (N is an integer of 2 or more) obtained by dividing the imaging data of one frame, the one or more of the fields from the first field to the (N−1) th field. Based on the divided imaging data, X divided field imaging data including X fields (X is an integer equal to or greater than 1) including the Nth field are generated.
Storing the divided imaging data of the generated X fields in a storage unit;
An image processing method. Of the divided imaging data of the first to Nth fields (N is an integer of 2 or more) obtained by dividing the imaging data of one frame, the divided imaging data of the first field to the Mth field (M is an integer less than N). Is stored in the storage unit,
Storing the divided imaging data of (M + 1) th to Nth fields generated based on the divided imaging data of one or more fields among the divided imaging data of the first to Mth fields in the storage unit;
An image processing method. An imaging unit that outputs divided imaging data of first to Nth fields (N is an integer of 2 or more) obtained by dividing imaging data of one frame;
A storage unit capable of storing imaging data for at least one frame;
X fields including the Nth field based on the divided imaging data of one or more fields from the first field to the (N-1) th field among the divided imaging data of the first to Nth fields. A process including: a generation unit that generates (X is an integer equal to or greater than 1) divided imaging data; and a storage control unit that stores the divided imaging data of X fields generated by the generation unit in a storage unit And
An imaging device. An imaging unit that outputs divided imaging data of first to Nth fields (N is an integer of 2 or more) obtained by dividing imaging data of one frame;
A storage unit capable of storing imaging data for at least one frame;
A first storage control unit that stores, in a storage unit, the divided imaging data of the first field to the Mth field (M is an integer less than N) among the divided imaging data of the first to Nth fields; A generation unit that generates the divided imaging data of the (M + 1) th to Nth fields based on the divided imaging data of one or more fields among the divided imaging data of the 1st to Mth fields, and the generation unit A second storage controller that stores the divided imaging data of the (M + 1) th to Nth fields in the storage unit;
An imaging device.

Images