JP3015044B2 - Image recording / reproducing device, image reproducing device, and image recording device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recording apparatus using a solid-state imaging device combined with a color separation filter.

[Prior Art] A device that converts a signal of a solid-state imaging device into a digital signal, performs digital signal processing on the signal, and records and reproduces the image as a video signal has attracted attention. A typical example is an all-solid-state camera that converts video information into digital values and records the digital values in a semiconductor memory or the like. In the above system,
The signals output from the image sensor need to be subjected to various and various signal processes depending on the arrangement of a color separation filter attached to the image sensor (hereinafter, abbreviated as a sensor). Is converted into a standard signal form (for example, luminance signal + color difference signal) independent of the arrangement of the color separation filters, and is recorded in a semiconductor memory or the like.

At this time, the sensor signal is once converted to a standard form (for example,
R, G, B signals) and recorded.

[Problems to be Solved by the Invention] However, the conventional example described above has a disadvantage that the amount of information to be recorded is originally increased and recorded in the storage memory. For example, when the output of a sensor having a mosaic filter as shown in FIG. 1 is taken into a memory, the memory capacity for a luminance signal cannot be reduced in sampling frequency, so that it is practically the same as the number of pixels on the sensor. In addition, since the color difference signals obtained line-sequentially need to be synchronized and stored originally, a capacity several times larger than the information amount of the original sensor is required. Of course, it is possible to lower the sampling frequency for the color difference signal. However, in that case, a digital filter for lowering the sampling frequency is required. Further, since there is a limit in lowering the sampling frequency, a significant increase in the information from the original sensor cannot be avoided. This is the same in a sensor having a stripe filter as shown in FIG. To store the information in memory without increasing the amount of information,
Recording may be performed, but in this case, there is a disadvantage that compatibility is left with a large problem due to differences in signal processing due to the arrangement of various color filters.

[Means for Solving the Problems] An object of the present invention is to solve the problems of the conventional example described above, and stores a color signal from a solid-state imaging device in a memory without performing color conversion. And a matrix calculating means for converting the selected signal into a predetermined color signal or a luminance signal.

As a specific embodiment, the reproducing apparatus is provided with an n × m matrix signal processing means for converting the sensor signals into RGB and luminance signals, respectively. When recording in the memory by the recording apparatus, the sensor signals are directly stored as much as possible. Let me
By recording the conversion matrix constants of the matrix signal processing and the number of vertical and horizontal movement steps of the conversion matrix on the sensor on a memory, a signal output of the same form can be obtained for an arbitrary color arrangement. Is what makes it possible.

[Operation] According to the present invention, even if the signal of the sensor is almost directly written into the memory, compatibility is not impaired, so that significant memory saving can be achieved.

Embodiment First, the principle of matrix signal processing, which is a point of the present invention, will be described. Single and two-panel cameras use only a limited number of color separation filters of complementary and pure colors. In signal processing, to achieve white balance, return to the R, G, B system once, and then , Is converted into a required signal form.

For example, in a complementary color mosaic filter as shown in FIG. 1, such a conversion is performed by a signal processing circuit as shown in FIG. That is, regarding the color signal processing, the image sensor 1
Is passed through a matrix circuit A for converting to rgb,
Matrix circuit with white balance adjustment function
NTSC (PA) via γ correction circuit 2 and color difference matrix circuit 3
(L, SECAM, HD, etc.). On the other hand, with respect to the luminance signal processing system, high-frequency components are extracted through a gamma correction circuit 5 and a digital filter 6 through a matrix circuit D for converting the output of the image sensor 1 into a luminance signal. The low-frequency luminance signal is combined with the adder 7 and converted into a luminance signal Y. In other words, to obtain the necessary color signal from the viewpoint of the image sensor,
It suffices if there is matrix information of a color signal matrix G = A × B. To obtain a luminance signal, it is sufficient that a matrix D exists. In actual signal processing, vertical and horizontal sampling steps are further required. That is, since an n × m matrix is calculated by fitting it onto the pixels of the image sensor 1, the matrix is
Next, information on which pixel position to move is required.
The calculation movement of this matrix will be described with reference to FIG. First, the signals of the respective pixels of the image sensor 1 are arranged in x × y on the memory RAM so that they can be accessed randomly (or, using a delay line or the like, the data necessary for the matrix operation is appropriately It can be extracted.) In this case using a window matrix G _C shown in FIG. 2, the G _C, x × y on the coordinates (1, 1)
, And by the matrix operation in FIG.
g, b and Y are calculated. Next, the window matrix is moved according to a sampling step prepared in advance. For example, in the example of FIG. 1, the window matrix that was initially at (1,1) moves to (3,1) after two cycles, and this movement is set in advance. In addition, it can be performed by the sampling step = 2. Also, if the size of x × y of the sensor is set in advance, the window matrix can be stopped when the coordinates exceed the coordinates, and after the cycle corresponding to 1H, the x coordinates are again set. , Return to the initial position, and move the window matrix in a vertical sampling step set in advance for y, thereby performing signal processing for the next line.

The same applies to the luminance signal. The luminance signal is obtained by calculating a signal processing matrix D prepared in advance for the window matrix, and the color and the color are obtained by moving the window matrix in a sampling step. Similar signal processing can be performed. As described above, by preparing the window matrix and the signal processing matrix, the signal processing of the mosaic filter and the stripe filter, the signal processing of the pure color and the complementary color, which were completely different from the conventional signal processing form, can be simply performed by the same hardware form in the matrix. Can be processed simply by changing. FIG. 3 shows a signal processing block diagram of the present invention including a window matrix. Elements having the same reference numerals as those in FIG. 2 indicate the same elements. The RAM is a fixed or removable random access memory. FIG. 4 shows an example of a pure color stripe filter arrangement and a matrix at this time. An adaptation in the case of FIG. 4 in the hardware configuration of FIG. 3 will be described. In the case of a pure color filter, the portion of the matrix A in FIG. 4 is a 3 × 3 unit matrix as shown in FIG. 4, and the luminance signal is also represented by a 3 × 1 matrix. This can also be applied to a case where a Y signal is obtained by mixing line signals for the purpose of enhancing the sensitivity of a sensor in a movie or the like. In this case, as shown in FIG. 5, the window matrix G _C is expanded in the Y direction, and as a result, the array elements of D need only be lengthened, and Y ₁ or Y ₂ is selected for each field. You just have to put it out.

Of course, in this case, if the initial coordinates for each field of the window mask are newly prepared as information,
This matrix can be reduced. In the case of the Y system, if this window matrix is made large, it is possible to output it as a signal for a vertical enhancer, and it is needless to say that this operation can be included in a luminance matrix.

FIG. 6 is an example of an all-solid-state camera utilizing the above-described principle of the present invention, and its operation will be described below.
71 is an optical system lens group, 72 is an aperture, 73 is an infrared cut filter, 74 is an optical low-pass filter, 75 is an on-chip color separation filter, and the image passing through the 71 to 75 system is An image is formed on the solid-state image sensor, and is converted into electric charge by the solid-state image sensor 76. The converted electric charge is read out in synchronization with a signal from a system signal generator (hereinafter referred to as SSG) 97 and enters a variable gain amplifier 77. The variable gain amplifier 77 is used for sensitivity correction when the adjustment cannot be completed even with the aperture of 72, and this amplifier also performs KNEE correction processing at the same time. 78 is an optical black clamp circuit for fixing the black level of the sensor, where the signal whose DC level is fixed is converted into a digital value by the A / D converter of 79, and the memory of 98 The addresses are sequentially stored in the memory 80 indicated by the controller.
The memory 80 may be detachable or fixed. Reference numeral 96 denotes an average level detection circuit for performing exposure control using the output of the image pickup device. By reading this value, the aperture 72 and the gain-amplifying amplifier 77 are controlled to adjust the exposure level. In this embodiment, the electronic shutter operation is performed by controlling the drive timing of the image sensor 76. In this embodiment, the format of the writing to the memory is recorded in the form as shown in FIG. That is, as shown in FIG. 7, in the header file, the number of currently recorded images, their titles, and if anything other than images are recorded, their attributes are digitized and recorded. An ID file portion is provided at the head of each image signal recording area. In the ID file portion, matrix information of the signal processing matrices 81 and 82 and information necessary for a window mask such as a grid shape of a sensor and a sampling step are recorded. Subsequently, the image data is recorded. At the time of reproduction, first, through the memory controller 98, the ID is read out by the system controller, and the matrix constants and digital filter constants are transferred to the 81, 82 matrix signal processing circuits and the 95 digital filters. Then, the window mask information and the sampling step are transferred to the memory controller 98, and the reproduction state is set. During playback, 98 memory controllers form a window mask,
The information corresponding to the position of the window is transferred to the color signal processing matrix circuit 81 and the luminance signal processing matrix circuit 82 in accordance with the sampling step.
Matrix operation signal processing is performed based on the matrix information transferred from the file, and a luminance signal and a color difference signal are formed. The formed luminance signal then enters a digital filter 95, where only the high-frequency components are extracted, and the adder 94 controls the correction circuit 104 and the matrix circuit 105 calculated by the color signal processing matrix 81. is combined with the low frequency luminance signal Y _L obtained through, then the signal from the system signal controller 100 at the time of reproduction, receives the blanking process in the blanking circuit 85, then the adder
After a synchronizing signal is added by 93 and converted into an analog signal by a D / A converter 88, the sampling carrier is dropped by a low-pass filter 90 and output as a luminance signal via an amplifier 92. On the other hand, the color difference signal output from the matrix circuit 105 is first added with a blanking pulse by a blanking circuit of 83 and 84, and then converted into a chroma signal by an NTSC encoder 86. Converted to bandpass filter 89, amplifier
The color signal C is output through 91. A digital filter may be inserted after the color signal processing matrix circuit 81.

FIG. 8 is a diagram showing an example of a 3 × 4 matrix of an n × m window matrix according to the present invention;
With this window matrix, necessary data is obtained from the random access memory, and thereby, correspondence is made with d ₁ to d ₁₂ of the signal processing matrix.
The specific signal form is a one-bit signal as shown in FIG. 8, which is used to select data to be involved in the signal. In addition, as a signal for controlling the window mask, a signal for setting start coordinates is provided. Information such as information and sampling steps is required.

The signal obtained in this way is input to a 3 × 12 signal processing circuit as shown in FIG. 9 and is subjected to a matrix operation as shown in FIG. 10 to be output as r, g and b signals. At this time, the values of a _{1 to} a ₂₄ are values specific to the sensor, and the constants are values including white balance information, signal processing methods, characteristics of color filters, and the like.

Although the window matrix has been described as n × m, this is not limited to a rectangle, and the shape of the window matrix may be any shape of window having l elements. . The same can be said for the matrix of the luminance signal processing.

FIG. 11 is a diagram showing an example of a matrix circuit for luminance signal processing. In this example, an example of a 2 × 12 matrix is shown. y ₁ and y ₂ indicate the results after the signal processing in the first field and the second field, respectively. These results
The output is switched for each field by the multiplexer MPX. In addition, the constants b _{1 to} b ₂₄ at this time are specifically constants indicating information such as Y-system white balance, luminance step correction information, and vertical enhancer.

This luminance signal matrix can also be composed of k × j (k and j are natural numbers), and the present invention uses 2 × j as in this embodiment.
It is not limited to 12. When k × j and m × n are the same, the latch in front of the signal processing circuit can be shared, and if the signal processing circuit operates at sufficiently high speed, it can be operated in the same time-division manner. .

In the embodiment, the data is directly stored in the memory RAM. However, it is needless to say that the data may be compressed and stored, and may be expanded at the time of reproduction.

[Effects of the Invention] According to the present invention, various signal processing circuits can be configured with a common circuit by the arrangement of the color separation filters of the image sensor, and the effect of mass production can be achieved when integrated circuits are used. Can be expected. In addition, if a solid-state camera is manufactured using the present invention, even if the signal of the sensor is directly taken into the memory, compatibility is not impaired, so that the signal of the sensor can be directly recorded in the memory as a recording form. And increase the efficiency of memory usage.

[Brief description of the drawings]

FIG. 1 is a diagram showing an example of the arrangement of complementary color mosaic filters and an example of a matrix formula, FIG. 2 is a diagram for explaining the principle of the present invention, FIG. 3 is a diagram showing a configuration example of the present invention, FIG. FIG. 5 is a diagram showing an arrangement example of a pure color stripe filter and an example of a matrix type. FIG. 5 is a diagram showing another example of a matrix type for luminance for the pure color stripe filter. FIG. 6 is a specific configuration of an image recording apparatus of the present invention. FIG. 7 is a diagram showing an example of a memory space, FIG. 8 is a diagram showing another example of a window matrix, FIG. 9 is a diagram showing an example of a color signal processing matrix circuit, FIG. FIG. 9 is a diagram showing an example of a color signal matrix equation in FIG. 9, and FIG. 11 is a diagram showing an example of a luminance signal processing matrix circuit.

Continuation of the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) H04N 9/07 H04N 9/64-9/78 H04N 9/79

Claims (14)

(57) [Claims] 1. Recording means for recording an image signal from a solid-state imaging device without performing color conversion, and selecting means for selecting block data of an arbitrary size from the image signals recorded by the recording means. Matrix operation means for converting the selected image signal into a predetermined color signal or luminance signal; and the recording means, together with the image signal, information on the size of block data used for matrix operation, matrix At least one of an arithmetic constant, a digital filter constant for correcting a phase difference resulting from an array of color filters of the solid-state imaging device, a color difference matrix constant, and a digital filter constant for limiting the band of luminance and color difference signals. An image recording / reproducing apparatus configured to record one image. 2. An image pickup device, comprising: a recording unit for recording an image signal from the image pickup unit without performing color conversion; and the recording unit includes, together with the image signal, a size of block data used for matrix operation. Information, matrix operation constants, constants of digital filters for correcting the phase difference resulting from the arrangement of the color filters of the imaging means, constants of the color difference matrix, constants of the digital filters for limiting the band of the luminance and color difference signals. An image recording apparatus for recording at least one of the following. 3. A recording means for recording at least one of information on the size of block data used for matrix operation, a matrix operation constant, and a constant of a color difference matrix together with an image signal output from an image pickup means. An image reproducing apparatus comprising: reproducing means for reproducing the image data; and matrix calculating means for converting an image signal into a predetermined color signal or a luminance signal based on the information reproduced by the reproducing means. 4. The apparatus according to claim 3, wherein said recording means further comprises a digital filter constant for correcting a phase difference resulting from an arrangement of color filters of said image pickup means, and a digital filter for limiting a band of luminance and color difference signals. A digital filter that records at least one of the constants of the filter and reproduces the information from the recording unit by the reproducing unit to perform band limiting of the luminance and color difference signals based on the reproduced information. An image reproducing apparatus comprising: 5. An image pickup unit, and a recording unit for recording an image signal from the image pickup unit without performing color conversion, wherein the recording unit performs a matrix operation for performing a color conversion process together with the image signal. An image recording apparatus for recording information relating to the size of block data used for recording. 6. An image pickup unit, and a recording unit for recording an image signal from the image pickup unit without performing color conversion, wherein the recording unit performs a matrix operation for performing a color conversion process together with the image signal. An image recording apparatus for recording information about constants. 7. An image pickup means for picking up an image of a subject, and a recording means for recording an image signal from the image pickup means without performing color conversion, wherein the recording means performs a color conversion process together with the image signal. An image recording apparatus for recording information on a constant of a color difference matrix for use in recording. 8. An image pickup means, and a recording means for recording an image signal from the image pickup means without performing color conversion, wherein the recording means, together with the image signal, includes a color filter array of the image pickup means. An image recording apparatus for recording information on a constant of a digital filter for correcting a generated phase difference. 9. An image pickup device, comprising: a recording device for recording an image signal from the image pickup device without performing color conversion, wherein the recording device performs band limiting of a luminance and a color difference signal together with the image signal. An image recording apparatus for recording information relating to a constant of a digital filter according to (1). 10. A reproduction means for reproducing information from a recording means for recording information on the size of block data used for matrix calculation together with an image signal output from an imaging means, and reproduction means for reproducing the information. An image reproducing apparatus comprising: a matrix operation unit that converts an image signal into a predetermined color signal or a luminance signal based on information. 11. A reproducing means for reproducing information from a recording means for recording information on a matrix operation constant together with an image signal output from an imaging means, and an image signal based on the information reproduced by the reproducing means. And a matrix operation means for converting the image data into a predetermined color signal or luminance signal. 12. A reproducing means for reproducing information from a recording means for recording information relating to a constant of a color difference matrix together with an image signal output from an imaging means, and an image based on the information reproduced by the reproducing means. An image reproducing device comprising: a matrix operation unit that converts a signal into a predetermined color signal or a luminance signal. 13. A recording means for recording information relating to a constant of a digital filter for correcting a phase difference generated from an arrangement of color filters of said image pickup means together with an image signal outputted from said image pickup means to reproduce and reproduce said information. An image reproducing apparatus comprising: a reproducing unit; and a digital filter that limits a band of a luminance and a color difference signal based on information reproduced by the reproducing unit. 14. A reproducing means for reproducing information from a recording means for recording information on a constant of a digital filter for limiting a band of a luminance signal and a color difference signal together with an image signal outputted from an imaging means, and said reproducing means. An image reproducing apparatus, comprising: a digital filter that limits a band of a luminance signal and a color difference signal based on information reproduced by the image reproducing apparatus.