JP3902824B2 - Image processing apparatus and method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, and more specifically to an image processing apparatus and method for recording a moving image and a desired still image in the moving image on a recording medium.
[0002]
[Prior art]
Electronic still cameras that digitally record captured images using a semiconductor memory, a magnetic disk, a magneto-optical disk, or the like instead of a silver salt film as a recording medium have been researched and commercialized. If the recording medium is randomly accessible, it can be played back in an order different from the recording order, and any recorded image can be played back immediately. Digital recording also has advantages such as that image quality is not deteriorated even by dubbing, and scalability on the time axis is easily utilized. However, digital recording has a drawback that the amount of recording data is enormous compared to analog recording.
[0003]
As a means for reducing the amount of recorded data, an information compression technique is useful, and various methods such as a moving image MPEG method have been proposed in addition to a JPEG method for still images. In the JPEG method, the value in the spatial coordinate system is orthogonally converted (DCT conversion) into a value on the frequency coordinate in units of 8 × 8 pixels, the conversion coefficient is quantized, and the variable length code is formed by the Huffman encoding method or the like. Thereby, the data amount is compressed to approximately 1/10 or more. Since it is not lossless encoding, the input signal cannot be faithfully reproduced, but this is not a problem at a normal image quality level. To restore the compressed information, the reverse process of the compression is followed.
[0004]
In still image shooting alone, image quality is given priority, and even if post-processing (compression processing and recording processing to a recording medium) after capturing an image takes some time, it was acceptable. However, in the case of high-speed continuous shooting, the time interval for continuous shooting is limited by the time required for post-processing, so the time required for compression processing and recording processing can be reduced, or either part can be operated in parallel. It is necessary to reduce the waiting time for image capture.
[0005]
In addition, when trying to record a short-time moving image, a frame rate of about 30 frames / second (or 60 fields / second) must be secured in order to smooth the screen movement. For this reason, it is necessary to shorten the time required for the post-processing, and of course, it is necessary to increase the recording capacity of the recording medium and increase the image compression rate. In particular, when compression and recording are performed in real time, the configuration becomes complicated, and a recording medium that can be written at high speed must be used. On the other hand, there has been proposed an electronic still camera configured to transfer image information to a high-capacity second semiconductor memory for low-speed writing via a first semiconductor memory capable of high-speed writing (for example, a patent in 2001) Application Publication No. 10784 and 1993 Patent Application Publication No. 49000).
[0006]
In the MPEG system, which is a well-known compression technique for moving images, the compression rate is increased by adding compression in the time axis direction to compression of each screen. For example, the frames before and after the reference screen (or the start frame (or field; the same applies hereinafter)) should have a high correlation with the reference screen (similar pixel information can be obtained) even if there is a movement. If the difference from the reference image value is obtained at the same pixel address and encoded, the compression effect is enhanced, and further, the amount of motion can be reduced by performing block matching with the comparison target of the comparison frame in a certain pixel block unit. The difference is obtained by shifting in the horizontal pixel direction and the vertical line direction by the corresponding amount, and the compression effect is further enhanced.
[0007]
If such moving image compression processing is performed, a higher compression rate can be achieved, but the configuration of the apparatus becomes complicated and expensive.
[0008]
As described above, the random accessibility should be able to be played by designating an arbitrary frame when playing a moving image. When inter-frame compression is performed at the time of moving image compression recording, if the frame to be reproduced and displayed is an intra image that does not depend on inter-frame compression, it can be reproduced and displayed only with the information of that frame. In this case, the frame alone cannot be expanded, and image information of the previous or previous frame as a reference is required. Furthermore, an arbitrary frame cannot be output (reproduced or printed out) with higher image quality than a moving image.
[0009]
If attention is paid to the fact that the image quality requirement of moving images is generally lower than that of still images, inter-frame compression is a high-efficiency compression coding method that uses human visual mechanisms. Is very effective. However, as described above, when one arbitrary frame is reproduced and output, it is not possible to expect a high quality image that can sufficiently withstand viewing or printing.
[0010]
[Problems to be solved by the invention]
For example, it is possible to execute still image processing for a desired frame separately from the moving image processing even during moving image recording. Specifically, a release switch for still image shooting is provided separately from the trigger button for starting and stopping moving image shooting, and even during continuous shooting, pressing the release switch for still image shooting allows you to freeze the captured image during that time. Execute image processing.
[0011]
The operation will be specifically described with reference to FIG. 3, FIG. 4 and FIG. 3 to 5 are schematic diagrams showing a flow on a time axis of a series of images. FIG. 3 is a schematic diagram showing the flow of the captured image on the time axis. In the NTSC system, an image of about 720 × 480 pixels is interlaced and captured at 60 fields / second. The sampling frequency is 13.5 MHz for the luminance signal and 6.75 MHz for the chrominance signals RY and BY. After the sampling frequency of the chrominance signal is halved to a 4: 1: 1 component signal , Compress information. In addition to displaying on a TV, it may be incorporated into a personal computer and used for various purposes. At that time, the number of pixels and the frame rate may not be required. For example, 320 × 240 pixels and 30 frames / second may be sufficient. However, even a moving image of this level requires a transfer rate of 18 Mbps with 8 bits taken in without compression.
[0012]
When a 4: 1: 1 component signal is compressed to about 1/10 of each frame by the JPEG method, the video information per frame has a capacity of about 12 Kbytes. Even in this case, the data amount is 0.4 Mbytes in one second and 21 Mbytes in one minute, and the number (or time) of images that can be recorded cannot be increased (or increased). However, in this state, since each frame is an intra-frame compressed image (intra image), decompression display is possible with only a single frame. That is, random reproduction is easy. There are many recording media such as a semiconductor memory, a magnetic disk, an optical disk, and a hard disk. However, there is no recording medium that satisfies all the conditions of high-speed writing, small size, large capacity, and low cost.
[0013]
If the data is written on the recording medium in a highly compressed state as in the MPEG system, the transfer rate to the recording medium is lowered and the recording time is drastically increased. FIG. 4 shows a time series of image information in the case of MPEG compression. The series of image information shown in FIG. 3 is temporarily stored in the moving image processing frame memory, and the moving image compression / decompression circuit compresses the correlation between frames in the time axis direction in real time.
[0014]
In FIG. 4, only 10 frames during moving image shooting are shown. The number assigned to the frame is a frame number indicating the compression processing order.
[0015]
The first frame is called an intra picture, and all block information is information-compressed within the frame. What is necessary is just to consider that it is the same as the frame (a) of FIG. The frames (b) and (c) in FIG. 3 are temporarily stored in the frame memory, and the frame (d) is first compressed. The frame (d) is differentially encoded with the first frame as a reference frame or a reference frame. At this time, if there are few motion components for each block, a simple difference value may be used, but if necessary, motion compensation may be performed from a motion vector and compression coding may be performed.
[0016]
In the third and fourth frames, the frames (b) and (c) in FIG. 3 are subjected to frame correlation compression under prediction from both directions of the first and second frames. In the figure, arrows indicate reference relationships for inter-frame compression.
[0017]
In the case of FIG. 3, still images are continuous. However, when recording is compressed by the motion JPEG method, the resolution must be about 320 × 240 pixels in consideration of the recording capacity, and displayed or printed as a still image. It's hard to say that the image quality is enough to go out. Now, assuming that the frame (e) is a still image frame intended by the user, the corresponding sixth frame in FIG. 4 is not an intra image.
[0018]
Therefore, for a frame (for example, frame (e)) that has been requested to record a still image during moving image recording, as shown in FIG. 5, as a still image with a higher resolution, for example, 640 × 480 pixels, that is, a single frame. A method of compressing information can be considered.
[0019]
However, since the amount of information is four times the pixel size of the moving image, the compression process takes a lot of time. If the maximum compression processing capacity of the video is 30 frames per second, it is possible to simply process 4 times the amount of information in 1/30 seconds by interrupting a still image. That is, time for 4 frames of moving images is spent. Therefore, in the interrupt simultaneous processing, as shown by a dotted line in FIG. 5, the moving image following the high-definition still image is lost for three frames.
[0020]
In other words, if high-definition still image processing is interrupted only during moving image processing, because high-definition still image processing is high-definition and compression processing takes time, so-called frame dropping such as a drop in the frame rate of moving images occurs. . Naturally, it is sufficient if high-definition still images can be processed at high speed within the frame rate of the moving image, but at the same time, the configuration of the imaging device is complicated, and at the same time, the compression processing capacity is fully utilized in the moving image compression processing during moving image shooting. There will be no waste.
[0021]
It is an object of the present invention to provide an image processing apparatus and method that solve such problems.
[0022]
[Means for Solving the Problems]
An image processing apparatus according to the present invention includes a first compression unit that compresses input moving image data, and a memory that temporarily stores image data of a desired frame in the moving image data as still image data. A second compression unit that compresses still image data stored in the memory, moving image data obtained from the first compression unit, and still image data obtained from the second compression unit The second compression means performs a compression process of moving image data by the first compression means in parallel with the compression process by the first compression means. The compression processing is performed within a time range not hindering.
An image processing method according to the present invention includes a first compression step of compressing input moving image data, and a step of temporarily storing image data of one desired frame in the moving image data in a memory as still image data. A second compression step for compressing still image data stored in the memory, moving image data obtained in the first compression step, and still image data obtained in the second compression step And a recording step for recording the moving image data in the first compression step in parallel with the compression processing in the first compression step. The compression processing is performed within a time range not hindering.
[0023]
Thereby, a desired frame can be recorded as a still image without dropping frames in the moving image recording.
[0024]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0025]
FIG. 1 shows a schematic block diagram of an embodiment of the present invention. 10 is a CCD image sensor, 12 is an A / D converter that converts the analog output of the image sensor 10 into a digital signal, and 14 is a component signal that separates output data of the A / D converter 12 into luminance data and color difference data. Is a digital signal processing circuit. A portion composed of the A / D converter 12 and the digital signal processing circuit 14 is referred to as a camera signal processing circuit 16.
[0026]
Reference numeral 18 denotes a moving image compression / decompression circuit that performs image compression / decompression processing on image signals using the frame memory 20, and reference numeral 22 denotes still image compression / decompression processing that performs image information compression / decompression processing on image information using the frame memory 24. Circuit.
[0027]
Reference numeral 26 denotes a system control circuit for controlling the whole, 28 denotes a final recording medium for recording captured image information, and 30 denotes an interface for connecting the recording medium 28 to the system control circuit 26. 32 is an electronic viewfinder (or optical viewfinder), and 32 is an external output terminal for video / audio.
[0028]
Reference numeral 36 denotes an operation key including a continuous shooting trigger 38 and a still image shooting request release 40, 42 denotes a display for displaying the number of shot frames, the remaining amount of recording memory, and the like, and 44 denotes a power supply circuit.
[0029]
The basic operation of the embodiment shown in FIG. 1 will be described. When continuous shooting is instructed by the continuous shooting trigger 38, the output image of the camera signal processing circuit 16 is temporarily stored in the moving image frame memory 20, compressed by the moving image compression / decompression circuit 18 by the motion JPEG method, It is recorded on the recording medium 28.
[0030]
When the still image release 40 is operated during continuous shooting (that is, when a still shooting request is input), the system control circuit 26 compresses the moving image by the moving image compression / decompression circuit 18 and applies it to the recording medium 28. While continuing the recording, the image information of the corresponding frame is stored in the frame memory 24 and the still image shooting flag is written in the recording medium 28. At the end of continuous shooting, the system control circuit 26 reads the image information from the still image frame memory 24, compresses the still image by the still image compression / decompression circuit 22, and separates it from the moving image recording area of the recording medium 28. Record in the area. Compared to a moving image, real-time characteristics are not required, so it may take some time for the still image compression / decompression circuit 22 to compress the still image.
[0031]
In the present embodiment, the recording medium 28 is, for example, a card type flash memory, but is not limited thereto, and may be a hard disk, an optical disk, a magneto-optical disk, a magnetic disk memory, or the like.
[0032]
During reproduction, moving image information is read from the recording medium 28 and sequentially decompressed by the moving image compression / decompression circuit 18. In the middle, when a still image frame shooting flag is detected, high-definition still image data recorded in another area of the recording medium 28 is read out and decompressed by the still image compression / decompression circuit 22.
[0033]
With reference to FIGS. 3, 4 and 5, the characteristic operation of the present embodiment will be described in detail. 3 can be regarded as a schematic diagram showing the flow on the time axis of the image output from the camera signal processing circuit 16. The moving image compression / decompression circuit 18 temporarily stores the series of image information shown in FIG. 3 in the moving image processing frame memory 20 and compresses it by taking the correlation between frames in the time axis direction in real time.
[0034]
In the present embodiment, for a frame (for example, frame (e)) for which a still image recording request was made during moving image recording, a high-definition still image having four times the information amount is temporarily stored in the frame memory 24 as it is. Alternatively, the image data is stored with a pre-processing level that requires only a short time, and the real-time compression of the moving image is preferentially processed. After moving image shooting is completed, still image data is read from the frame memory 24, compressed by the still image compression / decompression circuit 22, and written into the still image recording area of the recording medium 28. The frame memory 24 is preferably capable of storing a plurality of still images.
[0035]
More preferably, during moving image recording, still image data is slowly read out from the frame memory 24 and recorded on the same recording medium 28 or another recording medium within a time range that does not hinder the real-time compression processing of the moving image. In this case, it is possible to afford to capture a new still image frame in the frame memory 24. As a result, the frame memory 24 can be saved.
[0036]
FIG. 2 shows a schematic block diagram of a modified embodiment of the present invention. The same components as those in FIG. 1 are denoted by the same reference numerals. A moving image compression / decompression circuit 50 in place of the moving image compression / decompression circuit 18 includes a DCT circuit 50a, a quantization circuit 50b, a variable length coding circuit 50c, an inverse DCT circuit 50d, an inverse quantization circuit 50e, a variable length decoding circuit 50f, and It consists of a motion compensation circuit 50g. A still image compression / expansion circuit 52, which replaces the still image compression / expansion circuit 22, includes a DCT circuit 52a, a quantization circuit 52b, a variable length coding circuit 52c, an inverse DCT circuit 52d, an inverse quantization circuit 52e, and a variable length decoding circuit. 52f and uses two frame memories 54a and 54b. In this embodiment, there are two recording media 56 and 58, the recording medium 56 is made of a magnetic tape, and the recording medium 58 is made of a semiconductor memory capable of writing at high speed, for example, a flash memory. 60 is a large-scale gate array CPU that comprehensively controls writing and reading to and from the moving image compression / decompression circuit 50, still image compression / decompression circuit 52, and recording media 56, 58, and 62 connects the CPU 50 to the recording media 56, 58. Interface.
[0037]
The moving image compression / decompression circuit 50 performs pixel conversion, thinning, DCT processing, and the like on the image data of 640 × 480 pixels from the camera signal processing circuit 16 by using the moving image frame memory 20, and further performs motion compensation. Inter-frame correlation compression is performed using the motion vector of the circuit 52g. The compressed image data is recorded on a recording medium 56 made of a magnetic tape.
[0038]
Still image recording during moving image shooting is performed as follows. The image data of the frame specified by the still image release 16 is taken into the still image frame memory 54a with the resolution of 640 × 480 pixels. When another still image frame is captured during moving image shooting, the still image is stored in the second frame memory 54b. The still image compression / decompression circuit 52 compresses still images stored in the still image frame memories 54a and 54b. The compressed image data is recorded on the recording medium 58.
[0039]
In this embodiment, by providing the still image compression / decompression circuit 52 separately from the CPU 60, the load on the CPU 60 can be reduced, and still image compression processing can be executed at high speed. Of course, it goes without saying that the functions of the moving image compression / decompression circuit 50 and the still image compression / decompression circuit 52 can also be realized by software, depending on the processing capability of the CPU 60.
[0040]
【The invention's effect】
As can be easily understood from the above description, according to the present invention, compression recording is simplified in a format suitable for the quality and quantity of moving images, and any frame during movie shooting is dropped. High-definition recording can be done without
[0041]
Furthermore, at the time of playback display, by linking the still image frame flag and still image data in the moving image data, cueing of the moving image scene from the still image, or conversely, high-definition still image printing from the moving image scene, etc. Can be done easily.
[Brief description of the drawings]
FIG. 1 is a schematic block diagram of a first embodiment of the present invention.
FIG. 2 is a schematic block diagram of a second embodiment of the present invention.
FIG. 3 is a schematic diagram showing image information output from the camera signal processing circuit 16 in time series.
FIG. 4 is a schematic diagram showing, in a time series, image information to be compressed.
FIG. 5 is a schematic diagram showing a compression result of a conventional example in time series when there is a still image request during continuous shooting.
[Explanation of symbols]
10: CCD image pickup device 12: A / D converter 14: digital signal processing circuit 16: camera signal processing circuit 18: moving image compression / expansion circuit 20: moving image compression frame memory 22: still image compression / expansion circuit 24: still image Image compression frame memory 26: system control circuit 28: recording medium 30: interface 32: electronic viewfinder (or optical viewfinder)
34: video / audio external output terminal 36: operation key 38: continuous shooting trigger 40: still image shooting request release 42: display 44: power supply circuit 50: moving image compression / expansion circuit 50a: DCT circuit 50b: quantization Circuit 50c: Variable length coding circuit 50d: Inverse DCT circuit 50e: Inverse quantization circuit 50f: Variable length decoding circuit 50g: Motion compensation circuit 52: Still image compression / decompression circuit 52a: DCT circuit 52b: Quantization circuit 52c: Variable Long encoding circuit 52d: Inverse DCT circuit 52e: Inverse quantization circuit 52f: Variable length decoding circuits 54a, 54b: Frame memory 56, 58: Recording medium 60: Large-scale gate array CPU
62: Interface

Claims (6)

First compression means for compressing input moving image data ;
A memory for temporarily storing image data of a desired frame in the moving image data as still image data;
Second compression means for compressing still image data stored in the memory;
Recording means for recording the moving image data obtained from the first compression means and the still image data obtained from the second compression means on a recording medium
And
The second compression means performs the compression processing in parallel with the compression processing by the first compression means and within a time range that does not inhibit the compression processing of the moving image data by the first compression means. A featured image processing apparatus. The still image, the image processing apparatus according compared to the moving image data to claim 1 is a high-definition image. The image processing apparatus according to claim 1, wherein the recording medium includes a first recording medium that records a moving image and a second recording medium that records a still image. A first compression step of compressing processing input moving image data,
A step for temporarily storing in a memory image data of a desired frame in said video image data as still image data,
A second compression step of compressing processed still image data stored in said memory,
A recording step of recording the moving image data obtained in the first compression step and the still image data obtained in the second compression step on a recording medium.
And
In the second compression step, the compression processing is performed in parallel with the compression processing in the first compression step and within a time range that does not inhibit the compression processing of the moving image data in the first compression step. A featured image processing method. The still image, the image processing method according to claim 4 in comparison with the moving image data is a high-definition image. The image processing method according to claim 4 , wherein the recording medium includes a first recording medium that records a moving image and a second recording medium that records a still image.

Images