JP5631598B2 - Data processing device - Google Patents

Description

  The present invention relates to a data processing apparatus, and more particularly to a data processing apparatus that is applied to a digital video camera and assigns common audio data to image data having different qualities.

  An example of this type of device is disclosed in Patent Document 1. According to this background art, a video signal and an audio signal are input from a camera unit and a microphone, respectively. The input video signal and audio signal are recorded on the tape cassette by the tape recording / reproducing unit and also recorded on the memory card by the memory card recording / reproducing unit.

JP 2005-109923 A

  However, the background art does not assume that the recording timing of the audio signal with respect to the tape cassette exactly matches the recording timing of the audio signal with respect to the memory card. Here, if the respective recording timings are to be accurately matched, the load may increase due to the difference in the recording method.

  SUMMARY OF THE INVENTION Therefore, a main object of the present invention is to provide a data processing apparatus capable of reducing a load when assigning common audio data to image data corresponding to different methods or qualities.

  A data processing apparatus according to the present invention (10: reference numeral corresponding to the embodiment; the same applies hereinafter) is based on image capturing means (22) for periodically capturing original image data, based on the original image data captured by the image capturing means. First creating means (26, 38) for creating first image data corresponding to the first quality, second corresponding to the second quality lower than the first quality based on the original image data captured by the image capturing means. Second creation means (28, 40) for executing processing for creating image data in parallel with the creation processing of the first creation means, the first image data created by the first creation means and the second creation means Further, there are provided assigning means (S21, S25) for assigning audio data to each of the second image data, and control means (S17) for activating the assigning means with reference to the progress of creation processing by the first creating means.

  Preferably, the frame rate of the first image data corresponds to 1 / M (M: integer) of the frame rate of the original image data, and the frame rate of the second image data is 1 / N (N : Integer greater than M).

  Preferably, the resolution of the second image data is lower than the resolution of the first image data.

  Preferably, the control means activates the assigning means each time the number of frames of the first image data created by the first creating means reaches a predetermined value.

  More preferably, the first image data corresponds to stream data in which an intra-coded image and an inter-coded image are mixed, and the default value corresponds to a cycle in which the intra-coded image appears.

  Preferably, a voice capturing means (36) for capturing voice data is further provided, and the assigning means targets the voice data captured by the voice capturing means.

  Preferably, imaging means (16) for outputting image data representing the object scene is further provided, and the original image data captured by the capturing means corresponds to the image data output from the imaging means.

  According to this invention, the first image data and the second image data are created based on the common original image data that is periodically captured. The first image data creation process and the second image data creation process are executed in parallel with each other. Furthermore, the quality of the second image data is lower than the quality of the first image data. Therefore, when attention is paid to the original image data captured in a certain cycle, the corresponding second image data is completed at the time when the creation processing of the first image data based on the original image data is completed. The audio data is assigned to each of the first image data and the second image data with reference to the process of creating the first image data. Thus, it is possible to reduce a load when assigning common audio data to image data corresponding to different methods or qualities.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

It is a block diagram which shows the basic composition of this invention. It is a block diagram which shows the structure of one Example of this invention. FIG. 3 is an illustrative view showing one example of a mapping state of an SDRAM applied to the embodiment in FIG. 2; It is an illustration figure which shows an example of the allocation state of two cutout areas in the raw image area of SDRAM. (A) is an illustrative view showing an aspect ratio of an image corresponding to MP4 data, and (B) is an illustrative view showing an aspect ratio of an image corresponding to 3GP data. (A) is an illustrative view showing an example of a data structure of an MP4 file, and (B) is an illustrative view showing an example of a data structure of a 3GP file. FIG. 3 is a timing chart showing a part of the operation of the embodiment in FIG. 2; It is a flowchart which shows a part of operation | movement of CPU applied to the FIG. 2 Example. It is a flowchart which shows a part of other operation | movement of CPU applied to the FIG. 2 Example. FIG. 11 is a flowchart showing still another portion of behavior of the CPU applied to the embodiment in FIG. 2; FIG. 10 is a flowchart showing yet another portion of behavior of the CPU applied to the embodiment in FIG. 2;

Embodiments of the present invention will be described below with reference to the drawings.
[Basic configuration]

  Referring to FIG. 1, the image processing apparatus of the present invention is basically configured as follows. The image capturing unit 1 periodically captures original image data. The first creating unit 2 creates first image data corresponding to the first quality based on the original image data captured by the image capturing unit 1. The second creating means 3 performs a process for creating the second image data corresponding to the second quality lower than the first quality on the basis of the original image data fetched by the image fetching means 1. And run in parallel. The assigning unit 4 assigns audio data to each of the first image data created by the first creating unit 2 and the second image data created by the second creating unit 3. The control means 5 activates the assignment means 4 with reference to the progress of the creation processing by the first creation means 2.

The first image data and the second image data are created based on common original image data that is periodically captured. The first image data creation process and the second image data creation process are executed in parallel with each other. Furthermore, the quality of the second image data is lower than the quality of the first image data. Therefore, when attention is paid to the original image data captured in a certain cycle, the corresponding second image data is completed at the time when the creation processing of the first image data based on the original image data is completed. The audio data is assigned to each of the first image data and the second image data with reference to the process of creating the first image data. Thus, it is possible to reduce a load when assigning common audio data to image data corresponding to different methods or qualities.
[Example]

  Referring to FIG. 2, the digital video camera 10 of this embodiment includes a focus lens 12 and an aperture unit 14 driven by drivers 18a and 18b, respectively. The optical image of the object scene is irradiated onto the imaging surface of the image sensor 16 through these members. Note that the effective image area on the imaging surface has a resolution of horizontal 2560 pixels × vertical 1600 pixels.

  When the power is turned on, the CPU 46 activates the driver 18c to execute the moving image capturing process under the recording control task. In response to the vertical synchronization signal Vsync generated every 1/60 seconds, the driver 18c exposes the imaging surface and sequentially reads out the charges generated on the imaging surface in a scanning manner. From the image sensor 16, raw image data representing the object scene is output at a frame rate of 60 fps.

  The preprocessing circuit 20 performs processing such as digital clamping, pixel defect correction, and gain control on the raw image data output from the image sensor 16. The raw image data subjected to such preprocessing is written into the raw image area 24a (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  Referring to FIG. 4, cutout areas CT1 and CT2 are allocated to raw image area 24a. The cutout area CT1 has a resolution (aspect ratio is 16: 9) corresponding to horizontal 1920 pixels × vertical 1080 pixels. On the other hand, the cutout area CT2 has a resolution (aspect ratio is 4: 3) corresponding to horizontal 640 pixels × vertical 480 pixels.

  The post-processing circuit 26 accesses the raw image area 24a through the memory control circuit 22, and reads out raw image data corresponding to the cut-out area CT1 every 1/60 seconds in a sequential scanning manner. The read raw image data is subjected to processing such as color separation, white balance adjustment, YUV conversion, edge enhancement, zooming, etc. As a result, image data corresponding to the 1080 / 60p system (see FIG. 5A) is obtained. Created. The created image data is written into the YUV image area 24b (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  The LCD driver 30 repeatedly reads out the image data stored in the YUV image area 24b, reduces the read-out image data to match the resolution of the LCD monitor 32, and the LCD monitor based on the reduced image data. 32 is driven. As a result, a real-time moving image (through image) representing the scene is displayed on the monitor screen.

  The preprocessing circuit 20 also simply converts the raw image data into Y data, and provides the converted Y data to the CPU 46. The CPU 46 performs AE processing on the Y data under the imaging condition adjustment task, and calculates an appropriate EV value. The aperture amount and the exposure time that define the calculated appropriate EV value are set in the drivers 18b and 18c, respectively, and thereby the brightness of the through image is appropriately adjusted. The CPU 46 also performs AF processing on the high frequency component of the Y data when the AF activation condition is satisfied. The focus lens 12 is arranged at the focal point by the driver 18a, and thereby the sharpness of the through image is continuously improved.

  When a recording start operation is performed toward the key input device 48, the CPU 46 accesses the recording medium 44 through the I / F 42 under the recording control task, and newly creates an MP4 file and a 3GP file in the recording medium 44 ( The created MP4 file and 3GP file are opened).

  When the file creation & opening process is completed, the CPU 46 activates the MPEG4 encoder 38 to start the MP4 encoding process, activates the post-processing circuit 28 and the MPEG4 encoder 40 to start the 3GP encoding process, and captures audio. The AAC encoder 36 is activated to start the process.

  The MP4 encoder 38 repeatedly reads the 1080 / 60p format image data stored in the YUV image area 24b through the memory control circuit 22, encodes the read image data according to the MPEG4 format, and encodes encoded image data, that is, MP4 data. Is written into the encoded image area 24d (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  The post-processing circuit 28 reads out part of the raw image data belonging to the cut-out area CT2 through the memory control circuit 22, and processes such as color separation, white balance adjustment, YUV conversion, edge enhancement, and zooming into the read raw image data. 480 / 30p image data (see FIG. 5B) is created, and the created image data is written into the YUV image area 24c (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  The MP4 encoder 40 repeatedly reads the 480 / 30p format image data stored in the YUV image area 24c through the memory control circuit 22, encodes the read image data according to the MPEG4 format, and encodes the encoded image data, that is, 3GP data. Is written into the encoded image area 24e (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  The AAC encoder 36 encodes the audio data output from the microphone 34 according to the AAC method, and writes the encoded audio data, that is, AAC data, into the encoded audio area 24 f (see FIG. 3) of the SDRAM 24 through the memory control circuit 22.

  After that, every time 60 frames of MP4 data is obtained, the CPU 46 transfers the latest 60 frames of MP4 data and the latest 1 second of AAC data to the open MP4 file, and the latest 30 frames of 3GP data and the latest 1 second of data. Are transferred to an open 3GP file.

  The latest 60 frames of MP4 data and the latest one second of AAC data are read from the encoded image area 24d and the encoded audio area 24f by the memory control circuit 22, and written to the MP4 file via the I / F 42. The latest 30 frames of 3GP data and the latest AAC data for 1 second are read from the encoded image area 24e and the encoded audio area 24f by the memory control circuit 22 and written to the 3GP file via the I / F 42. .

  Therefore, MP4 data and AAC data are stored in the MP4 file in the manner shown in FIG. Further, the 3GP data and the AAC data are stored in the 3GP file in the manner shown in FIG.

  When a recording end operation is performed toward the key input device 48, the CPU 46 stops the MPEG4 encoder 38 in order to end the MP4 encoding process, and causes the post-processing circuit 28 and the MPEG4 encoder 40 to end in the 3GP encoding process. Stop and stop the AAC encoder 36 to start the audio capture process.

  Thereafter, the CPU 46 performs termination processing. As a result, the MP4 data of less than 60 frames and the AAC data of less than 1 second remaining in the SDRAM 24 are written to the MP4 file, and the 3GP data of less than 30 frames and the AAC data of less than 1 second remaining in the SDRAM 24 are written to the 3GP file. . The open MP4 file and 3GP file are closed after termination processing is completed.

  Referring to FIG. 7, the MP4 encoding process and the 3GP encoding process are executed in parallel with each other. However, since the quality of the 3GP data is lower than the quality of the MP4 data (quality: frame rate and / or resolution), when the MP4 data creation process based on the raw image data of a certain frame is completed, the corresponding 3GP data Is completed. Focusing on such time characteristics, in this embodiment, AAC data is assigned to each of MP4 data and 3GP data while referring to the progress of the MP4 data creation processing.

  More specifically, both MP4 data and 3GP data are data streams in which intra-coded pictures (= I pictures) and inter-coded pictures (= P pictures or B pictures) are mixed. One GOP is formed by 60 frames of MP4 data starting from an I picture or 30 frames of 3GP data starting from an I picture. Furthermore, both 60 frames of MP4 data and 30 frames of 3GP data represent one second of moving images.

  Based on this, AAC data for one second is allocated to each of MP4 data for one second and 3GP data for one second at the timing when an I picture forming MP4 data is created. As a result, it is possible to reduce the load when assigning common audio to moving images corresponding to different methods or qualities.

  The CPU 46 processes a plurality of tasks including the recording control task shown in FIGS. 8 to 10 and the imaging condition adjustment task shown in FIG. 11 in parallel. Note that control programs corresponding to these tasks are stored in the flash memory 50.

  Referring to FIG. 8, in step S1, a moving image capturing process is executed. As a result, a through image is displayed on the LCD monitor 32. In step S3, it is repeatedly determined whether or not a recording start operation has been performed. If the determination result is updated from NO to YES, the process proceeds to step S5. In steps S5 and S7, the recording medium 44 is accessed through the I / F 42, and an open MP4 file and 3GP file are newly created in the recording medium 44. In step S9, the MPEG4 encoder 38 is activated to start the MP4 encoding process. In step S11, the post-processing circuit 28 and the MPEG4 encoder 40 are activated to start the 3GP encoding process. In step S13, the audio capturing process is started. Accordingly, the AAC encoder 36 is activated.

  The MP4 encoder 38 repeatedly reads the 1080 / 60p format image data stored in the YUV image area 24b through the memory control circuit 22, encodes the read image data according to the MPEG4 format, and encodes encoded image data, that is, MP4 data. Is written into the encoded image area 24d through the memory control circuit 22.

  The post-processing circuit 28 reads a part of raw image data belonging to the cut-out area CT2 through the memory control circuit 22, creates 480 / 30p format image data based on the read raw image data, and creates The image data is written into the YUV image area 24c through the memory control circuit 22.

  The MP4 encoder 40 repeatedly reads the 480 / 30p format image data stored in the YUV image area 24c through the memory control circuit 22, encodes the read image data according to the MPEG4 format, and encodes the encoded image data, that is, 3GP data. Is written into the encoded image area 24e through the memory control circuit 22.

  The AAC encoder 36 encodes the audio data output from the microphone 34 in accordance with the AAC method, and writes the encoded audio data, that is, AAC data, into the encoded audio area 24 f through the memory control circuit 22.

  In step S15, the variable K is set to “1”, and in step S17, it is determined whether or not the MP4 data of the 60 * K frame is completed. When the determination result is updated from NO to YES, the processing after step S19 is executed.

  In step S19, the latest 60 frames of MP4 data stored in the encoded image area 24d are read through the memory control circuit 22, and the read MP4 data is written into the open MP4 file via the I / F 42. In step S21, the latest one-second AAC data stored in the encoded audio area 24f is read through the memory control circuit 22, and the read AAC data is written to the open MP4 file via the I / F 42.

  In step S23, the latest 30 frames of 3GP data stored in the encoded image area 24e are read through the memory control circuit 22, and the read 3GP data is written to the open 3GP file via the I / F 42. In step S25, the latest one-second AAC data stored in the encoded audio area 24f is read through the memory control circuit 22, and the read AAC data is written into the open 3GP file via the I / F 42.

  In step S27, it is determined whether or not a recording end operation has been performed. If the determination result is NO, the variable K is incremented in step S29 and then the process returns to step S17. Therefore, the processes of steps S19 to S25 are executed every time 60 frames of MP4 data are obtained. When the determination result of step S27 is updated from NO to YES, the process proceeds to step S31.

  In step S31, the MPEG4 encoder 38 is stopped to end the MP4 encoding process. In step S33, the post-processing circuit 28 and the MPEG4 encoder 40 are stopped to end the 3GP encoding process. In step S35, the audio capturing process is started. Therefore, the AAC encoder 36 is stopped.

  In step S37, termination processing is executed. As a result, the MP4 data of less than 60 frames and the AAC data of less than 1 second remaining in the SDRAM 24 are written to the MP4 file, and the 3GP data of less than 30 frames and the AAC data of less than 1 second remaining in the SDRAM 24 are written to the 3GP file. . In steps S39 to S41, the recording medium 44 is accessed through the I / F 42, and the open MP4 file and 3GP file are closed. When the closing process is completed, the process returns to step S3.

  Referring to FIG. 11, in step S51, the focus, aperture amount, and exposure time are initialized. In step S53, it is determined whether or not the vertical synchronization signal Vsync has been generated. If the determination result is updated from NO to YES, an AE process is executed in step S55. As a result, the brightness of the through image is appropriately adjusted. In step S57, it is determined whether or not the AF activation condition is satisfied. If NO, the process directly returns to step S53. If YES, the AF process is performed in step S59, and then the process returns to step S53. As a result of the AF process, the focus lens 12 is disposed at the focal point, thereby improving the sharpness of the through image.

  As can be seen from the above description, the memory control circuit 22 periodically fetches the raw image data output from the preprocessing circuit 20 into the SDRAM 24. The post-processing circuit 26 converts the captured raw image data into 1080 / 60p format image data, and the MPEG4 encoder 38 creates MP4 data based on the converted image data. The post-processing circuit 28 converts the captured raw image data into 480 / 30p format image data, and the MPEG4 encoder 40 creates 3GP data based on the converted image data. Here, MP4 data and 3GP data are created in parallel with each other, and the quality of MP4 data is higher than the quality of 3GP data. The CPU 46 assigns the AAC data output from the AAC encoder 36 to each of the MP4 data and 3GP data thus created (S21, S25). The CPU 46 also starts AAC data allocation processing with reference to the progress of MP4 data creation processing (S17).

  Thus, MP4 data and 3GP data are created based on common raw image data that is periodically captured. Also, the MP4 data creation process and the 3GP data creation process are executed in parallel with each other. Furthermore, the quality of MP4 data is higher than the quality of 3GP data. Therefore, when attention is paid to the raw image data captured in a certain cycle, the corresponding 3GP data is completed when the MP4 data creation process based on the raw image data is completed. AAC data is assigned to each of MP4 data and 3GP data with reference to the progress of the MP4 data creation process. Thereby, it is possible to reduce a load when assigning common audio data to image data having different methods or qualities.

  In this embodiment, both the 1080 / 60p format image data and the 480 / 30p format image data are encoded by the MPEG4 format. However, either the 1080 / 60p format image data or the 480 / 30p format image data may be encoded by the JPEG format.

  In this embodiment, a digital video camera is assumed, but the present invention can also be applied to a stationary video recorder.

  Furthermore, in this embodiment, the AAC system is adopted as the audio data encoding system, but the audio data may be encoded by an encoding system other than the AAC system.

DESCRIPTION OF SYMBOLS 10 ... Digital video camera 16 ... Image sensor 26, 28 ... Post-processing circuit 38, 40 ... MPEG4 encoder 44 ... Recording medium 46 ... CPU

Claims (3)

Image capturing means for periodically capturing original image data;
First creation means for creating first image data corresponding to the first quality based on the original image data captured by the image capture means;
A process of creating second image data corresponding to a second quality lower than the first quality based on the original image data fetched by the image fetching means is executed in parallel with the creation process of the first creating means. Second preparation means,
Assigning means for assigning audio data to each of the first image data created by the first creating means and the second image data created by the second creating means;
Control means for activating the assigning means with reference to the progress of creation processing by the first creating means ; and
The first image data and the audio data assigned to the first image data are recorded as a first file, and the second image data and the audio data assigned to the second image data are different from the first file. A data processing apparatus comprising recording means for recording as a second file . Image capturing means for periodically capturing original image data;
First creation means for creating first image data corresponding to the first quality based on the original image data captured by the image capture means;
A process of creating second image data corresponding to a second quality lower than the first quality based on the original image data fetched by the image fetching means is executed in parallel with the creation process of the first creating means. Second preparation means,
Assigning means for assigning audio data to each of the first image data created by the first creating means and the second image data created by the second creating means;
Control means for activating the assigning means with reference to the progress of the creation processing by the first creating means,
The data processing apparatus according to claim 1, wherein the control means activates the assignment means every time the number of frames of the first image data created by the first creation means reaches a predetermined value. The first image data corresponds to stream data in which an intra-coded image and an inter-coded image are mixed,
The data processing apparatus according to claim 2, wherein the predetermined value corresponds to a cycle in which the intra-coded image appears.

Images