JP2005124049A - Device and method for processing image data - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten data fetching time of image data from a camera device and transmitting time of the image data to the other terminal. <P>SOLUTION: This device for processing image data is provided at least with a camera control part 1 and a camera interface part 2. The camera control part performs camera I/F control such as start, stop of fetching the image data from the camera device 10 and ON, OFF, stop, etc. of DMA transfer of the image data, instructs that a frame rate value to be integer multiples of a specified frame rate be set in the camera device within a range of the maximum value selective by the camera device, and calculates timing for thinning the image data by difference between the set frame rate and the specified frame rate. The camera interface part sets the setting contents and the control contents from the camera control part in the camera device, stores the image data fetched from the camera device and performs buffering for writing the image data in a memory 4 by the DMA transfer with timing other than thinning timing. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to image data processing, and more particularly to image data processing for processing image data captured by a camera device.

  In a videophone system that communicates using images and sounds, a general problem is that the difference between image data with a large amount of data and sound data with a small amount of data causes a shift between the image and sound on the receiving terminal side. A phenomenon called Lip Sync is known. Conventionally, in an image processing system for solving this type of problem, a value called a skew value is transmitted to the receiving terminal side on the transmitting terminal side, and the start of audio is delayed by the amount of the skew value on the receiving terminal side ( For example, refer nonpatent literature 1).

  Also, since image data has a large amount of information, the frame rate is adjusted according to the remaining capacity of the battery that supplies power to the portable information terminal device (for example, see Patent Document 1) or a camera. There is known a technique in which a frame rate is set according to the magnitude of a camera parameter threshold such as a subject distance at the time of shooting and a video signal is thinned out in accordance with the frame rate (for example, see Patent Document 2).

Japan Telegraph and Telephone Technical Committee: Control Protocol for Multimedia Communication, 8th Edition Established on May 30, 2002 (pp. 175-176) Japanese Patent Laid-Open No. 10-248063 (first page-second page, FIG. 1) JP-A-11-112844 (first page-second page, FIG. 2)

  However, in the conventional technique described in Non-Patent Document 1 described above, since the skew value is set unilaterally on the transmission terminal side, the range in which the image / sound deviation is reduced by the encoding / decoding capability on the reception terminal side. The problem remains that there is a difference.

  Furthermore, since the frame rate value required for the entire system is set for the camera device as it is, the performance of the camera device cannot be maximized, and it takes time to acquire the image data. As a result, the image data There is also a problem that the process from the acquisition to the receiving terminal side is not optimized.

  Further, the conventional technique described in Patent Document 1 described above merely adjusts the frame rate, while the conventional technique described in Patent Document 2 performs setting of the frame rate to the camera and thinning processing of the video signal according to the setting. It does not provide a solution to the Lip Sync phenomenon.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image data processing method that realizes speeding up of image capturing time and reduction of system load in order to solve the above problems.

  The image data processing apparatus according to claim 1 is an image data processing apparatus that processes image data captured by a camera device, and at least starts and stops capturing image data from the camera device, and turns on DMA transfer of the image data, Instructs the camera device to set the frame rate value that is an integral multiple of the specified frame rate within the range of the maximum value that can be selected by the camera device. In addition, a camera control unit (1 in FIG. 2) that calculates the timing for thinning out image data based on the difference between the set frame rate and the specified frame rate, and the setting content and control content from the camera control unit are stored in the camera device. Save the image data set and captured from the camera device, and save it with DMA transfer at a timing outside the thinning timing Characterized by (2 in Figure 2) and it was provided camera interface unit which performs buffering for writing to.

  The image data processing apparatus according to claim 2 is the image data processing apparatus according to claim 2, wherein the camera interface unit is a FIFO (2 in FIG. 2) that functions as a buffer for writing the image data captured from the camera device into the memory. -1) and the camera I / F register register (see Fig. 2), which is set by the camera controller whether the DMA mode is ON / OFF, VSYNC falling interrupt enabled / disabled, and whether the FIFO is cleared 2-2) and a counter (2-3 in FIG. 2) that generates timing for thinning out and writing the image data captured from the camera device into the memory from the FIFO. The image data written in the FIFO is When the DMA mode is ON, DMA transfer to the memory is possible and writing is possible. When the DMA mode is OFF, it is discarded by the FIFO clear setting and the VSYNC falling interrupt is set When the DMA mode is ON, it is “invalid”. On the other hand, when one frame of image data is written to the memory and becomes OFF, it becomes “valid” and enables the VSYNC falling interrupt to notify the camera controller. As a result, if the next time is the memory write timing, the camera control unit will set the DMA mode to ON, set the VSYNC falling interrupt to “invalid”, clear the FIFO and clear the counter value, and at other timings the counter value It is characterized by clearing.

  The image data processing method according to claim 3 is an image data processing method for processing image data captured by a camera device, and at least a frame rate value that is an integer multiple of a specified frame rate can be selected by the camera device. To instruct the camera device to set within the range of the maximum value, and to calculate the timing to thin out the image data based on the difference between the set frame rate and the specified frame rate, and the image captured from the camera device And a procedure for storing data, discarding it at a thinning-out timing, and writing it into a memory at a timing outside the thinning-out timing.

  The image data processing method according to claim 4 is the image data processing method according to claim 3, wherein a procedure for determining the ON / OFF state of the DMA mode capable of writing to the memory and a camera device if the DMA mode is ON If the DMA mode is OFF, the DMA mode is turned off, the VSYNC falling interrupt setting is set to “valid”, the counter that generates the write timing is incremented by 1 and the DMA mode is turned off Determine whether the VSYNC falling interrupt setting is “valid”, and if it is “valid”, determine whether the next is the write timing, and discard the image data captured from the camera device if the next is not the write timing When the next is the write timing, set the DMA mode to ON, set the VSYNC falling interrupt setting to `` Disable '', and clear the counter Characterized in that it.

  The first effect of the present invention is that a maximum value that is an integral multiple of the frame rate value required for the entire system is set in the camera device as a frame rate value within the maximum value that can be selected by the camera device. This is to maximize the performance of the camera device and to shorten the time from the start of the camera output until the actual acquisition of the camera image.

  The shortened time differs depending on the difference between the designated frame rate value and the maximum frame rate value that can be set in the camera device. In the above example, it takes 134 milliseconds at the frame rate of 7.5 fps until the first image data is acquired from the start of image data capture from the camera device, but it takes 34 milliseconds at 30 fps, so the first image Time until data is acquired is reduced to 1/4. As a result, the image data can be transmitted 100 milliseconds earlier. Also, if the specified frame rate value is 15 frames per second, it will be possible to reduce the time to start processing by approximately 33 milliseconds compared to the conventional case, and the problem of Lip Sync can be solved.

  By this shortening, the arrival of the image data is accelerated by the amount shortened to the receiving terminal side, and the decoding process of the image data received on the receiving terminal side can be started earlier. As a result, it is possible to reduce the gap between the image and the sound without being affected by the difference in encoding / decoding performance on the receiving terminal side.

  The second effect of the present invention is that the image data captured from the camera device is written into the memory only once for the number of acquisitions equal to the ratio of the frame rate value set in the camera device to the requested frame rate value. The other image data is discarded, that is, the CPU load is reduced by the thinning process, and the processing speed is increased.

  For example, if the required frame rate value is 7.5 frames per second (7.5 fps), and 30 frames per second (30 fps) is set as the frame rate value for the camera device, the ratio between 7.5 fps and 30 fps will be 4, and every 4 frames Image data is acquired from the camera device only once, and the other three frames are discarded.

  The image data processing apparatus of the present invention is an image data processing apparatus that processes image data captured by a camera device, and includes at least a camera control unit and a camera interface unit.

  The camera control unit performs camera I / F control such as starting / stopping image data capture from the camera device and turning on / off / stopping DMA transfer of the image data, and becomes an integer multiple of the specified frame rate. Instructs the camera device to set the frame rate value within the range of the maximum value selectable by the camera device, and calculates the timing to thin out the image data based on the difference between the set frame rate and the specified frame rate .

  The camera interface unit sets the settings and control details from the camera control unit to the camera device, saves the image data captured from the camera device, and writes it to the memory by DMA transfer at a timing outside the decimation timing I do. For this purpose, a FIFO, a camera I / F register register and a counter are provided.

  The FIFO functions as a buffer for writing image data captured from the camera device into the memory. The camera I / F register register is set by the camera control unit to determine whether the DMA mode is on / off, whether the VSYNC falling interrupt is enabled / disabled, and whether to clear the FIFO. The counter generates timing for thinning out and writing image data captured from the camera device from the FIFO to the memory.

  When the DMA mode is ON, the image data written to the FIFO is DMA-transferred to the memory and can be written.When the DMA mode is OFF, it is discarded by the FIFO clear setting, and the VSYNC falling interrupt is set. Is “invalid” when the DMA mode is ON, while it is “valid” when one frame of image data is written to the memory and becomes OFF, and a VSYNC falling interrupt can be sent to the camera controller As a result, if the next time is the memory write timing, the camera control unit will set the DMA mode to ON, set the VSYNC falling interrupt to “invalid”, set the FIFO to clear, and clear the counter value. Clear the value.

  The image data processing method of the present invention is an image data processing method for processing image data captured by a camera device, and at least a frame rate value that is an integer multiple of a specified frame rate can be selected by the camera device. Instructs the camera device to set within the range of values, calculates the timing to thin out the image data based on the difference between the set frame rate and the specified frame rate, and the image data captured from the camera device Storing, discarding at the thinning timing, and writing to the memory at a timing outside the thinning timing.

  More specifically, the procedure to determine the ON / OFF status of the DMA mode that can be written to the memory, and if the DMA mode is ON, the image data captured from the camera device is written to the memory by DMA transfer, and the DMA mode is OFF , Set the VSYNC falling interrupt setting to “valid”, add 1 to the counter that generates the write timing, and if the DMA mode is OFF, determine whether the VSYNC falling interrupt setting is “valid” ”If the next is the write timing, if the next is not the write timing, discard the image data captured from the camera device, if the next is the write timing, turn on the DMA mode, VSYNC And a procedure for clearing the counter by setting the falling interrupt setting to “invalid”.

  Next, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a configuration diagram showing an outline of an image processing unit in a videophone system, and includes a camera device 10 that captures image data, an image system driver 20, and a video DSP 30.

  The image system driver 20 includes a camera image storage unit 21 that performs processing for acquiring image data (YCbCr422 format) from the camera device 10, and an encoded data storage unit that stores image data encoded by a video DSP (Digital Signal Processor) 30. 22 and packet processing means 23 for packetizing the encoded data stored in the encoded data storage means 22 in accordance with the videophone protocol and sending it out to the outside (the partner terminal).

  The video DSP 30 performs a process of encoding the image data in the YCbCr422 format saved in the camera image storage means 21 into the MPEG4 format.

  FIG. 2 is a block diagram showing a detailed configuration of the main part of the camera image storage means 21. As shown in FIG. 2, the camera image storage unit 21 includes a camera control unit 1, a camera interface unit 2, a DMA transfer unit 3, and a memory 4.

  The camera control unit 1 performs camera I / F control such as setting of a frame rate to the camera device 10, starting and stopping of image data acquisition from the camera device 10, and ON / OFF / stop of DMA transfer of image data. This is done via the interface unit 2.

  Further, the camera control unit 1 instructs the camera device to set a frame rate value that is an integral multiple of the specified frame rate within the range of the maximum value that can be selected by the camera device, and the set frame rate And the timing for thinning out the image data based on the difference between the designated frame rate.

  The camera interface unit 2 sets the setting content and control content from the camera control unit 1 in the camera device 10. Further, the image data taken from the camera device 10 is saved, and buffering is performed for thinning out and writing to the memory 4. For such processing, the camera interface unit 2 includes a FIFO 2-1, a camera I / F register register 2-2, and a counter 2-3.

  The FIFO 2-1 functions as a buffer for writing image data captured from the camera device 10 into the memory 4.

  In the camera I / F register 2-2, the camera control unit 1 sets whether the DMA mode is ON / OFF, whether the VSYNC falling interrupt is enabled / disabled, and whether to clear the FIFO 2-1 or not.

  When the DMA mode is in the ON state, the image data written in the FIFO 2-1 can be transferred to the memory 4 and written by the DMA transfer unit 3. On the other hand, when the DMA mode is OFF, the image data written in the FIFO 2-1 is discarded. When one frame of image data is written to the memory 4, the DMA mode is set to the OFF state to deal with the thinning, and when the writing to the memory 4 is to be performed, the DMA mode needs to be set to the ON state. is there.

  Further, the VSYNC falling interrupt means that the camera interface unit 2 notifies the camera control unit 1 that one frame of image data has been written in the memory 4. The setting of the VSYNC falling interrupt means setting by the camera control unit 1 as to whether the VSYNC falling interrupt is permitted (valid) or not (invalid).

  The ON / OFF state of the DMA mode and the validity / invalidity of the VSYNC falling interrupt are set so as not to contradict each other due to the above-described significance. In other words, the VSYNC falling interrupt setting should be “invalid” when the DMA mode is ON, and the VSYNC falling interrupt setting should be “valid” when the DMA mode is OFF. It is.

  The counter 2-3 generates timing for thinning out and writing the image data captured from the camera device 10 from the FIFO 2-1 to the memory 4. In the above example, when the count value becomes 3 at 0 origin, the next timing for capturing image data from the camera device 10 is the writing timing. That is, image data captured from the camera device 10 at this time is image data to be written in the memory 4.

  The camera device 10 indicates a hardware camera sensor, reads and generates image data, and writes the image data in the buffer area of the FIFO 2-1 provided by the camera interface unit 2.

  The DMA transfer unit 3 acquires the image data written in the FIFO 2-1 and writes the image data in the area of the memory 4 designated by direct memory access. Further, the camera control unit 1 is notified of the completion of the DMA transfer of the image data by a DMA transfer completion interrupt.

  The memory 4 is a recording medium unit that temporarily stores the image data acquired from the DMA transfer unit 3, forms a camera image buffer, and the image data stored therein is encoded by the video DSP 30.

  Hereinafter, the operation of this embodiment will be described with reference to the flowchart of FIG.

  First, the user designates a frame rate value in the operation unit (not shown) of the videophone terminal and requests to start shooting (step S1 in FIG. 3). At this time, the camera control unit 1 can automatically set the frame rate value as long as the user simply requests to start shooting.

  The camera control unit 1 sets a frame rate value that is an integral multiple of the designated frame rate to the camera device 10 within the range of the maximum value that can be selected by the camera device 10 (step S2).

  Further, the camera control unit 1 calculates the timing for thinning out the image data based on the difference between the designated frame rate value and the frame rate value set in the camera device 10 (step S3). For example, when the designated frame rate value is 1 and the set frame rate value is 4, thinning out is 3 frames per 4 frames.

  Next, when the user requests start of image acquisition (step S4), camera I / F control including an instruction to start capturing image data in the route of camera control unit 1 → camera interface unit 2 → camera device 10 is performed by the camera device. 10, acquisition of image data from the camera device 10 is started. This acquisition is performed at high speed according to the frame rate value set in the camera device 10 (step S2). At this time, the DMA mode set in the camera I / F register 2-2 of the camera interface unit 2 is ON as an initial value.

  The image data captured from the camera device 10 to the camera interface unit 2 is written into the FIFO 2-1 (step S5).

  When the DMA mode of the camera I / F register 2-2 is ON (Yes in step S6), the DMA transfer unit 3 writes the image data written in the FIFO 2-1 to the designated area of the memory 4. Then, the DMA transfer completion is notified to the camera control unit 1 by a DMA transfer completion interrupt (step S7). Upon receiving the notification of the completion of DMA transfer, the camera control unit 1 turns off the DMA mode and enables the VSYNC falling interrupt for the camera I / F register 2-2 (step S8). The counter 2-3 is incremented by 1 (step S9).

  The processes in steps S5 to S9 are repeated unless a camera stop is requested (No in step S10). On the other hand, when the stop of photographing is requested (Yes in step S10), the camera control unit 1 ends the image data capturing process from the camera device 10 and performs the camera stop process (step S11).

  On the other hand, if the DMA mode is OFF (No in step S6), the camera interface unit 2 performs a VSYNC falling interrupt after confirming that the VSYNC falling interrupt setting is valid (Yes in step S12). (Step S13). Then, depending on the number of times image data is acquired by the counter 2-3 (step S14), the process branches to either step S15 and step S9 or steps S16 to S18. Note that if the DMA mode is OFF (No in step S6) and the VSYNC falling interrupt setting is not valid (No in step S12), the image data to the memory 4 is stored in spite of the thinning period. An error occurs because there is no writing.

  When the next timing is not the image data writing timing (No in step S14), the camera interface unit 2 sets clear of FIFO2-1 in the camera I / F register 2-2 (step S15) and checks the specified number of times. The value of the counter 2-3 is incremented (step S9).

  On the other hand, when the next timing is the image data write timing (Yes in step S14), the camera interface unit 2 turns on the DMA mode for the camera I / F register 2-2 and sets the VSYNC falling interrupt to “invalid”. At the same time (step S16), clearing of FIFO2-1 is set (step S17). Further, the value of the counter 2-3 for checking the specified number of times is cleared (step S18).

  If the DMA mode is OFF (No in step S6) but the VSYNC falling interrupt setting is invalid (No in step S11), an error occurs. This is because it contradicts the fact that the DMA mode is set to OFF and the VSYNC falling interrupt is enabled (step S8).

  FIG. 4 shows the camera I / F register 2-2 and the counter 2- when the image data is fetched from the camera device 10 four times and written into the memory 4 only once and the other three image data are discarded. The transition of the contents of 3 is shown in the time chart. Each of timings t1 to t5 indicates a timing at which image data is captured from the camera device 10.

  If the image data is DMA-transferred at the timing t1 (step S7 in FIG. 3), the DMA mode is set to the OFF state and the VSYNC rising interrupt setting is set to “valid” (step S8). Since the counter 2-3 is cleared to 0 at the previous timing (step S9), 1 is added to 1 (step S9).

  Since the value of the counter 2-3 is 1 at the timing t2 and the value of the counter 2-3 is 2 at the timing t3, the next is not the writing timing of the image data to the memory 4 (No in step S14), and the DMA mode is The setting of the OFF state and VSYNC rising interrupt is kept “valid”, the FIFO2-1 is cleared, and the image data captured from the camera device 10 is simply discarded (step S15).

  At timing t4, since the counter value is 3, next is the timing of writing image data to the memory 4 (Yes in step S14), the DMA mode is ON, and the VSYNC rising interrupt setting is “ “Invalid” is set (step S16). This prepares for the writing of the image data to the memory 4 at the next timing t5. Also, clearing of FIFO2-1 is set, the image data captured from the camera device 10 is discarded (step S17), the counter 2-3 is cleared to 0 (step S18), and the decimation count is started again from here. To start. Thereafter, processing similar to that at timings t1 to t4 is repeated from timing t5 until a stop request is issued (Yes in step S10).

  Note that the above description is based on an example of a videophone system, but the present invention is not limited to this. For example, a moving image shooting function that converts image data and audio data into a file in MPEG4 format and provides it as MPEG4 data. The present invention can be similarly applied to the camera image acquisition process in the system including the above, and an effect equivalent to that in the videophone system can be obtained.

Block diagram showing an outline of the configuration of the image processing unit in the videophone system The block diagram which shows the detailed structure of the principal part of the camera image storage means 21 Flow chart showing processing of the present invention Time chart of processing example of the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera control part 2 Camera interface part 3 DMA transfer part 4 Memory 10 Camera device 20 Image system driver 21 Camera image storage means 22 Encoding data storage means 23 Packet processing means 30 Video DSP
2-1 FIFO
2-2 Camera I / F register
2-3 Counter

Claims (4)

In an image data processing apparatus that processes image data captured by a camera device, at least,
Performs camera I / F control such as starting / stopping image data acquisition from the camera device, turning on / off / stopping DMA transfer of the image data, and setting the frame rate value that is an integral multiple of the specified frame rate to the camera A camera control unit that sets the camera device within a range of a maximum value that can be selected by the device, and calculates a timing at which the image data is thinned out according to a difference between the set frame rate and the specified frame rate;
A camera that sets the setting contents and control contents from the camera control unit to the camera device, stores image data captured from the camera device, and performs buffering for writing to the memory by DMA transfer at a timing outside the thinning timing An image data processing apparatus comprising an interface unit. The camera interface unit
FIFO that functions as a buffer for writing image data captured from the camera device to the memory,
The camera I / F register register that is set by the camera control unit according to whether the DMA mode is ON / OFF, VSYNC falling interrupt valid / invalid, and whether to clear the FIFO,
A counter for generating timing for thinning out and writing image data captured from the camera device from the FIFO to a memory;
When the DMA mode is in the ON state, the image data written in the FIFO is DMA transferred to the memory and can be written, and when the DMA mode is in the OFF state, it is discarded by the clear setting of the FIFO.
The VSYNC falling interrupt setting is “invalid” when the DMA mode is in the ON state, and becomes “valid” when one frame of image data is written in the memory and becomes the OFF state. Enable VSYNC falling interrupt to notify
As a result, the camera control unit clears the counter value by setting the DMA mode to the ON state, VSYNC falling interrupt is “invalid”, and clearing the FIFO if the next time is the memory write timing. The image data processing apparatus according to claim 1, wherein the value of is cleared. An image data processing method for processing image data captured by a camera device, at least,
A frame rate value that is an integral multiple of the specified frame rate is set in the camera device within the range of the maximum value selectable by the camera device, and the difference between the set frame rate and the specified frame rate A procedure for calculating a timing for thinning out the image data;
An image data processing method comprising: storing image data captured from a camera device, discarding the image data at the thinning timing, and writing the data to a memory at a timing outside the thinning timing. A procedure for determining the ON / OFF state of the DMA mode in which writing to the memory is possible,
If the DMA mode is ON, the image data captured from the camera device is written to the memory by DMA transfer, the DMA mode is OFF, the VSYNC falling interrupt setting is “valid”, and 1 is added to the counter that generates the write timing. Procedure and
If the DMA mode is OFF, determine whether the VSYNC falling interrupt setting is `` valid '', and if it is `` valid '', determine whether the next is the write timing,
When the next is not the write timing, the procedure to discard the image data captured from the camera device,
4. The image data according to claim 3, further comprising: a step of clearing the counter by setting the DMA mode to an ON state and setting the VSYNC falling interrupt to “invalid” when the next is a write timing. Processing method.

Images