KR100673955B1 - Method to capture moving images from video presenter with audio - Google Patents

Abstract

A method for capturing a moving image from a video presenter with audio data is provided to allow a computer to capture a moving image from a video presenter together with audio data from a microphone connected to the computer on lip-synchronization level. Video frame data is received from a video presenter. Input audio data is received from an audio input terminal while the video frame data is received. The time when the video frame data is received is recorded. The operation of receiving video frame data and audio data and recording the video frame data receiving time is repeated until the time when capturing of video frame data and audio data is finished. At the time when capturing of video frame data and audio data is finished, audio data corresponding to video frame data is detected according to a difference between the video frame data input time and the audio data input time, and one of the video frame data and the audio data is controlled so that the video frame data receiving time corresponds to the audio data receiving time.

Description

How to capture moving images from video presenter with audio}

1 is a perspective view showing a video presenter and a computer executing a moving image capture program according to the present invention.

FIG. 2 is a block diagram illustrating an internal configuration of the video presenter of FIG. 1.

3 is a flow chart showing a main program executed by the computer of FIG. 1 for use of a video presenter.

4 is a flowchart illustrating an algorithm of the video capturing step of FIG. 3.

FIG. 5 is a flowchart showing step S81a of FIG. 4 in detail.

FIG. 6 is a flowchart showing step S81b of FIG. 4 in detail.

FIG. 7 is a flowchart showing step S83a of FIG. 4 in detail.

FIG. 8 is a flowchart showing step S83b of FIG. 4 in detail.

FIG. 9 is a diagram for explaining steps S85 and S86 of FIG. 4.

<Explanation of symbols for the main parts of the drawings>

1 ... video presenter, 15 ... video detector,

13a, 13b ... lighting device, 16 ...

18 ... lock button, 3 ... subject,

11 ... with a basin, 12 ... key input device,

14 ... remote receiver, 7 ... mouse,

5 ... computer, 6 ... keyboard,

2 ... monitor, S ... screen,

21 Paint, 3a ...

P1, P2 ... picture, 22 ... pointer,

9 ... microphone,

15a ... optical system, 15b ... photoelectric conversion unit,

101 microprocessor, 102 timing circuit,

103 ... analog signal processing section, 104 ... analog-to-digital conversion section,

105 ... digital camera processor, 106 ... SDRAM,

107 frame-specification section, 108 video output section,

109 ... USB interface.

The present invention relates to a moving image capture method, and more particularly, to a moving image capture method in which a computer captures a moving image from a video presenter together with audio.

An example of a typical video presenter is the video presenter of US Pat. No. 5,822,013, filed and registered by the applicant. Such a video presenter provides image data to a computer while performing serial communication with a computer running its data processing program. Accordingly, the computer processes the image data from the video presenter to display the moving image and to capture the still image when the still image capture signal is generated.

Meanwhile, a protocol of high speed serial communication, for example, a protocol of "USB (Universal Serial Bus) 2.0" capable of data transmission of 480 Mbps (Mega bits per second), is used between a computer and a video presenter. Accordingly, the video presenter can transmit a video to a computer at high speed. For example, the video presenter may transmit image data having a resolution of the extended graphics array (XGA) standard, for example, 1,024 × 768 pixels, to a computer at a rate of 20 frames per second (FPS). As a result, the video from the video presenter can not only be displayed on the monitor of the computer but also captured in the video file.

Here, if video from a video presenter can be captured with audio, the value of utilizing the video presenter as a lecture or a meeting will increase. However, when an audio processing device such as an audio coder (Coder and Decoder) is added to the video presenter, not only the manufacturing cost of the video presenter increases but also the video presenter and the computer smoothly multiplex the audio / video data from the video presenter. There are problems that must have the ability to send and receive easily.

Thus, if a computer can capture audio from a microphone connected to it at the level of lip-synchronization with a video from a video presenter, the following effects will occur.

First, since an audio processing device such as an audio coder and a coder need not be added to the video presenter, the manufacturing cost of the video presenter does not increase.

Second, the video presenter and the computer need not have the capability to smoothly send and receive multiplexed audio / video data from the video presenter.

However, there is a difference in time point at which the image frame data from the video presenter and the input audio data from the audio input terminal of the computer are input to the computer, and the reception time of each image frame data changes according to the transmission situation. Thus, it is difficult for a computer to capture audio from a microphone connected to it at the level of lip-synchronization with video from a video presenter.

An object of the present invention is that there is a difference in time point at which image frame data from a video presenter and input audio data from an audio input terminal of a computer are input to the computer, and the reception time of each image frame data varies according to a transmission situation. Nevertheless, it provides a video capture method that allows a computer to capture audio from a microphone connected to it at the level of lip-synchronization along with video from a video presenter.

The present invention for achieving the above object is a moving picture capture method in which a computer captures a moving picture from a video presenter with audio, comprising steps (a) to (e). In step (a), image frame data from the video presenter is received. In step (b), input audio data from an audio input terminal is received during the time when the image frame data is received. In step (c), the reception time of the image frame data in step (a) is recorded. In the step (d), the steps (a) to (c) are repeatedly performed until the capturing end point. In the step (e), when the capturing end point is reached, the respective audio data corresponding to the respective video frame data is found according to the difference in the input time points of the video frame data and the input audio data, and the reception time of each video frame data is found. Any one of each of the image frame data and the respective audio data is adjusted so that the reception time of the audio data corresponding to each of the image frame data is equal to each other.

According to the moving picture capturing method of the present invention, there is a difference in time point at which video frame data from a video presenter and input audio data from an audio input terminal of a computer are input to the computer, Despite the change in reception time, the computer can capture audio from the microphone connected to it at the level of lip-synchronization along with video from the video presenter. Accordingly, the following effects can be obtained.

First, since an audio processing device such as an audio coder and a coder need not be added to the video presenter, the manufacturing cost of the video presenter does not increase.

Second, the video presenter and the computer need not have the capability to smoothly send and receive multiplexed audio / video data from the video presenter.

Hereinafter, preferred embodiments according to the present invention will be described in detail.

Referring to FIG. 1, the video presenter 1 according to the present invention includes an image sensing unit 15, lighting devices 13a and 13b, a strut 16, a lock button 18, a subject 11, and a key input. Device 12, and remote receiving device 14.

The image sensing unit 15 that can move back and forth and rotates is provided with an optical system and a photoelectric conversion unit. An optical system for optically processing light from a subject includes a lens unit and a filter unit. A photoelectric conversion unit of a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) converts light incident from an object through an optical system into an electrical analog signal.

The user can move the strut 16 by pressing the lock button 18. Underneath the cover 11 is another lighting device. The key input device 12 is used to control the driving of each unit such as the image sensing unit 15 and the lighting devices 13a and 13b by a user's manipulation. On the other hand, the user operates a remote transmission device (not shown) to input a control signal to the remote reception device 14, thereby controlling the driving of each unit such as the image sensing unit 15 and the lighting devices 13a and 13b. Can be.

The computer 5 executing the dedicated program of the video presenter 1 including the video capturing program according to the present invention processes the video data from the video presenter 1 and displays the video on the display screen S of the monitor 2. The still image is captured when the still image capturing signal from the user is generated, and the video is captured when the video capturing signal from the user is generated. To this end, the main control unit of the video presenter 1 is a high-speed serial communication protocol, and the computer 5 is connected via an interface of "USB (Universal Serial Bus) 2.0" protocol capable of data transmission of 480 Mbps (Mega bits per second). Communicate with Accordingly, the video presenter 1 rapidly transmits image data having a resolution of 1,024 X 768 pixels as a resolution of an extended graphics array (XGA) standard to a computer at a rate of 20 frames per second (FPS).

The computer 5 receives and processes the image data from the video presenter 1 to display the moving image on the display screen S of the monitor 2. Accordingly, the video of the subject 3 on the subject 11 is displayed on the display screen S of the monitor 2. The computer 5 also captures a still image from the video presenter 1 in accordance with a still image capture signal from the user. In addition, the computer 5 captures the moving picture from the video presenter 1 in accordance with the moving picture capture signal from the user (see Fig. 3).

In capturing the video, the computer 5 performs the algorithm to be described with reference to FIGS. 4 to 9 to thereby lip-sync the audio from the microphone 9 connected to it with the video from the video presenter 1. can be captured at the level of (lip-synchronization).

On the other hand, the user can edit still images and moving images from the video presenter 1. For example, the paint plate 21 is displayed on the display screen S of the monitor 2, and the user uses the mouse 7, the keyboard 6, and the paint plate 21 to draw on the subject image 3a. Since the fields P1 and P2 may be overlapped and drawn, the user may perform various descriptions. In FIG. 1, reference numeral 22 denotes a pointer of the mouse 7.

Of course, the output image data from the video presenter 1 may be directly input to the monitor 2.

Referring to FIG. 2, the video presenter 1 of FIG. 1 according to the present invention includes a key input device 12, a remote receiving device 14, a universal serial bus (USB) interface 109, an optical system 15a, and a photoelectric device. Converter 15b, analog signal processor 103, analog-digital converter 104, digital camera processor 105, timing circuit 102, microprocessor 101 as the main controller, SDRAM 106 as frame memory , A synchronous dynamic random access memory), a memory controller 107, and an image output unit 108. In FIG. 2, the same reference numerals as used in FIG. 1 indicate objects of the same function.

The optical system 15a optically processes light from a subject. The photoelectric conversion section 15b of the CCD or CMOS converts the light from the optical system 15a into an electrical analog signal. The timing circuit 102, which operates under the control of the microprocessor 101 as the main controller, for example, the timing generator element controls the operation of the photoelectric conversion section 15b. The analog signal processing unit 103, for example, a CDS-AGC (Correlation Double Sampler and Automatic Gain Controller) element processes an analog signal from the photoelectric conversion unit 15b to remove the high frequency noise and adjust the amplitude. The analog-digital converter 104 converts the analog signal from the analog signal processor 103 into digital signals of red (R), green (G), and blue (B). The digital camera processor 105 processes the digital signal from the analog-to-digital converter 104 to generate image data of " Y: Cb: Cr 4: 2: 2 " format, which is well known as a luminance and chromaticity format.

The SDRAM 106 as a frame memory stores image data from the digital camera processor 105 in units of frames. The memory control unit 107 composed of a Field Programmable Gate Array (FPGA) provides the frame data from the SDRAM 106 to the image output unit, and provides the frame data to the microprocessor 101 under the control of the microprocessor 101. Optionally enter Accordingly, the microprocessor 101 communicates with the computer (5 of FIG. 1) through the USB interface 109, and receives frame data from the memory controller 107 in response to a request from the computer 5. 5) to send.

The video output unit 108, for example, a video graphics array (VGA) engine element converts the video data from the memory control unit 107 into an analog composite video signal and outputs it. When the video presenter (1 in FIG. 1) is directly connected to the monitor 2, the analog composite video signal from the image output unit 108 is directly input to the monitor 2. On the other hand, the microprocessor 101 controls the operations of the respective parts, for example, the timing circuit 102 and the digital camera processor 105 according to the signals input from the key input device 12 and the remote receiving device 14. .

1 to 3, a main program executed by the computer of FIG. 1 for use of the video presenter 1 will be described.

First, it is checked whether the USB interface 109 of the video presenter 1 and the USB interface (not shown) of the computer 5 are well connected (step S1). If it is not well connected, the guidance message accordingly is displayed on the monitor 2 (step S2). If well connected, the following data processing steps are performed.

First, initialization for USB communication with the video presenter 1 is performed (step S3). Next, while USB communication with the video presenter 1 is performed, data of consecutive frames from the video presenter 1 is processed to display a moving image of the subject 3 (step S4).

If a still image capture signal is generated from the user while displaying the moving image as described above (step S5), the data of a single frame from the video presenter 1 is processed to capture the still image (step S6).

Further, if a moving picture capture signal is generated from the user while displaying the moving picture as described above (step S7), data of consecutive frames from the video presenter 1 is processed to capture the moving picture (step S8). The algorithm of this video capture step S8 will be described in detail with reference to Figs.

The steps S4 to S8 are repeatedly performed until the end signal is input (step S9). That is, while the still image capturing step S6 is not performed, the moving image display step S4 or the moving image capturing step S8 is repeatedly performed, and the moving images are displayed by successively displaying the frames processed in each moving image display step. .

4 shows an algorithm of the video capture step S85 of FIG. FIG. 9 is a diagram for explaining steps S85 and S86 of FIG. 4. 1, 4, and 9, the algorithm of the video capture step S8 of FIG. 3 will be described.

First, the central processing element of the computer 5 receives the image frame data from the video presenter 2 (step S81a). Further, the central processing element of the computer 5 receives input audio data from its audio input terminal during the time when image frame data is received (step S81b).

Next, the central processing element of the computer 5 records the reception time of the image frame data in step S81a (step S82). Here, the central processing element of the computer 5 may use the internal timer to measure and record the reception time of the image frame data. However, in order to reduce the load on the central processing element of the computer 5, the microprocessor 101 of the video presenter 2 measures the transmission time of the image frame data using an internal timer, and the data of this transmission time is framed. In addition to the end of the data, it is preferable to transmit the data of the reception time to the central processing element of the computer 5.

Next, the central processing element of the computer 5 processes the received image frame data and stores it in the moving image file (step S83a), while processing the received audio data and stores it in the audio file (step S83b).

The steps S81a to S83b are repeatedly performed until the capturing end point is reached, for example, until the set capturing time has elapsed (step S84). Accordingly, each image frame data and each audio data are stored during the reception time of each image frame data.

As described above, in the respective video frame data V1 to Vn and the respective audio data A1 to An stored in the video file and the audio file during the recording time, that is, the video capturing time, the video frame from the video presenter 1. There is a difference in time point at which data and input audio data from the audio input terminal of the computer 5 are input to the computer 5.

In the case of this embodiment, in performing the steps S81a to S84, after the first audio data A1 is received in the step S81b, the steps S82 to S84 are performed, and the second audio data ( While A2) was input, the first image frame data V1 was input in step S81a. Therefore, in the case of the present embodiment, the time difference between the image frame data from the video presenter 1 and the input audio data from the audio input terminal of the computer 5 is input to the computer is the time of one frame (see FIG. 9). .

Here, if the reception time of each image frame data V1 to Vn does not change, the respective audio data corresponding to each image frame data is found and multiplexed according to the difference in the input viewpoint of the image frame data and the input audio data. Audio from microphone 9 connected to computer 5 may be captured at the level of lip-synchronization along with video from video presenter 15.

However, the reception time of each image frame data changes depending on the transmission situation (see FIG. 9). Accordingly, the central processing element of the computer 5 finds the respective audio data A1 to An corresponding to the respective image frame data V1 to Vn according to the difference in the input viewpoints of the image frame data and the input audio data. Each image frame data V1 to Vn is adjusted so that the reception time of each image frame data V1 to Vn is equal to the reception time of each audio data A1 to An corresponding to each image frame data. (Step S85).

For example, referring to FIG. 9, since the reception time of V1 is longer than the reception time of A1, the data of V1 is reduced. This reduction technique uses the well-known frame-rate down conversion. Next, since the reception time of V2 is shorter than the reception time of A2, the data of V2 is expanded. This extension technique uses the well-known frame-rate up conversion.

Of course, instead of adjusting the respective image frame data V1 to Vn according to the user's selection, the respective audio data A1 to An may be adjusted.

Accordingly, there is a difference in time point at which the image frame data from the video presenter 1 and the input audio data from the audio input terminal of the computer 5 are input to the computer 5, and each image frame data according to the transmission situation. Despite the change in the reception time of, the computer 5 can capture audio from the microphone 9 connected to it at the level of lip-synchronization with the video from the video presenter 1. .

In step S86, the central processing element of the computer 5 multiplexes each of the adjusted image frame data V1 to Vn and the respective audio data A1 to An corresponding thereto. Further, in step S87, the central processing element of the computer 5 stores the multiplexed audio / video data in the generated audio / video file.

1, 2, and 5, the following steps are performed in step S81a of FIG.

First, the central processing element of the computer 5 requests the image frame data from the microprocessor 101 of the video presenter 1 (step S81a1). Accordingly, the image frame data from the video presenter 1 is input to the computer 5, and the central processing element of the computer 5 stores the input image data in the frame buffer until the reception of the image frame data is completed. (Steps S81a2 and S81a3).

1, 2, and 6, the following steps are performed in step S81b of FIG.

First, the central processing element of the computer 5 enables the input end of the audio buffer at the time when the input of the image frame data of step S81a2 starts (steps S81b1 and S81b2). Accordingly, the input audio data is stored in the audio buffer.

Next, the central processing element of the computer 5 disables the input terminal of the audio buffer at the point when the input of the image frame data of step S81a3 is finished (steps S81b3 and S81b4). As a result, the storage of the input audio data is stopped.

1, 2, and 7, the following steps are performed in step S83a of FIG. 4.

First, the central processing element of the computer 5 converts the image frame data of the "Y: Cb: Cr 4: 2: 2" format stored in the frame buffer into the image frame data of the RGB (Red-Green-Blue) 24 format. Are converted to (step S83a1).

Next, the central processing element of the computer 5 carries out the image frame data of the RGB (Red-Green-Blue) 24 format so that it can be used in the GDI (Graphic Device Interface) of the operating system (OS) of the computer 5. The video frame data is converted into DIB (Device Independent Bitmap) format (step S83a2). Then, the central processing element of the computer 5 outputs the image frame data of the device independent bitmap (DIB) format to the graphic device interface (GDI) (step S83a3). Accordingly, image frame data from the video presenter 1 is displayed by the operating system (OS) of the computer 5.

On the other hand, the central processing element of the computer 5 performs the steps S83a2 and S83a3 and simultaneously compresses the image frame data in the RGB (Red-Green-Blue) format according to the user's selection (steps S83a4 and S83a5), the compressed or uncompressed image frame data is stored in the generated moving image file (steps S83a6).

1 and 8, in step S83b of FIG. 4, the central processing element of the computer 5 compresses the audio data stored in the audio buffer (step S83a5), and generates the compressed audio data. In step S83b.

As described above, according to the video capturing method according to the present invention, there is a difference in time point at which the image frame data from the video presenter and the input audio data from the audio input terminal of the computer are input to the computer. Although the reception time of each image frame data changes, the computer can capture audio from a microphone connected to it at a level of lip-synchronization along with a video from a video presenter. Accordingly, the following effects can be obtained.

First, since an audio processing device such as an audio coder and a coder need not be added to the video presenter, the manufacturing cost of the video presenter does not increase.

Second, the video presenter and the computer need not have the capability to smoothly send and receive multiplexed audio / video data from the video presenter.

The present invention is not limited to the above embodiments, but may be modified and improved by those skilled in the art within the spirit and scope of the invention as defined in the claims.

Claims (7)

In a video capturing method in which a computer captures a video from a video presenter with audio, (a) receiving image frame data from the video presenter; (b) receiving input audio data from an audio input terminal during the time when the image frame data is received; (c) recording a reception time of the image frame data in step (a); (d) repeating steps (a) to (c) until the acquisition end point; And (e) When the capturing end point is reached, the respective audio data corresponding to the respective video frame data is found according to the difference in the input viewpoint of the image frame data and the input audio data, and the reception time of each image frame data and each image frame are found. And adjusting one of each of the image frame data and each of the audio data such that the reception time of each of the audio data corresponding to the data is the same. In a video capturing method in which a computer captures a video from a video presenter with audio, (a) receiving image frame data from the video presenter; (b) receiving input audio data from an audio input terminal during the time when the image frame data is received; (c) recording the reception time of the image frame data in step (a); (d) processing the received image frame data and storing it in a moving image file; (e) processing the received audio data and storing it in an audio file; (f) repeating steps (a) to (e) until the acquisition end point; And (g) When the capturing end point is reached, the respective audio data corresponding to each image frame data is found according to the difference between the input time points of the image frame data and the input audio data, and the reception time and each image frame of each image frame data are found. Adjusting any one of each image frame data and each audio data such that the reception time of each audio data corresponding to the data is the same. The method of claim 2, (h) multiplexing each image frame data and the respective audio data corresponding thereto; And (i) storing the multiplexed audio / video data in an audio / video file. The method of claim 2, wherein step (a) comprises (a1) requesting image frame data from the video presenter; And (a2) storing the input image data in a frame buffer until the reception of the image frame data is completed. The method of claim 4, wherein in step (d), A moving picture capturing method in which frame data of a red-green-blue (RGB) format is compressed according to a user's selection and stored in the moving picture file. The method of claim 5, wherein in step (d), The image data of the RGB (Red-Green-Blue) format is converted into frame data of the Device Independent Bitmap (DIB) format, And a frame data in the device independent bitmap (DIB) format is output to a graphics device interface (GDI). The method of claim 2, wherein step (b) (b1) enabling an input of an audio buffer at the time when the input of the image frame data starts; And and (b2) disabling the input terminal of the audio buffer when the input of the image frame data ends.

Images