JP2007329681A - Video data transmitting device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video data transmitting device which has small processing delay of communication protocol processing, achieves high throughput, and flexibly adapts itself to various network protocols that applications requests. <P>SOLUTION: Disclosed is a system which has a plurality of network protocol processing blocks 101 adapted to stream formats and transmission formats; and one of the network protocol processing blocks is protocol-processed by software executed by a CPU and another one comprises a dedicated block which divides stream data output from an encoder into transmission packets, automatically adds a packet header of a previously set streaming protocol to generate a transmission packet in real time, and transmits the transmission packet. In this constitution, a control processing section for timing wherein packets from the respective processing blocks are sent out to a MAC of a network is provided to actualize efficient transmission processing for a video stream. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention is a system for distributing large-capacity video data in response to requests from a plurality of terminals connected to a network, and in particular, can simultaneously distribute video from a camera connected to a network in a plurality of formats. The present invention relates to a possible video data transmission apparatus.

  In recent years, the spread of networks such as the Internet has progressed, and the transmission band has been expanded. Accordingly, devices and applications such as network cameras that transmit a large amount of video data using a network are increasing.

  For example, in a network camera device that monitors and shoots a moving image from a remote location with a camera device connected to a network, first, the network camera uses a compression encoding process to encode the video stream data in the bit stream format. Has been converted. And it was transmitted on the network as an IP / UDP / RTP packet. In the network transmission processing of these video stream data, packet processing divided into a plurality of layers is performed as represented by the OSI reference model or the four-layer model defined by TCP / IP. It is common to exchange data. For example, in the case of streaming video streams, IP packets are generated on the Internet layer, UDP packets are generated on the transport layer, and RTP packets are generated on the application layer in the TCP / IP four-layer model, which are sent to the network as multiplexed packets. The

  On the other hand, demands for higher image quality and higher functionality of captured images distributed from network cameras are increasing year by year. For this reason, the number of simultaneous delivery of a plurality of streams is increasing in order to respond to an increase in the amount of data accompanying an increase in the bit rate of video stream data or simultaneous requests from a plurality of terminals. In such a situation, network protocol processing in data transmission must be performed at high speed, and techniques aimed at solving these requirements are described in Patent Documents 1 and 2, for example. In these techniques, the following processing is performed. That is, the encoding processing performance is improved by providing a plurality of encoders in parallel to simultaneously generate a plurality of types of video streams. Also, stream transmission throughput is improved by automatically adding a packet header to the transmission stream and simultaneously executing a plurality of protocol processing modules.

JP 2000-59463 A (first page, FIG. 2) JP 2001-45025 A (first page, FIG. 1)

  However, in a network camera in which a request for distributing video bitstreams of various formats is made from a plurality of terminals, as proposed in Patent Document 2, the circuit scale of the apparatus is large when a configuration including a plurality of encoders is provided. There is a drawback of becoming.

  Further, in the case where the packet stream is automatically added to the video stream data output from the encoder and transmitted, it is possible to cope with a predetermined protocol such as UDP / RTP. However, in this case, it is difficult to realize extensibility such as correspondence to an upper protocol by an application.

  Further, as proposed in Patent Document 1, in a configuration provided with a protocol dedicated processing means for automatically adding a packet header to data generated by an application executed by a CPU, protocol processing is performed. Can be executed at high speed. However, since the payload data passes through the buffer on the system memory through the CPU system bus, high-speed memory and broadband system bus performance are required. As a result, in an inexpensive system configuration, the data transmission throughput is reduced.

  The present invention has been made to solve the above problems, and in a video data transmission apparatus that transmits video stream data compressed and encoded in a plurality of formats simultaneously to a plurality of terminals connected to a network. An encoder capable of generating compression-coded data of a plurality of formats for uncompressed video data, first network protocol processing means executed by software by a processor, and in particular real-time transmission processing of image data Second network protocol processing means to execute, first transmission packet data generated by the first network protocol processing means and second transmission packet data generated by the second network protocol processing means The first transmission packet and the second transmission packet. A transmission packet to provide a video data transmitting apparatus or the like, characterized in that it comprises a transmission timing switching means for performing control so as to selectively output to the MAC.

  According to the present invention, priority is given to each protocol processing block, and control is performed such that packet data generated by a processing block with a high priority is transmitted preferentially, thereby reducing processing delay and high throughput. Transmission can be realized. In addition, it is possible to perform video distribution processing that can flexibly support various network protocols required by applications.

(First embodiment)
FIG. 1 is a diagram showing an example of the overall configuration of the present invention.
According to the present invention, a protocol processing unit for performing general-purpose communication and a protocol processing unit capable of real-time streaming transmission are configured as different forms.

  In FIG. 1, the former general-purpose protocol processing unit is a software communication protocol stack executed by a CPU 101. On the other hand, the latter real-time streaming protocol processing unit is shown as a protocol processing unit 102 dedicated to real-time transmission of image data. The overall configuration of the embodiment of the present invention will be described below with reference to FIG.

Reference numeral 103 denotes an imaging unit. The imaging unit 103 includes a lens, a CCD (photoelectric conversion element), a CCD control unit, and the like. In the imaging unit 103, imaging projected through the lens is converted into an analog electrical signal by the CCD.
In addition, the image processing unit includes an image processing unit that performs processing such as noise removal from the captured image converted into the analog electric signal, and performs A / D conversion for conversion into digital data.

Reference numeral 104 denotes an encoder that performs encoding (compression encoding) of uncompressed digital image data. The imaging unit 103 outputs uncompressed digital image data at a constant cycle, and the encoder 104 encodes the image data to generate a moving image data stream.
The encoder 104 in this embodiment is a hardware device that realizes high-speed encoding processing of image data, and corresponds to a plurality of encoding formats. The encoding format includes JPEG and MPEG4.
In addition, it is possible to set and encode a plurality of encoding conditions having different encoding formats and compression efficiencies within the output period from the imaging unit 103, and generate a plurality of moving image streams.

Setting and control of the encoder 104 is performed by an input switching control unit 105.
For example, the embodiment is realized by a microprocessor including a local memory device.
The input switching control unit 105 sets parameters such as an encoding format and compression efficiency for the encoder 104 for each image frame data output from the imaging unit 103. Also, encoder control is performed such as instructing the encoding start timing and receiving an interrupt signal notifying completion of encoding.

  The image data from the encoder 104 is output to either of the double buffer configuration of the image buffer 1 106 and the image buffer 2 107. The image buffer 1 and the image buffer 2 may be disposed on the same memory device such as a DRAM, or may be implemented by separate dedicated memory devices.

The input switching control unit 105 controls the data path selector 108 of the image data output from the encoder 104 to determine which image buffer to output.
For example, when the image buffer is on the DRAM and the output destination can be changed for each image data by setting the address of the image buffer in the encoder 104, the input switching control unit 105 controls the encoder 104. You may implement | achieve by the method of setting an output destination.

A data transfer switching unit 109 has a function of transferring data from one of the image buffers 106 and 107 to the protocol processing unit 102.
This function is used by providing a DMA device in the data transfer switching unit 109. Further, the CPU 101 has a data access function that enables the image buffer to be read with a memory address.

This function has an address decoder function in the data transfer switching unit 109, and data access from the CPU 101 to the image buffer can be accessed with the same memory address regardless of whether the image buffer is 106 or 107. It is trying to become.
The data transfer switching unit 109 transfers the image data from one of the image buffers to the protocol processing unit 102 when transmitting the image data to the network in real time. Further, when image data is transmitted to the network by the general-purpose communication protocol of the CPU 101, the CPU 102 is controlled so that the data in the image buffer can be read.

The data transfer switching unit 109 is controlled by the input switching control unit 105. That is, the input switching control unit 105 determines whether the image data output to the image buffer 1 (106) or the image buffer 2 (107) is data transmitted by the CPU 101 or data transmitted by the protocol processing unit 102. The data transfer switching unit 109 is controlled accordingly.
If the data is to be transmitted by the CPU 101, the CPU 101 is notified of the completion of image data preparation.
If the data is transmitted by the protocol processing unit 102, the protocol processing unit 102 is notified of the completion of image data preparation and checks whether the protocol processing unit 102 is ready to accept data transfer. Thereafter, the data transfer switching unit 109 is started to transfer image data.

The CPU 101 executes application software that transmits and receives image data. For example, a monitoring camera can be considered as an application form of the present invention. In this embodiment, it is conceivable that the CPU 101 operates a captured video distribution server program.
The CPU 101 also has protocol processing software that enables general-purpose TCP / IP communication, and transmits and receives TCP / IP packets.

In this embodiment, the protocol processing unit 102 is a hardware circuit device dedicated to transmission that minimizes the protocol processing time for transmitting image data.
However, it may be realized by a configuration of a processor that executes a dedicated program for transmitting image data in real time, or a configuration of a processor and a hardware acceleration circuit.
Image data created by the imaging unit 103 and the encoder 104 can be packetized and transmitted on the network in real time.

  The protocol processing unit 102 performs packet creation processing in a data link layer, IP, UDP, TCP protocol, and a moving image streaming protocol used by an application executed by the CPU 101. Examples of the video streaming protocol include RTP and HTTP.

  When transmitting image data from the encoder 104 in real time, the image data is immediately transferred to the protocol processing unit 102 by the data transfer switching unit 109 when it is output to one of the image buffers 106 and 107. The protocol processing unit 102 sequentially packetizes the transferred image data.

The communication management data 110 holds protocol control information in TCP / IP or moving picture streaming, statistical information of packets transmitted by the protocol processing unit 102, and the like between the CPU 101 and the protocol processing unit 102.
In this embodiment, the communication management data 110 is realized by a shared memory.
The TCP / IP control information includes a TCP / IP session state, a sequence number of transmission / reception packets, flow control, and the like. Also included is streaming control information in the upper streaming protocol.

  Transmission packets from the CPU 101 and the protocol processing unit 102 are finally subjected to transmission control by 115 MACs and sent to the network.

In the present invention, in order to control 115 MACs related to transmission of packets output from both sides, 111 transmission timing control units are provided.
The transmission timing control unit 111 adjusts the TCP / IP protocol processing of the CPU 101 and the transmission of the packet created by the protocol processing unit 102. The transmission timing control unit 111 has a feature that the streaming packet created by the protocol processing unit 102 is preferentially transmitted. Then, the packet transmission from both sides is controlled so that the packet from the CPU 101 is transmitted while there is no transmission packet from the protocol processing unit 102.

In the TCP / IP protocol processing of the CPU 101, the transmission packet is temporarily queued in the 112 transmission buffer. After queuing the transmission packet to the transmission buffer 112, the CPU 101 issues a transmission request to the transmission timing control unit 111.
Similarly, the protocol processing unit 102 issues a transmission request to the transmission timing control unit 111 when outputting a transmission packet.

  The transmission timing control unit 111 determines the transmission order of transmission packets from both sides, controls the selector 114 for determining the data transfer path from which the MAC 115 reads the transmission data, and issues a transmission instruction to the MAC 115.

The MAC 115 reads the packet from the transmission buffer 112 when transmitting a packet created by the TCP / IP protocol processing of the CPU 101.
On the other hand, when transmitting a packet created by the protocol processing unit 102, the MAC 115 transfers the packet directly from the protocol processing unit 102.

  The transmission timing control unit 111 receives a notification of packet transmission completion from the MAC 115. After receiving the transmission completion notification, either the CPU 101 or the protocol processing unit 102 notifies the transmission completion to the block that created the packet for which transmission has been completed.

A reception buffer 113 is a memory in which received packets are written.
In this embodiment, a packet received by the MAC 115 is written in the reception buffer 113.

  Then, the MAC 115 notifies the CPU 101 of a reception interrupt. All received packets are interpreted by the CPU 101 TCP / IP protocol processing.

  When the CPU 101 receives a streaming control packet such as an RTCP packet, an ACK packet, and control information of a TCP communication session in the streaming communication of the protocol processing unit 102, the communication management data 110 is updated.

  The above is the overall implementation configuration of the present embodiment. Next, an example of a control method in the input switching control unit 105 will be described with reference to FIGS. 2, 3, and 4.

  FIG. 2 shows an example of attribute information for each moving image stream necessary for controlling the encoder 102, the selector 108, and the data transfer switching unit 109 included in the input switching control unit 105.

The attribute information of each stream is given by the CPU 101. FIG. 2 shows four streams as No. (201) a to d as stream identifiers.
For each stream attribute, image (video) format (202), frame angle (203), frame rate (204), and image data output from encoder 104 should be passed to CPU 101 or protocol processing unit 102 There is a value (205). The CPU 101 sets the attribute information of these streams.

The input switching control unit 105 controls a series of operation sequences for the encoder 104, the selector 108, and the data transfer switching unit 109 based on the attribute information of each stream.
FIG. 3 shows a flowchart of the control program in the image frame data of each stream when the input switching control unit 105 is realized by a microprocessor.

  The processing flow in an image frame of the input switching control unit 105 starts from step S301. First, in step S302, the selector 108 sets the image buffer that is the output destination from the encoder 104 so that either the image buffer 1 (106) or the image buffer 2 (107) is different from the output destination of the previous encoding. Is switched.

  In step S303, the encoder 104 is set with an encoding format of image data.

  In step S304, the encoder 104 is instructed to start encoding. Then, the process waits for the encoding of the image data to end, and receives an encoding completion interrupt notification from the encoder in step S305.

  In step S306, it is determined whether the image data output to the image buffer 1 (106) or the image buffer 2 (107) is data to be transmitted by the protocol processing unit 102. If the protocol processing unit 102 transmits the data, the process proceeds to step S307. If not, the data is processed by the CPU 101, and the subsequent processing starts from step S312.

  In step S307, prior to image data transfer, the protocol processing unit 102 is notified of completion of image data preparation. In step S308, image data transfer is waited until the protocol processing unit 102 is ready to accept image data transfer.

  In step S309, if the image buffer in which the image data has been written is any image buffer 1 (106), an instruction to start transferring image data to the protocol processing unit 102 to the data transfer switching unit 109 in step S310. Put out.

  The same applies to the image buffer 2 (107). Next, transfer of image data is started in step S311. Then, the process proceeds to step S316, and a series of processing flows for this image data is completed.

  If image data is transmitted by the CPU 101 in step S306, the process proceeds to step S312.

  In the case of the image buffer 1 (106), the data transfer switching unit 109 is set so that the CPU can access the image buffer 1 in step S313 depending on which image buffer the image data is written in in step S312.

  If the data is written in the image buffer 2 (107), the data transfer switching unit 109 is set so that the image buffer 2 can be accessed from the CPU in step S314. From step S313 and step S314, the process then proceeds to step S315, where the CPU 101 is notified of the completion of image data preparation, and the process proceeds to step S316, where a series of processing flow for this image data is completed.

  The input switching control unit 105 repeatedly executes the above processing flow for each stream, and controls image data input from the encoder 104 to the CPU 101 or the protocol processing unit 102.

  FIG. 4 is a schematic time chart showing an example of image data input control executed by the input switching control unit 105 for the four streams (ad) shown in FIG.

  In FIG. 4, the time between the vertical axes 401 and 402 represents a time width of 100 ms, and the control content repeated by the input switching control unit 105 is represented within the time width.

On the horizontal axis 403, the output timing of the uncompressed image data from the imaging unit 103 is represented by 409 to 410. On the horizontal axis 404, the time required for encoding is represented by 412 to 419.
Within each time period 412 to 419, the input switching control unit 105 includes the encoding setting for the encoder 104 and the operation time of the encoder 104.
Reference numerals a to d described in 412 to 419 are Nos. Of the streams shown in FIG. 2 and indicate the target streams.
For example, uncompressed image data output at the timing of 409 is encoded three times at 412 to 414, indicating that the streams b, c, and a are to be encoded. Similarly, uncompressed image data output in 410 is encoded for b, d, and a streams in 415 to 417, and 411 is also encoded for b and a streams in 418 to 419. .
A horizontal axis 405 indicates a selection time of an output destination image buffer from the encoder 104. “1” or “2” written in each selection time indicates the image buffer 1 (106) and the image buffer 2 (107), respectively.

As described above, the input switching control unit 105 controls the selector 108 to switch between the double buffers of the image buffers 1 and 2 for each encoding.
Horizontal axes 406 and 407 indicate the operation of the data transfer switching unit 109 under the control of the input switching control unit 105, 406 indicates the operation corresponding to the image buffer 1 (106), and 407 indicates the operation corresponding to the image buffer 2 (107).

Symbols Δ or ▽ of triangles 428 to 430 and 432 to 436 written on the respective horizontal axes indicate operation switching timing of the data transfer switching unit 109.
ΔP represents the start of data transfer from the image buffer to the protocol processing unit 102, and ΔC represents the timing at which the CPU 101 can access the data in the image buffer. A hatched area 431 indicates a period during which the CPU 101 can access the image buffer.
In other words, the input switching control unit 105 executes data transfer from the image buffer 1 (106) to the protocol processing unit 102 at time points 428, 429, and 432. Further, the data transfer switching unit 109 is controlled from the image buffer 2 (107) so as to be executed at each of the times 433 to 436.

  On the horizontal axis 408, timings 437 to 444 are used to indicate an interrupt notification of input image data preparation completion from the input switching control unit 105 to the CPU 101 or the protocol processing unit 102. Triangle symbols ▲ P of 437 to 440 and 442 to 444 represent an interrupt notification to the protocol processing unit 102, and ▲ C of 441 represents an interrupt notification to the CPU 101.

  Based on the above, regarding the control processing repeated by the input switching control unit 105 in a certain stream, the stream No. in FIG. b and No. An example will be described with reference to FIG.

  First, no. The stream b will be described. Stream No. Three consecutive frames of b are uncompressed image data 409, 410, and 411 in FIG.

  In the case of 409 uncompressed image data, the input switching control unit 105 performs control so that the selector 108 selects the image buffer 1 (106) as the encoding output destination in 420 and the encoding is executed in 412.

  Next, stream No. Since b is transmitted by the protocol processing unit 102 (see FIG. 2), the input switching control unit 105 notifies the protocol processing unit 102 of an interruption of image data preparation completion in 437 after completion of the encoding 412.

  Thereafter, in 428, the data transfer switching unit 109 is started to transfer image data.

  The input switching control unit 105 performs the same control for the other uncompressed images 410 and 411.

  Stream No. In the case of d, the uncompressed image data 410 in FIG. 4 is encoded.

  The input switching control unit 105 performs control such that the image buffer 1 (106) is selected by the selector 108 as the encoding output destination at 424 and the encoding is executed at 416.

  Stream No. Since d is transmitted by the CPU 101 (see FIG. 2), the input switching control unit 105 allows the image buffer 1 (106) to be accessible from the CPU 101 at 430 after the encoding of 416 is completed. To control. Thereafter, in 441, the CPU 101 is notified of the interruption of the completion of the image data preparation.

  The period during which the CPU 101 can access the data in the image buffer 1 (106) is the period 431 that is until 426 when the encoded data is written to the image buffer 1 (106).

  The example of the control method of the input switching control unit 105 in the present embodiment has been described above with reference to FIGS. 2, 3, and 4.

  Next, a method for adjusting packet transmission from both the CPU 101 and the protocol processing unit 102 in the transmission timing control unit 111 will be described with reference to FIG.

  FIG. 5 shows an example of a packet transmission process sequence between the CPU 101, the protocol processing unit 102, the transmission timing control unit 111, and the MAC 115.

The time transition from the top to the bottom of FIG. 5 is shown, and a transmission request and a notification of transmission completion between the processing units are indicated by arrows.
In FIG. 5, an arrow with “P” in the symbol name means packet transmission processing from the protocol processing unit 102, and an arrow with “C” in the symbol name is packet transmission processing from the CPU 101 like P501-1. Means.

  A series of transmission sequences of one packet is indicated by four arrows having different branch numbers “1” to “4”. For example, P501-1 is a transmission request from the protocol processing unit 102 to the transmission timing control unit 111, and P501-2 is a transmission instruction from the transmission timing control unit 111 to the MAC 115 for the packet. P501-3 is a transmission completion notification of the same packet from the MAC 115 to the transmission timing control unit 111, and P501-4 is a transmission completion notification of the same packet from the transmission timing control unit 111 to the protocol processing unit 102.

As described above, the CPU 101 and the protocol processing unit 102 issue a transmission request to the transmission timing control unit 111 for the created packet.
The CPU 101 requests transmission after queuing the transmission packet in the transmission buffer 112.

Since the protocol processing unit 102 sequentially packetizes and transmits the image data transferred from the image buffer, the protocol processing unit 102 continuously requests transmission of a set number of packets for each image data.
A broken line range 512 in FIG. 5 indicates that the protocol processing unit 102 continuously transmits packets P501 to P504. Similarly, a broken line range 513 indicates that packets P508 to P510 are continuously transmitted.
Continuous packet transmission from the protocol processing unit 102 is realized by starting transmission immediately after creating each packet.

When receiving a packet transmission request from the protocol processing unit 102, the transmission timing control unit 111 sets the packet as the next transmission packet and instructs the MAC 115 to transmit.
On the other hand, packet transmission from the CPU 101 is queued in the transmission buffer 112. In FIG. 5, C505 to C507 packets are transmitted to the transmission timing control unit 111 during continuous transmission processing from the protocol processing unit 102 in the range of 512.

  The packets C505 to C507 are queued, and the transmission timing control unit 111 starts transmitting these packets after the transmission process in the range 512 is completed.

  In FIG. 5, the C505 packet transmission process has been completed, and during the next C506 packet transmission process, the protocol processing unit 102 starts continuous transmission of the P508 to P510 packets described in the range 513.

  For this reason, the transmission timing control unit 111 transmits P508 with priority given to the transmission processing of the protocol processing unit 102 after completing the transmission processing of C506.

  The C507 packet is transmitted after the transmission of the protocol processing unit 102 within the range 513 is completed.

  Similarly, the transmission timing control unit 111 delays transmission of the C511 packet, and performs transmission after transmission of C507.

  In the transmission request from the protocol processing unit 102 to the transmission timing control unit 111, the protocol processing unit 102 adds information indicating whether image data to be transmitted still remains, that is, whether the transmission request continues continuously. .

  As a result, the transmission timing control unit 111 determines the interval between successive transmission requests from the protocol processing unit 102 and starts transmission of the transmission packet from the CPU 101.

  As described above, the example of the packet transmission adjustment method of the transmission timing control unit 111 has been described with reference to the example illustrated in FIG. 5.

  By this method, it is possible to prioritize packets from the protocol processing unit 102 and to minimize the amount of delay in transmission processing.

  Finally, image data transmission processing in the CPU 101 will be described.

  In this embodiment, when the image data to be transmitted by the CPU 101 is output from the encoder 104 to either the image buffer 106 or 107, an interrupt notification indicating that the image data is ready is received from the input switching control unit 105.

  Then, the image data in the image buffer is accessed, and transmission packet processing is performed by the application and the general protocol stack.

  For packet transmission, the transmission packet is written in the transmission buffer 112 and a transmission request is issued to the transmission timing control unit 111.

  Further, the CPU 101 sets the attribute information of the moving image stream in the input switching control unit 105.

  A processing flow of such processing of the CPU 101 is shown in FIG.

  The processing flow of the CPU 101 starts from step S601. First, in step S602, the input switching control unit 105 is initialized. In the initial setting, the attribute information of each stream to be transmitted is set by the CPU 101 and the protocol processing unit 102. In step S603, the protocol processor 102 is initialized.

  Also, initial setting information in a protocol for streaming transmission is written in the communication management data 110. Subsequently, processing of the imaging unit 103 and the input switching control unit 105 is started.

  From step S605, the CPU 101 waits for image data from the encoder 104 to perform transmission. In step S605, an interrupt notification indicating completion of image data preparation waits from the input switching control unit 105, and an interrupt notification indicating completion of image data preparation is received in step S606.

  When the image data to be transmitted by the CPU 101 is prepared in the image buffer, the image data transmission processing from step S608 to step S613 is repeated until there is no untransmitted image data in the determination of step S607.

  When there is no untransmitted image data, the process returns to step S605 to wait again for an interrupt notification of image data preparation completion.

  In step S608, it is checked whether there is enough free space in the transmission buffer 112 to queue the transmission packet. If there is a free area, the process advances to step S609 to execute streaming protocol processing by the application and general-purpose TCP / IP protocol processing to create a transmission packet from the image data.

  Subsequently, in step S610, the created packet is queued in the transmission buffer 112, and a transmission request is issued to the transmission timing control unit 111 in step S611.

  Then, returning to step S607, if there is untransmitted image data, transmission processing of the next packet starts again from step S608.

  In step S608, if there is no free area in the transmission buffer 112 for queuing the transmission packet, the process proceeds to step S612.

  In step S612, the transmission timing control unit 111 waits for a notification of completion of transmission of a packet for which a transmission request has been issued first.

  When the notification of transmission completion is received in step S613, the process returns to step S608, and the free area of the transmission buffer 112 is checked again.

  With the above processing flow, the CPU 101 starts transmission processing when image data to be transmitted is generated, and performs an operation of waiting for image data again after the image data transmission processing.

  In the above, an example of embodiment in this invention was shown. According to this embodiment, the application can set the input switching control unit 105 so that the CPU 101 and the protocol processing unit 102 switch the transmission process for each stream. The image data from the encoder 104 is automatically distributed to the CPU 101 and the protocol processing unit 102.

  Therefore, for example, real-time transmission of a captured moving image that reduces transmission delay as much as possible is executed by the protocol processing unit 102, and flexible transmission processing can be performed by general-purpose protocol processing even if the transmission delay time is acceptable. The CPU 101 can perform transmission processing.

(Second Embodiment)
In the first embodiment, the transmission timing control unit 111 in FIG. 1 performs control to prioritize transmission of a packet requested by the transmission protocol processing unit 102 over a packet requested by the CPU 101.
Since the transmission timing control unit 111 gives priority to the transmission packet from the protocol processing unit 102, the transmission packet from the CPU 101 is queued in the transmission buffer 112.
Therefore, when the total number of streams processed by the CPU 101 and the protocol processing unit 102 is large, there is a possibility that the transmission packet data may overflow due to a restriction on the amount of data that can be queued by the transmission buffer 112 even with communication media having a large transmission band. . The same applies to a case where the amount of image data to be transmitted on the CPU 101 side is very large.

In the present embodiment, an implementation configuration and packet transmission control for avoiding overflow in the transmission buffer on the CPU side will be described with reference to FIG.
FIG. 7 is a diagram in which the latter part is changed from the transmission processing of the CPU 101 and the protocol processing unit 102 in the overall configuration shown in FIG.

  In the present embodiment, the configuration before the CPU and the protocol processing unit is the same as that of the first embodiment.

  In FIG. 7, 701 is a CPU, 702 is a protocol processing unit, 703 is a transmission timing control unit, 704 is a transmission buffer, 705 is a MAC, and 706 is a selector for determining a data transfer path from which the MAC reads data. .

  Reference numeral 708 denotes an input image data path to the CPU 701, and 709 denotes an input image data path to the protocol processing unit 702.

  In the present embodiment, the transmission packet from the protocol processing unit flowing in the data path 710 in FIG. 7 is limited to the UDP packet. That is, in this embodiment, it is assumed that the upper streaming protocol uses a UDP packet. An example of such a higher-level protocol for streaming is RTP.

In this embodiment, the buffer monitoring unit 707 is provided. The buffer monitoring unit 707 has a function of monitoring the remaining free space in the transmission buffer 704. The transmission buffer 704 includes management data 711 for the transmission packet queue and a storage area 712 for packet data on the memory device.
The buffer monitoring unit 707 reads the transmission packet queue management data 711 and monitors the remaining free space in the packet data storage area 712. Hereinafter, the remaining free space in the transmission buffer 704 means the remaining free space in the packet data storage area 712.

  The buffer monitoring unit 707 is connected to the transmission timing control unit 703 through the signal line 713, and the state of the signal line 713 is determined depending on whether or not the remaining free space in the transmission buffer 704 is smaller than a threshold value at which no overflow occurs. To change.

The transmission timing control unit 703 gives priority to the transmission packet from the protocol processing unit 702 when the state of the signal line 713 from the buffer monitoring unit 707 is in a state where the remaining amount of the transmission buffer 704 is larger than the threshold. Send.
On the other hand, when the remaining free space in the buffer 704 is smaller than the threshold, the transmission of packets from the protocol processing unit 702 is suppressed and adjusted until the remaining free space becomes larger than the threshold, and is queued in the transmission buffer 704. The control which can transmit the transmitted packet is performed.

This control is realized by thinning out transmissions from the protocol processing unit 702. That is, in response to a UDP packet transmission request from the protocol processing unit 702, for each random request count, a transmission instruction to the MAC 705 is not given, and transmission to the protocol processing unit 702 is immediately completed without performing actual transmission. Notice.
That is, the transmission timing control unit 703 decreases the number of packet transmissions from the protocol processing unit 702 and increases the timing at which packets queued in the transmission buffer 704 and packets requested by the CPU 701 are transmitted.

  With the above control method, the transmission buffer 704 realizes transmission control so that overflow does not occur.

(Third embodiment)
In the second embodiment, in order to avoid an overflow in the transmission buffer on the CPU side, the implementation configuration and the packet transmission control in the subsequent stage from the transmission processing of the CPU 101 and the protocol processing unit 102 in the first embodiment will be described. ing.

  In the present embodiment as well, as in the second embodiment, an object is to avoid overflow in the transmission buffer on the CPU side. Hereinafter, the implementation configuration and image data encoding control will be described with reference to FIGS.

FIG. 8 is a diagram showing the overall configuration of the present embodiment.
801 to 815 in FIG. 8 have the same configurations as 101 to 115 in FIG. 1 in the first embodiment. That is, reference numerals 801 to 809 denote a CPU 801, a protocol processing unit 802, an imaging unit 803, an encoder 804, an input switching control unit 805, an image buffer 1 (806), an image buffer 2 (807), an output image buffer selector 808, and data transfer. A switching unit 809; Reference numerals 810 to 815 denote communication management data 810, a transmission timing control unit 811, a transmission buffer 812, a reception buffer 813, a transmission data input selector 814, and a MAC 815.

In the present embodiment, 816 buffer monitoring units are provided. The buffer monitoring unit 816 has a function of monitoring the remaining free space in the transmission buffer 812. The transmission buffer 812 includes management data 817 for the transmission packet queue and a storage area 818 for packet data on the memory device.
The buffer monitoring unit 816 reads the transmission packet queue management data 817 and monitors the remaining free space in the packet data storage area 818.
Hereinafter, the remaining free space in the transmission buffer 812 means the remaining free space in the packet data storage area 818.

  The buffer monitoring unit 816 is connected to the transmission timing control unit 811 through the signal line 819, and changes the state of the signal line 819 depending on the free space of the transmission buffer 812. The buffer monitoring unit 816 checks the remaining amount of the transmission buffer 812 with two thresholds, and holds and changes the signal line 819 to a three-level state.

  Specifically, when the remaining free space in the transmission buffer 812 is larger than the first threshold, there is a free space remaining and the transmission buffer 812 does not overflow, and is set to the first level.

  When the remaining free space in the transmission buffer 812 is smaller than the first threshold value and larger than the second threshold value, the remaining free space is insufficient, and the second level is set. If the remaining free space in the transmission buffer 812 is smaller than the second threshold, the remaining free space in the transmission buffer 812 is small and the possibility of overflow is high, and the third level is set.

  The buffer monitoring unit 816 holds one of the signal lines 819 in the three-level state.

  When the state of the signal line 819 from the buffer monitoring unit 816 changes, the transmission timing control unit 811 of the present embodiment issues an encoding control change instruction to the input switching control unit 805 (820). That is, it is possible to perform feedback to image data encoding control according to the remaining free space in the transmission buffer 812.

  The encoding control change instruction 820 indicates whether the stream subject to control change is only the transmission stream on the CPU 801 side or all the transmission streams, and whether the bit rate of the target stream is lowered or returned to the normal bit rate. To do. Specifically, when the state of the signal line 819 changes from the first level to the second level and when the state of the signal line 819 changes from the third level to the second level, the transmission timing control unit 811 A change instruction 820 for lowering the bit rate for the transmission stream is issued.

  Also, an encoding control for lowering the bit rate for all transmission streams when the state of the signal line 819 changes from the second level to the third level and from the first level to the third level. The change instruction to is issued.

  Furthermore, when the state of the signal line 819 changes from the second level to the first level, and when the state changes from the third level to the first level, all streams are targeted at a normal bit rate. A change instruction to output encoding control is issued.

  In response to the encoding control change instruction 820 from the transmission timing control unit 811 to the input switching control unit 805, the input switching control unit 805 performs three types of encoding control.

In the first encoding control, image data is encoded at a normal bit rate so as to match the attribute information of each stream set in the input switching control unit by the CPU 801.
The second encoding control is a control for encoding image data so that only the moving image stream to be transmitted by the CPU 801 is targeted and the bit rate of each stream is lowered.
The third encoding control is control for encoding image data so as to lower the bit rate of each stream for all moving picture streams.

  Control for lowering the bit rate of each stream differs depending on the encoding format. For example, in the case of JPEG, it is realized by switching the quantization table and setting the quantization level to be large. In the case of MPEG, the setting of the bit rate to the encoder 804 for each stream is lowered, or when the detailed control to the encoder 804 is possible, the quantization step size is increased or the frame interval of intra-frame compression encoding is increased. Change encoding conditions such as enlarging.

  Next, a processing flow for changing the encoding control of the input switching control unit 805 according to the encoding control change instruction 820 from the transmission timing control unit 811 will be described with reference to FIG.

  In this embodiment, the encoding control change process of the input switching unit control 805 shown in FIG. 9 starts from step S901, and is a loop from step S902 to step S911 for each uncompressed digital image data output from the imaging unit 903. It is processing.

  First, in step S903, it is checked whether there is an encoding control change instruction 820 from the transmission timing control unit 811. If there is an instruction, the process proceeds to step S904.

  If there is no instruction to change the encoding control, it is assumed that the current encoding control is maintained, and the process proceeds to step S910.

  In step S904, it is checked whether or not the encoding control change instruction 820 is control for lowering the bit rate. If it is an instruction to lower the bit rate, the process proceeds to step S905; otherwise, the process proceeds to step S906.

  In step S905, if the target of the control instruction for reducing the bit rate is a stream to be transmitted by the CPU 801, the process proceeds to step S908. Otherwise, all streams transmitted by both the CPU 801 and the protocol processing unit 802 are targeted, and the process proceeds to step S907.

  On the other hand, when the process proceeds from step S904 to step S906, it is an instruction to return to control for encoding at a normal bit rate for each stream. In step S906, it is checked whether or not the target stream whose bit rate is to be returned is all streams. If all the streams are the target, the process proceeds to step S909. If not, the process proceeds to step S908.

  In step S907, the control shifts to encoding control for lowering the bit rate of each stream for all streams. That is, it means a shift to the third encoding control.

  In step S908, the transmission stream on the CPU 801 side is targeted, encoding is performed to lower the bit rate of each stream, and the stream for transmission processing by the protocol processing unit 802 is shifted to control for encoding at a normal bit rate. That is, it means a shift to the second encoding control. In step S909, control is performed so that encoding of all the streams is performed at a normal bit rate in each stream. That is, it means a shift to the first encoding control.

  When the encoding control in steps S907, S908, and S909 shifts, the process proceeds to step S910. Step S910 is the processing of the input switching control unit 802 for encoding and transmitting image data, and is the processing flow shown in FIG.

  Next, it progresses to step S911 and the processing loop from step S902 is completed.

  Through the processing flow as described above, the input switching control unit 805 changes the encoding control of the image data in accordance with the encoding control change instruction 820 from the transmission timing control unit 811.

The details of the present embodiment have been described above. In this embodiment, the buffer monitoring unit 816 can monitor the remaining capacity of the transmission buffer 812 that queues transmission packets from the CPU 801 and can cause the transmission timing control unit 811 to detect the remaining capacity of the transmission buffer. . As a result, the transmission timing control unit 811 can issue an encoding control change instruction 820 to the input switching control unit 805.
On the other hand, the input switching control unit 805 performs control to lower the bit rate of the moving picture stream according to the encoding control change instruction 820 from the transmission timing control unit 811.

  According to the present embodiment, when the remaining free space in the transmission buffer 812 decreases, the amount of data queued in the transmission buffer can be reduced by reducing the data amount of the stream to be transmitted by the CPU 801. In addition, the data amount of the stream to be transmitted by the protocol processing unit 802 can be reduced to reduce the number of transmissions, thereby increasing the timing at which the CPU 801 transmits a packet for which transmission is requested.

  Therefore, the transmission buffer 812 can realize transmission control so that overflow does not occur.

(Fourth embodiment)
In the first to third embodiments, the setting and control of the encoder 104 and the control of the data transfer switching unit 109 are performed by software control of the input switching control unit 105 configured by a microprocessor. In the embodiment, the CPU 101 executes application software for transmitting and receiving image data and a general-purpose communication protocol stack.

In the fourth embodiment, an embodiment in which all of these controls are performed by a single CPU will be described.
FIG. 10 is a diagram showing an example of the overall configuration in the fourth embodiment of the present invention.
The protocol processing unit 102 to MAC 115 in FIG. 10 are the same configuration blocks as those in FIGS. 1 and 102 to 115 described in the first embodiment. In FIG. 10, reference numeral 1001 denotes a CPU, which executes the following control by software executed by the CPU.
The CPU 1001 sets parameters such as an encoding format and compression efficiency for the encoder 104 for each image frame data output from the imaging unit 103. Also, encoder control is performed such as instructing the encoding start timing and receiving an interrupt signal notifying completion of encoding.
Further, the selector 108 of the data path of the image data output from the encoder 104 is controlled to determine which image buffer to output. The data transfer switching unit 109 depends on whether the image data output to the image buffer 1 (106) or the image buffer 2 (107) is data transmitted by the CPU 1001 or data transmitted by the protocol processing unit 102. To control.

  If the data is to be transmitted by the CPU 1001, the CPU 1001 is notified of the completion of image data preparation. If the data is transmitted by the protocol processing unit 102, the protocol processing unit 102 is notified of the completion of image data preparation and checks whether the protocol processing unit 102 is ready to accept data transfer. Thereafter, the data transfer switching unit 109 is started to transfer image data.

  Also, it has protocol processing software that enables execution of application software that transmits and receives image data and general-purpose TCP / IP communication, and transmits and receives TCP / IP packets.

  In the TCP / IP protocol processing of the CPU 1001, the transmission packet is temporarily queued in the 112 transmission buffer, and then a transmission request is issued to the transmission timing control unit 111.

  Next, a control flowchart by the CPU 1001 in the fourth embodiment will be described with reference to FIG.

  In FIG. 11, the processing flow of the CPU 101 starts from step S1101, and first, initial setting of the protocol processing unit 102 is executed in step S1102.

  Also, initial setting information in a protocol for streaming transmission is written in the communication management data 110. Subsequently, in step S1103, an imaging start instruction is given to the imaging unit 103, whereby capturing of imaging data is started.

  In step S1104, the process waits for arrival of uncompressed image data for one frame imaged by the imaging unit 103. When the data arrives, the process proceeds to step S1105. In step S1105, the selector 108 is set so that the image buffer serving as the output destination from the encoder 104 is different from the output destination of the previous encoding, either the image buffer 1 (106) or the image buffer 2 (107). Execute the switching process. Next, in step 1106, the encoder 104 is set to the encoding format of the image data.

  In step S1107, the encoder 104 is instructed to start encoding.

  In step S1108, it is checked whether there is already encoded stream data in an image buffer different from the image buffer output by the encoder in step S1107. If there is data, the process proceeds to step S1110 while the encoding process by the encoder 104 is continued.

  If there is no data, in step S1109, the process waits for the encoding to end. When an encoding completion interrupt notification is received from the encoder 104, the process proceeds to step S1110.

  In step S1110, it is determined whether the image data output to the image buffer 1 (106) or the image buffer 2 (107) is data to be transmitted by the protocol processing unit 102. If the protocol processing unit 102 transmits the data, the process proceeds to step S1121. If not, the data is processed by the CPU 1001, and the subsequent processing starts from step S1111.

  In step S1121, prior to the transfer of the image data, the protocol processing unit 102 is notified of the completion of the preparation of the image data.

  Next, in step S1122, the image processing apparatus waits for the image data transfer until the protocol processing unit 102 is ready to accept the image data transfer. In step S1123, depending on which image buffer the image data is written in, in the case of the image buffer 1 (106), the image data to the protocol processing unit 102 is sent to the data transfer switching unit 109 in step S1124. Give instructions to start the transfer. Similarly, in the case of the image buffer 2 (107), transfer of image data is started in step S1125.

  When the process of step S1124 or S1125 ends, in step S1126, it is checked whether transmission of all encoded streams for one frame of uncompressed image data captured by the imaging unit 103 is completed. In step S1127, when transmission is completed, the process returns to the process of waiting for arrival of uncompressed data in step S1104 again.

If transmission has not ended, the process returns to the process of switching the selector in step S1105.
On the other hand, when the image data is transmitted by the CPU 1001 in step S1110, the process proceeds to step S1111. In the case of the image buffer 1 (106), the data transfer switching unit 109 is set so that the CPU can access the image buffer 1 in step S1113 depending on which image buffer the image data is written in in step S1111. If the data has been written in the image buffer 2 (107), the data transfer switching unit 109 is set so that the CPU can access the image buffer 2 in step S1112. When the process of step S1112 or S1113 ends, the process proceeds to step S1114.

  In step S1114, it is checked whether there is untransmitted image data in the image buffer to be transmitted. If there is no image data, that is, if transmission of all image data is completed, transmission of all streams in step S1126 is completed. Proceed to check if finished.

  If untransmitted image data remains in step S1114, the image data transmission processing from step S1115 to step S1120 is repeated until there is no untransmitted image data in the determination in step S1114.

  In step S1115, it is checked whether or not there is enough free space in the transmission buffer 112 to queue the transmission packet. If there is a free area, the process advances to step S1116 to execute a streaming protocol process and a general-purpose TCP / IP protocol process by the application, and create a transmission packet from the image data.

  In step S1117, the created packet is queued in the transmission buffer 112, and a transmission request is issued to the transmission timing control unit 111 in step S1118.

  Then, the process returns to step S1114, and if there is untransmitted image data, the next packet transmission process starts again from step S1115.

  In step S1115, if there is no free space in the transmission buffer 112 for queuing the transmission packet, the process proceeds to step S1119. In step S1119, the transmission timing control unit 111 waits for a notification of completion of transmission of a packet for which a transmission request has been issued first.

  When the notification of transmission completion is received in step S1120, the process returns to step S1115, and the free space in the transmission buffer 112 is checked again.

  With the above processing flow, the CPU 1001 starts transmission processing when image data to be transmitted is generated, and performs an operation of waiting for image data again after the image data transmission processing.

  The embodiment of the present invention can be realized by, for example, a computer executing a program. Also, means for supplying a program to a computer, for example, a computer-readable recording medium such as a CD-ROM recording such a program, or a transmission medium such as the Internet for transmitting such a program is also applied as an embodiment of the present invention. Can do. The above program can also be applied as an embodiment of the present invention. The above program, recording medium, transmission medium, and program product are included in the scope of the present invention.

It is a figure which shows an example of the whole structure of this invention. It is a figure which shows an example of the attribute information of a moving image stream. It is a flowchart which shows the control in the image frame data of each stream, when an input switching control part is implement | achieved by the microprocessor. It is a typical time chart which shows the example of control of the image data input of an input switching control part. It is a figure which shows an example of the sequence of the packet transmission process between CPU, a protocol process part, a transmission timing control part, and MAC. It is a figure which shows the processing flow of CPU in 1st Embodiment. It is a figure which shows the structure of a back | latter stage part from CPU and protocol process part in 2nd Embodiment. It is a whole block diagram in 3rd Embodiment. It is a figure which shows the processing flow of the input switching control part in 3rd Embodiment. It is a figure which shows an example of the whole structure in 4th Embodiment. It is a figure which shows the processing flow of CPU in 4th Embodiment.

Explanation of symbols

101, 701, 801, 1001: CPU
102, 702, 802: Protocol processing unit 103, 803: Imaging unit 104, 804: Encoder 105, 805: Input switching control unit 106, 806: Image buffer 1
107, 807: Image buffer 2
108, 808: Encoder output data path selector 109, 809: Data transfer switching unit 110, 810: Communication management data 111, 703, 811: Transmission timing control unit 112, 704, 812: Transmission buffer 113, 813: Reception buffer 114, 706, 814: MAC input data path selector 115, 705, 815: MAC
707, 816: Buffer monitoring unit 708: Image data to CPU 709: Image data to protocol processing unit 710: UDP packet data transmitted by protocol processing unit 711, 817: Management data of queue of transmission buffer 712, 818: Transmission Transmission packet data in buffer queue 713, 819: Signal line between buffer monitoring unit and transmission timing control unit 820: Encoding control change instruction

Claims (4)

In a video data transmission device that simultaneously transmits video stream data compressed and encoded into a plurality of formats to a plurality of terminals connected to a network,
An encoder capable of generating compression-coded data of a plurality of formats for uncompressed video data;
First network protocol processing means executed by software by a processor;
Second network protocol processing means for executing real-time transmission processing of image data;
A MAC for transmitting the first transmission packet data generated by the first network protocol processing means and the second transmission packet data generated by the second network protocol processing means to the network;
Transmission timing switching means for performing control to selectively output the first transmission packet and the second transmission packet to the MAC;
A video data transmitting apparatus comprising:   2. The video data transmitting apparatus according to claim 1, wherein the transmission timing switching unit performs control so as to change an output order of the first transmission packet and the second transmission packet with respect to the MAC.   2. The video data transmitting apparatus according to claim 1, wherein the transmission timing switching unit performs control so as to always increase the output priority of the second transmission packet.   A transmission buffer for queuing the first transmission packet data generated by the first network protocol processing means; and 2. The video data transmitting apparatus according to claim 1, wherein processing is performed so that the transmission buffer does not overflow with transmission packet data.

Images