WO2012042701A1

WO2012042701A1 - Multi-stream encoding control device and camera system

Info

Publication number: WO2012042701A1
Application number: PCT/JP2011/002916
Authority: WO
Inventors: 岩橋賢二
Original assignee: パナソニック株式会社
Priority date: 2010-09-28
Filing date: 2011-05-25
Publication date: 2012-04-05
Also published as: JP2012074844A

Abstract

In a multi-stream encoding control device, when each video data is encoded by two or more mutually different coding systems, an encode prediction unit (204), in the encoding by each coding system in an encoding unit (202), determines the setting in the next encoding on the basis of any one or more of an code amount and an encode parameter after encoding, the code amount and encode parameter at encoding before the encoding, the frame rate to be encoded, and the output rate of an output unit (203). Thus, the total of code amount per unit time encoded respectively by a plurality of coding systems can be controlled in accordance with the transmission rate of the output unit 203. Hereby, when code data encoded respectively by a plurality of coding systems is outputted at the same time, the deterioration of image quality is suppressed and a high-resolution image can be delivered.

Description

Multi-stream encoding control apparatus and camera system

The present invention is a multi-stream code that provides high-quality moving images in a limited output band when a multi-stream image is distributed in real time in an image distribution or TV conference system mainly as a Web camera. The present invention relates to a computerized control device.

In recent years, it has become commonplace for general households to use IP always-on connections capable of broadband and high-speed data transmission using dedicated lines such as ADSL, optical fiber, and CATV network. As a result, the Internet is always connected to home and office terminals, and communication means such as videophone systems and videoconferencing systems that transmit multimedia information such as still images, videos, audio, etc. are realized. It has become possible.

As a video input device in a communication terminal in the above videophone system or the like, a technique has been advanced in which a video signal captured by a camera is processed as a digital signal inside the camera and a digital video signal is distributed.

This makes it possible to distribute the video signal captured by the camera to multiple users via the Internet, etc., and the video signal data to be distributed is encoded according to the output transmission speed of the distribution destination, and the captured video In real time.

At this time, the encoded video signal handled on the user side differs depending on the user environment, and it is necessary to perform encoding corresponding to the environment on the distribution side.

FIG. 1 is a block diagram showing a system for inputting video signals from a camera and encoding and distributing desired resolution data.

In FIG. 1, an image input unit 101 inputs a video image from a camera. The YC generation / resize processing unit 102 generates YC from the input video signal and resizes it to a desired resolution. The encoding unit 103 performs encoding by the encoding processing unit 105 according to the state of the distribution destination, and the output unit 104 performs the encoding processing according to an external output transmission rate via an external interface such as Ether or USB. The code amount is controlled by the unit 105 and the encoded video is distributed.

JP 2004-88480 A

However, in the configuration as described above, output data from the output unit 104 is only one system, but the encoding unit 103 has a plurality of encoding processing units 105 and encodes them in two or more different encoding formats. When it is attempted to simultaneously output such multiple types of code data from the output unit 104, it is assumed that the code data output rate of the output unit 104 will be exceeded.

Thus, for example, it is conceivable to adjust a part or all of the code results in a plurality of encoding processing units by re-encoding to suppress the total code amount. In this case, when re-encoding is performed. There is a problem that the memory access increases, the output of the code data from the output unit is delayed, and further the memory increases due to buffering.

Further, when dealing with the total code amount in the plurality of encoding processing units so as not to exceed the maximum total code amount corresponding to the output rate of the output unit, in each encoding process in the plurality of encoding processing units In consideration of the change in the amount of code over time, the amount of code to be encoded in each encoding processing unit is limited to a small amount of code in consideration of the maximum code amount in other encoding processing units, and a plurality of encoding processes It is also assumed that the total amount of codes encoded by the unit will be extremely small. In this case, the total code amount per unit time is extremely small as compared with the output rate of the output unit, and there is a problem that image quality deterioration occurs.

An object of the present invention is to provide a plurality of encoding processing units that perform encoding in two or more different encoding formats in a multi-stream encoding control apparatus, and code data encoded by the plurality of encoding processes. Is output simultaneously from the output unit, image quality deterioration is suppressed and a high-quality image can be transmitted in real time.

In order to achieve the above object, in the present invention, the total code amount per unit time encoded by a plurality of encoding processing units does not exceed the code data output rate of the output unit and is close to the code data output rate. To control.

Specifically, the multi-stream encoding control apparatus according to the first aspect of the present invention includes an image input unit that inputs a video signal and two or more different encodings of the video signal input to the image input unit. An encoding unit that performs encoding in accordance with a scheme; an output unit that outputs two or more types of encoded data encoded by the encoding unit; and the encoding unit based on the encoding result of the encoding unit. An encoding prediction unit that determines the next encoding setting in the encoding unit, and the encoding prediction unit is set at the time of encoding the encoding results in two or more different encoding methods The total code amount in the two or more different encoding schemes is controlled based on the encoding parameters.

The invention according to claim 2 is the multi-stream encoding control apparatus according to claim 1, wherein the two or more different encoding schemes are two or more different moving image encoding schemes, and the encoding prediction The unit, for each of the two or more different encoding schemes, the encoding amount and encoding parameters after encoding, and the encoding amount and encoding parameters at the previous encoding than the encoding, The next encoding parameter is determined according to one or more of a frame rate to be encoded and an output rate of the output unit.

The invention according to claim 3 is the multi-stream encoding control apparatus according to claim 1, wherein the two or more different encoding methods are one or more still image encoding methods and one or more moving image codes. The encoding prediction unit, for each of the two or more different encoding methods, the encoding amount after encoding and the encoding parameter, and the encoding time before the encoding time. The next encoding parameter is determined according to any one or more of the following code amount and encoding parameter, the encoding frame rate, and the output rate of the output unit.

According to a fourth aspect of the present invention, in the multi-stream encoding control apparatus according to the second aspect, the two or more different moving image encoding methods have different encoding processes for each frame of the video signal, The encoding prediction unit predicts that the encoding amount varies due to different encoding processing for each frame of the video signal in each encoding method, and controls the encoding amount to be encoded next. And

According to a fifth aspect of the present invention, the multi-stream encoding control apparatus according to the third aspect further includes a dividing unit that divides a still image encoding result in the still image encoding method into a plurality of divided results. Then, the output unit notifies the encoded prediction unit to the encoded prediction unit, and the encoded prediction unit determines the moving image code based on the output rate of the output unit and the divided data size. The code amount of the moving image encoding process in the encoding method is controlled.

According to a sixth aspect of the present invention, in the multi-stream coding control apparatus according to the fourth aspect of the invention, the coding prediction unit holds information in which the code amount fluctuates periodically for the control of the code amount to be coded next. In addition, the control is performed so that frames with a large amount of codes do not overlap.

According to a seventh aspect of the present invention, there is provided a camera system comprising the multi-stream encoding control apparatus according to any one of the first to sixth aspects.

As described above, according to the first aspect of the present invention, when video signals of two or more different encoding methods are output simultaneously, the sum per unit time of the amount of code encoded by each of the plurality of encoding methods. Can be controlled in accordance with the code data output speed of the output unit, so that the sum of the code amount per unit time can always be controlled to the maximum sum according to the transmission rate of the output unit. High-definition images can be distributed while suppressing image quality degradation associated with conversion.

In addition, since it is not necessary to re-encode the code amount encoded by each encoding method, it is possible to deliver a high-definition image without delay.

Further, in the invention according to claim 3, since a still image can be output in accordance with the characteristics of the moving image encoding method, it is possible to output a still image while delivering a high-definition image while suppressing deterioration of the image quality of the moving image. It is.

Furthermore, in the invention according to claim 4, since it is possible to control so that the pictures having large code amounts do not overlap each other when encoding by a plurality of encoding methods, the total code amount per unit time is obtained from the output unit. It is possible to prevent exceeding the code amount corresponding to the code data output speed, and to suppress deterioration in image quality.

In addition, in the invention according to claim 5, since a still image can be output in accordance with the characteristics of the moving image coding system, the output of the still image can be output while delivering a high-definition image while suppressing the deterioration of the image quality of the moving image. Is possible.

In the invention according to claim 6, H. Since it can be controlled so that pictures with a large code amount do not overlap in a moving image encoding method such as H.264, the total code amount per unit time can be prevented from exceeding the code amount according to the code data output speed of the output unit, It is possible to suppress image quality deterioration.

As described above, according to the present invention, when video signals of two or more different encoding methods are output simultaneously, the amount of code encoded by each of these plural encoding methods per unit time Since the sum can be controlled in accordance with the transmission rate of the output unit, the sum of the code amount per unit time can always be controlled to the maximum sum according to the transmission rate of the output unit, and a high-definition image can be controlled. Can be delivered.

In addition, since it is not necessary to re-encode the code amount encoded by each encoding method, a high-definition image can be distributed without delay.

FIG. 1 is a block diagram of a conventional encoded distribution system. FIG. 2 is a block diagram showing the overall configuration of the multi-stream coding control apparatus according to the first embodiment of the present invention. FIG. 3 is a flowchart showing the operation of the multi-stream coding control apparatus. FIG. 4 is a block diagram showing the overall configuration of a Web camera equipped with the multi-stream encoding control apparatus according to the second embodiment of the present invention. FIG. 5A is a flowchart showing the operation of the Web camera, and FIG. 5B is a flowchart showing details of steps S504 and S506 in the operation flowchart of FIG. FIG. 6 is a block diagram showing the overall configuration of a Web camera equipped with the multi-stream encoding control apparatus according to the third embodiment of the present invention. FIG. 7A is a flowchart showing the operation of the Web camera, and FIG. 7B is a flowchart showing details of steps S704 and S706 of the operation flowchart of FIG.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, embodiments of the invention will be described with reference to the drawings (Embodiment 1).
FIG. 2 shows the overall configuration of the multi-stream coding control apparatus according to the first embodiment of the present invention.

The multi-stream encoding control apparatus shown in FIG. 1 includes an input unit 201 that inputs a video signal to be encoded from the outside, and an encoding unit 202 that encodes the input video signal in accordance with a predetermined encoding method. Is provided. The encoding unit 202 includes encoding processing units 205 and 206 that encode an input video signal using two different encoding methods.

Further, the multi-stream encoding control apparatus shown in FIG. 1 includes an output unit 203 that outputs code data encoded by the encoding unit 202, and an encoding prediction unit 204. The encoding prediction unit 204 inputs the encoded data output speed of the output unit 203 and the encoding results of the two encoding processing units 205 and 206 of the encoding unit 202, and performs next encoding. In this case, the total code amount in the encoding processing units 205 and 206 is controlled.

Next, the operation when a video signal is input from the outside will be described based on the configuration diagram of FIG. 2 and the flowchart of FIG.

In step S301 of FIG. 3, an image input from the outside is acquired by the input unit 201. Next, in step S302, the acquired video signal is input to the encoding unit 202, and whether the desired encoding process is encoding in one encoding processing unit 205 or encoding in the other encoding processing unit 206 is performed. If the encoding processing unit 205 performs encoding, the process proceeds to step S303. If the encoding processing unit 206 performs encoding, the process proceeds to step S305.

Steps S303 and S304 and steps S305 and S306 respectively show the processing in the encoding processing units 205 and 206. In steps S303 and S305, the encoding prediction unit 204 outputs code data from the output unit 203. The speed, the sum of the two code amounts encoded by each encoding processing unit 205, 206 per unit time, the frame rate in each encoding process, the bit rate in each encoding process, Based on the encoding parameters set at the time of the previous encoding, the encoding processing units 205 and 206 determine the encoding parameters for the next encoding.

In steps S304 and S306, the input video signal is encoded according to the encoding method in the encoding processing units 205 and 206, respectively. Thereafter, in step S307, the result of encoding by the encoding processing units 205 and 206 is notified to the encoding prediction unit 204, and the encoding prediction unit 204 updates the held information based on the notification information. To do. In step S <b> 308, each code data encoded by each encoding processing unit 205, 206 is output from the output unit 203.

(Embodiment 2)
FIG. 4 shows a second embodiment of the present invention.

Figure 4 shows that two different video encoding formats are H.264. 2 shows a configuration of a Web camera equipped with a H.264 and MPEG4 multi-stream encoding control device.

The Web camera in FIG. 1 includes an image input unit 401 that acquires a video signal read from an image sensor that converts input video light into an electrical signal, YC generation and resizing to a desired image size. Part 402 and the obtained YC data. An encoding unit 403 including an encoding processing unit 406 that performs H.264 encoding and an encoding processing unit 407 that performs MPEG4 encoding, an output unit 404 that outputs the encoded code data to USB, and codes of the output unit 404 An encoding prediction unit 405 that inputs the data output speed and the encoding results of the encoding processing units 406 and 407 and controls the amount of code when the encoding processing units 406 and 407 perform encoding next time; And transmitting the code data from the output unit 404 to the connection destination host PC 408 via the USB cable.

Next, the operation when a video signal is input from the outside will be described with reference to the configuration diagram of FIG. 4 and the flowchart of FIG.

5A, in step S501, a video signal read from an image sensor that converts video light input from the outside into an electrical signal in the image input unit 401 is acquired. In step S502, the YC generation / resizing unit 402 regenerates the acquired video signal to YC generation and a desired image size. In the next step S503, the YC generated and resized image is input to the encoding unit 403, and the desired encoding processing unit 406 or 407 is selected.

Steps S504 and S505 and steps S506 and S507 indicate the encoding processes in the encoding processing units 406 and 407, respectively. In steps S504 and S506, the encoding prediction unit 405, based on the encoded data output speed of the USB output unit 404 and the encoding processing results of the encoding processing units 406 and 407, performs the encoding processing unit. In 406 and 407, encoding parameters for the next encoding are determined.

In steps S505 and S507, the encoding processing units 406 and 407 respectively encode the input image from the YC generation / resizing unit 402 in accordance with its own encoding method. In step S508, the results encoded by the encoding processing units 406 and 407 are notified to the encoding prediction unit 405, and the encoding prediction unit 405 determines the information held based on the notified information. Update. In step S509, the respective code data encoded by the encoding processing units 406 and 407 are output from the USB output unit 404 to the host PC 408.

FIG. 5B is a flowchart showing details of the encoding prediction process in steps S504 and S506 in FIG.

In the flowchart of FIG. 5B, in step S510, for example, the previous H.264 processing in the encoding processing unit 406 held in step S508 of FIG. The H.264 encoding result, the code amount, the Qp value (quantization parameter) that is the encoding parameter, and the next picture type are read out.

In step S511 and step S512, the previous encoding result (code amount, Qp value, next picture type) of the encoding processing unit 407 in the MPEG4 encoding process which is another encoding format is acquired. In step S513, the output bit rate and the total sum of output code amounts per unit time in the encoding processing units 406 and 407 are acquired as output information of the USB output unit 404.

In step S514, in order to determine the encoding parameter to be encoded next in each of the code processing units 406 and 407, the obtained output bit rate of the USB output unit 404, the total output code amount per unit time, Based on the above, the output possible code amount is calculated and, for example, the encoding processing unit 406 next outputs the H.264 code. When the amount of code per frame to be H.264 encoded is calculated and the next picture type is an I picture with a large amount of code, when the MPEG4 encoding process is also an I picture next in the encoding processing unit 407, Since the code amount per unit time increases, the encoding processing unit 407 selects the picture type to be MPEG4 encoded next as P or B picture, and then the previous code amount and the next code amount in the encoding processing unit 407 are selected. Are compared with the code amount per frame to be MPEG4 encoded, and if equal, the same Qp value as in the previous encoding, and if the previous code amount is smaller, the Qp value than in the previous encoding On the contrary, when the previous code amount is larger, the next encoding parameter Qp value is determined so as to perform control to increase the Qp value compared to the previous encoding.

(Embodiment 3)
FIG. 6 shows a third embodiment of the present invention.

The figure shows the video coding format H.264. 1 shows a configuration of a Web camera (camera system) equipped with a multi-stream encoding control device that uses H.264 and JPEG as a still image encoding format in two different encoding formats.

The web camera shown in FIG. 6 includes an image input unit 601 that acquires a video signal read from an image sensor that converts input video light into an electrical signal, and YC generation that resizes the image to a desired image size. The resizing unit 602 and the obtained YC data are converted into H.264. An encoding unit 603 including an encoding processing unit 606 that performs H.264 encoding, an encoding processing unit 607 that performs JPEG encoding, a USB output unit 604 that outputs the encoded code data via USB, and a USB output unit Encoding prediction unit for controlling the amount of code when the encoding data output speed of 604 and the encoding results of the encoding processing units 606 and 607 are input and the encoding processing units 606 and 607 perform encoding next time 605 and a dividing unit 608 that divides still image JPEG-encoded code data into fixed lengths and outputs the divided data to the output unit 604 at a fixed period. The USB output unit 604 sends a USB to the connection destination host PC 609. In this configuration, data is transmitted via a cable.

Next, the operation when a video signal is input from the outside will be described with reference to the configuration diagram of FIG. 6 and the flowchart of FIG.

7A, in step S701, a video signal read from an image sensor that converts video light input from the outside into an electrical signal in the image input unit 601 is acquired. In step S702, the YC generation / resizing unit 602 generates YC and resizes the acquired video signal to a desired image size. Thereafter, in step S703, the YC generated and resized image is input to the encoding unit 603, and a desired encoding process target is selected.

Steps S704 and S705 and steps S706 and S707 indicate processing in the encoding processing units 606 and 607, respectively. In steps S <b> 704 and S <b> 706, the encoding prediction unit 605 next performs each encoding process based on the encoded data output speed of the USB output unit 604 and the encoding results of the encoding processing units 606 and 607. Each encoding parameter at the time of encoding by the units 606 and 607 is determined.

In steps S705 and S707, the input image signal is encoded in accordance with the encoding method in each of the encoding processing units 606 and 607. In step S708, the encoding prediction unit 605 is notified of the encoding results of the encoding processing units 606 and 607, and the encoding prediction unit 605 updates the held information based on the notified information. . In step S707, information for dividing the JPEG-encoded data into a fixed length is notified to the dividing unit 608 at a fixed period.

Thereafter, in step S709, the encoded data encoded by the encoding processing units 606 and 607 is output from the USB output unit 604 to the host PC 609.

At this time, the still image code data encoded by the encoding processing unit 607 is divided into fixed lengths so as to be transferred within a desired time by the dividing unit 608, and is output to the USB output unit 604 at a constant cycle. .

FIG. 7B is a flowchart showing details of steps S704 and S706 in the flowchart of FIG. In the figure, in step S710, the H.D. stored in step S708 of FIG. The code amount, the Qp value, and the next picture type, which are the previous encoding results of the H.264 encoding process, are read out.

Thereafter, in step S711 and step S712, the previous encoding result (code amount, quantization table, scale factor) of the JPEG encoding format in the encoding processing unit 607 is acquired. In subsequent step S713, the output bit rate and the total output code amount per unit time of both encoding processing units 606 and 607 are acquired as output information of the USB output unit 604.

In step S714, in order to determine each encoding parameter to be encoded next in each encoding processing unit 606, 607, the acquired output bit rate, the total output code amount per unit time, and a constant JPEG Based on the output code amount of the period, the code amount that can be output is calculated. The H.264 encoding processing unit 606 calculates the code amount per frame to be encoded next, compares the previous code amount per frame with the code amount per frame to be encoded next, If the previous code amount is small, the Qp value is decreased. If the previous code amount is large, the next encoding parameter is determined so as to increase the Qp value. The JPEG encoding processing unit 607 calculates fixed length information used by the dividing unit 608 based on the previous code amount.

As described above, the multi-stream coding control apparatus of the present invention uses the output transfer speed of the output unit to use the full output band range, and suppresses the deterioration of the image quality of the moving image, thereby moving the high-quality moving image. Since images can be output in real time, it is suitable for use in, for example, network cameras such as surveillance cameras and WEB cameras, in-vehicle cameras, camera mounted on vehicles such as drive recorders, TV conference systems and TV phones connected to TVs and PCs, etc. It is.

201 Input unit 202 Encoding unit 203 Output unit 204 Encoding prediction unit 205, 206

Encoding processing unit

401, 601 Image input unit 402, 602 YC generation / resizing

unit

403, 603

Encoding unit

404, 604

USB output unit

405, 605 encoding prediction units 406 and 606 H.264. H.264 encoding processing unit 407 MPEG4

encoding processing unit

408, 609 Host PC
607 JPEG encoding processing unit 608 dividing unit

Claims

An image input unit for inputting a video signal;
An encoding unit that encodes the video signal input to the image input unit in accordance with two or more different encoding methods;
An output unit that outputs two or more types of encoded data encoded by the encoding unit;
An encoding prediction unit that determines a next encoding setting in the encoding unit based on an encoding result in the encoding unit;
The encoding prediction unit
Controlling the total amount of codes in the two or more different encoding schemes based on the encoding results in the two or more different encoding schemes and the encoding parameters set at the time of the encoding. A multi-stream coding control apparatus characterized by the above.
The multi-stream encoding control apparatus according to claim 1, wherein
The two or more different encoding schemes are two or more different video encoding schemes,
The encoding prediction unit
For each of the two or more different encoding schemes,
Code amount after encoding and encoding parameters;
Code amount and encoding parameter at the time of previous encoding than at the time of encoding, and
Encoding frame rate;
According to any one or more of the output rates of the output unit,
A multi-stream coding control apparatus characterized by determining a next coding parameter.
The multi-stream encoding control apparatus according to claim 1, wherein
The two or more different encoding methods are one or more still image encoding methods and one or more moving image encoding methods,
The encoding prediction unit
For each of the two or more different encoding schemes,
Code amount after encoding and encoding parameters;
Code amount and encoding parameter at the time of previous encoding than at the time of encoding, and
Encoding frame rate;
According to any one or more of the output rates of the output unit,
A multi-stream coding control apparatus characterized by determining a next coding parameter.
The multi-stream encoding control apparatus according to claim 2, wherein
Each of the two or more different video encoding methods has different encoding processing for each frame of the video signal,
The encoding prediction unit
Multi-stream coding control characterized by predicting that the coding amount varies due to different coding processing for each frame of the video signal in each coding method, and controlling the coding amount to be coded next apparatus.
The multi-stream encoding control apparatus according to claim 3, wherein
A dividing unit that divides the encoding result of the still image in the still image encoding method into a plurality of divisions,
The output unit notifies the encoded prediction unit of the divided data size,
The encoding prediction unit
A multi-stream encoding control apparatus that controls a code amount of a moving image encoding process in a moving image encoding method based on an output rate of the output unit and the divided data size.
The multi-stream encoding control apparatus according to claim 4,
The encoding prediction unit
Next, with respect to control of the code amount to be encoded, the multi-stream encoding control apparatus is characterized by holding information in which the code amount fluctuates periodically so that frames with a large code amount do not overlap.
A camera system comprising the multi-stream encoding control device according to any one of claims 1 to 6.