WO2024018525A1 - Video processing device, method, and program - Google Patents

Video processing device, method, and program Download PDF

Info

Publication number
WO2024018525A1
WO2024018525A1 PCT/JP2022/028052 JP2022028052W WO2024018525A1 WO 2024018525 A1 WO2024018525 A1 WO 2024018525A1 JP 2022028052 W JP2022028052 W JP 2022028052W WO 2024018525 A1 WO2024018525 A1 WO 2024018525A1
Authority
WO
WIPO (PCT)
Prior art keywords
quality
video
unit
video stream
stream
Prior art date
Application number
PCT/JP2022/028052
Other languages
French (fr)
Japanese (ja)
Inventor
淳一 中嶋
真二 深津
麻衣子 井元
馨亮 長谷川
Original Assignee
日本電信電話株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電信電話株式会社 filed Critical 日本電信電話株式会社
Priority to PCT/JP2022/028052 priority Critical patent/WO2024018525A1/en
Publication of WO2024018525A1 publication Critical patent/WO2024018525A1/en

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L47/00—Traffic control in data switching networks
    • H04L47/10—Flow control; Congestion control
    • H04L47/26—Flow control; Congestion control using explicit feedback to the source, e.g. choke packets
    • H04L47/265—Flow control; Congestion control using explicit feedback to the source, e.g. choke packets sent by intermediate network nodes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/60—Network structure or processes for video distribution between server and client or between remote clients; Control signalling between clients, server and network components; Transmission of management data between server and client, e.g. sending from server to client commands for recording incoming content stream; Communication details between server and client 
    • H04N21/63—Control signaling related to video distribution between client, server and network components; Network processes for video distribution between server and clients or between remote clients, e.g. transmitting basic layer and enhancement layers over different transmission paths, setting up a peer-to-peer communication via Internet between remote STB's; Communication protocols; Addressing
    • H04N21/647—Control signaling between network components and server or clients; Network processes for video distribution between server and clients, e.g. controlling the quality of the video stream, by dropping packets, protecting content from unauthorised alteration within the network, monitoring of network load, bridging between two different networks, e.g. between IP and wireless

Definitions

  • Embodiments of the present invention relate to a video processing device, method, and program.
  • MEC Multi-access Edge Computing
  • Non-patent documents 1 and 2 disclose examples in which processing distribution in MEC is utilized for video communication.
  • a GPU This is an example of generating a video stream with improved image quality using a Graphics Processing Unit (Graphics Processing Unit) server.
  • NVIDIA DEVELOPER SoftBank Solves Key Mobile Edge Computing Challenges Using NVIDIA Maxine”
  • NVIDIA Japan “Challenge to solve 5G issues by combining NVIDIA Maxine and MEC”
  • the present invention was made in view of the above circumstances, and its purpose is to provide a video processing device, method, and program that can appropriately improve the quality of a video stream. .
  • a video processing device includes: a receiving unit that receives a video stream; a determining unit that determines whether or not it is desirable to improve the quality of the video stream received by the receiving unit; and a quality improvement processing section that improves the quality of the video stream received by the reception section when the judgment section determines that it is desirable to improve the video stream.
  • a video processing method is a method performed by a video processing device, in which a video stream is received by a receiving unit of the video processing device, and a determining unit of the video processing device determines whether it is desirable to improve the quality of the received video stream, and when the determination unit determines that it is desirable to improve the quality of the video stream, the quality improvement processing unit of the video processing device determines whether the quality of the received video stream is desirable to improve. The quality of a video stream received by a receiving unit is improved.
  • the quality of a video stream can be appropriately improved.
  • FIG. 1 is a diagram showing a first application example of a video processing device according to an embodiment of the present invention.
  • FIG. 2 is a diagram showing a second application example of the video processing device according to an embodiment of the present invention.
  • FIG. 3 is a diagram showing a third application example of the video processing device according to an embodiment of the present invention.
  • FIG. 4 is a diagram showing a first example of processing results by each unit in the video processing device in a table format.
  • FIG. 5 is a diagram showing a second example of processing results by each unit in the video processing device in a table format.
  • FIG. 6 is a diagram showing a third example of processing results by each unit in the video processing device in a table format.
  • FIG. 7 is a diagram showing a fourth example of processing results by each unit in the video processing device in a table format.
  • FIG. 8 is a block diagram illustrating an example of the hardware configuration of a video processing device according to an embodiment of the present invention.
  • FIG. 1 is a diagram showing a first application example of a video processing device according to an embodiment of the present invention.
  • the video processing device 100 shown in FIG. 1 includes a stream receiving section 101, a quality improvement encoding section 103, and a stream transmitting section 104.
  • the stream receiving unit 101 receives a video stream from an external user terminal 201.
  • the quality determining unit 105 determines whether the quality of the stream is good or bad by measuring the quality of the stream received by the stream receiving unit 101 and comparing the measurement result with a threshold value.
  • the quality improvement encoding unit 103 improves the quality of the video stream and performs re-encoding processing on the video stream.
  • Stream transmitter 104 transmits the video stream to user terminal/server 202, which is an external network.
  • User terminal/server 202 is a user terminal or a server.
  • FIG. 2 is a diagram showing a second application example of the video processing device according to an embodiment of the present invention.
  • the video processing device 100 shown in FIG. 2 includes a stream receiving section 101, a video decoding section 102, a quality improvement encoding section 103, and a stream transmitting section 104.
  • a video stream has a packet structure, and is communicated as one packet every fixed short time period or every fixed amount of data.
  • a configuration in which a plurality of packets are processed at once may also be used.
  • Stream receiving unit 101, video decoding unit 102 The stream receiving unit 101 shown in FIGS. 1 and 2 receives encoded video stream packets communicated from the user terminal 201 via the communication network 203, and stores them in an internal buffer (see FIG. (See symbol a in 1 and 2).
  • the video decoding unit 102 shown in FIG. 2 converts the video stream received by the stream receiving unit 101 into pixel values based on communication session information and encoding information included in the video stream, which are transmitted and received separately. It decodes and sends the video data that is the result of the decoding to the quality improvement encoding unit 103 (see reference numeral a in FIG. 2).
  • the stream receiving unit 101 passes quality information to the quality determining unit 105 based on the status of the received stream.
  • quality information include (1) to (3) below.
  • the quality information passed to the quality determination unit 105 may be of one type or may be of multiple types.
  • the above-mentioned packet loss occurrence rate and reception bit rate are generally calculated as a time average, and are calculated as an average value from the reception status of the past one second, for example.
  • the stream receiving unit 101 Based on the instruction from the quality determining unit 105, the stream receiving unit 101 passes the received packet as is to the stream transmitting unit 104 without going through the quality improvement encoding unit 103, or passes the received packet or the video decoding unit The video data decoded by step 102 is passed to quality improvement encoding section 103.
  • Quality judgment unit 105 The internal memory of the quality determination unit 105 shown in FIGS. 1 and 2 stores a threshold value used for determining the quality of a stream. This threshold value may be a fixed value set in advance, or may be a variable value that is updated by some means while the video processing device 100 is in use.
  • the quality determining unit 105 compares the quality information acquired from the stream receiving unit 101 with the above-mentioned threshold value, and, for example, when the value of the quality information is equal to or greater than the threshold value, the quality of the stream is relatively good. When it is determined that the stream quality is "Q_P (Quality good)" and the value of the quality information is less than the threshold value, it can be determined that the quality of the stream is relatively poor "Q_P (Quality poor)".
  • the above-mentioned state in which the quality of the stream is relatively good means a state in which there is no particular need to improve the quality of the stream. Furthermore, the above-mentioned state where the quality of the stream is relatively poor means a state where it is necessary or desirable to improve the quality of the stream.
  • the quality determination unit 105 determines whether the value such as the packet loss occurrence rate is below the threshold value of the quality information.
  • the quality of the stream is determined to be the above "Q_G"
  • the quality of the stream is determined to be the above "Q_P”.
  • the quality determination unit 105 compares the value of the distribution status of each stream obtained from the stream transmission unit 104, for example, the number of distributions (sometimes referred to as the number of distributions or the number of simultaneous viewers) with the above threshold value. However, when the delivery status value is below the threshold, the quality of the stream is determined to be the above "Q_G", and when the delivery status value is above the threshold, the stream quality is determined to be the above "Q_P". It can be determined that
  • the quality determining unit 105 determines whether the quality of the stream is good or bad depending on the importance or popularity of each stream acquired from the stream transmitting unit 104. For example, in the case of a video stream that is distributed in large numbers, that is, is relatively popular, it is desirable to improve the image quality so that higher quality video can be viewed. The quality of is determined to be "Q_P".
  • the quality determining unit 105 sends an instruction to the stream receiving unit 101 to directly send packets of the stream to the stream transmitting unit 104. At this time, the quality determining unit 105 sends an instruction to the stream transmitting unit 104 to transmit the packet directly sent as described above from the stream receiving unit 101 to the external user terminal/server 202. On the other hand, if it is determined that the quality of the stream is the above-mentioned “Q_P”, the quality determining unit 105 sends an instruction to the stream receiving unit 101 to send the video data of the packet to the quality improvement encoding unit 103. At this time, the quality determination unit 105 instructs the stream transmission unit 104 to improve the image quality, that is, to send the encoded data after quality improvement output from the quality improvement encoding unit 103 to the external user terminal/server 202. send.
  • FIG. 3 is a diagram showing a third application example of the video processing device according to an embodiment of the present invention.
  • the configuration shown in FIG. 3 is a configuration in which the video processing device 100 can receive streams from each of a plurality of external user terminals 201, that is, a configuration in which multiple streams from the outside can be received.
  • 100 is provided with a plurality of pairs of stream receiving sections 101 and video decoding sections 102 (symbol a in FIG. 3).
  • Each of the pair of stream receiving section 101 and video decoding section 102 can receive and decode a stream from any one of the plurality of user terminals 201.
  • the quality determination unit 105 sends the stream whose quality determination result is “Q_P” to the quality improvement encoding unit 103 to the stream reception unit 101. send instructions to do so. Furthermore, the quality determining unit 105 sends an instruction to the stream receiving unit 101 to directly send a stream whose quality determination result is “Q_G” to the stream transmitting unit 104.
  • the quality determining unit 105 requests the stream receiving unit 101 to select the streams that can be processed by the quality improvement encoding unit 103 from among the plurality of streams whose quality determination result is “Q_P”. 103 and sends an instruction to send it.
  • the quality determining unit 105 causes the stream receiving unit 101 to directly send to the stream transmitting unit 104 the streams that cannot be processed by the quality improvement encoding unit 103 among the multiple streams whose quality determination result is “Q_P”. Send instructions.
  • Quality improvement encoding unit 103 improves the quality of video data obtained from stream reception section 101 or video decoding section 102.
  • Quality improvement processing is, for example, super-resolution processing that increases the resolution of the video, grain noise, or block noise that occurs when quantization is rough during encoding, that is, when the quantization parameter is large. Examples include filter processing to remove block noise.
  • super-resolution processing is "Maxine," a real-time video communication technology using AI (Artificial Intelligence) provided by NVIDIA Corporation of the United States. This is a technology that uses a GPU equipped with a deep learning model to increase the resolution of images, but the quality improvement method is not limited to this and other methods may also be used. .
  • the quality improvement encoding unit 103 performs re-encoding processing on the quality-improved video data using the same encoding method as the original stream, and transmits the encoded data generated by this re-encoding processing to the stream transmission unit. Pass it to 104.
  • re-encoding processing some encoded information is appropriately controlled so as to be the same as the original video data. This control uses, for example, a frame number or a time stamp indicating display timing, but is not limited to these.
  • the stream transmitter 104 transmits the stream packets directly sent from the stream receiver 101 to the external user terminal/server 202.
  • the stream transmitter 104 packetizes the encoded data sent from the quality improvement encoder 103 and sends it to the external user terminal/server 202.
  • the stream transmitting unit 104 appropriately controls some packet information so that there is no processing discrepancy between packets before and after the packet at the user terminal/server 202. This control may use, for example, a packet number, but is not limited to these.
  • FIG. 4 is a diagram showing a first example of processing results by each unit in the video processing device 100 in a table format.
  • the quality determination target by the quality determination unit 105 is the bit rate of the encoded video stream received by the stream reception unit 101.
  • bit rate of the video stream for the past 1 second at each time (sometimes referred to as the bit rate for the past 1 second) (Mbps)
  • the quality determination unit 105 based on this bit rate.
  • type of stream sent from the stream transmitter 104 (Mbps)
  • the threshold value for quality determination by the quality determination unit 105 is set to 1.0 (Mbps), and if the bit rate (BR) for the past 1 second (Mbps) ⁇ 1.0, the determination result is "Q_G” and BR If (Mbps) ⁇ 1.0, the determination result is "Q_P".
  • the quality is determined on a second-by-second basis, and it is determined whether image quality improvement processing is to be applied to video frames obtained from a single or multiple packets included in one second.
  • the determination result by the quality determining unit 105 is “Q_G”, and the type of stream sent from the stream transmitting unit 104 is is the raw input stream, that is, the raw stream received by the stream receiving unit 101.
  • the determination result by the quality determining unit 105 is "Q_P", and the stream transmitting unit 104 sends out the determination result.
  • the type of stream is an input stream that has been subjected to image quality improvement processing, that is, a stream that has been subjected to quality improvement processing by the quality improvement encoding unit 103 on a stream received by the stream reception unit 101.
  • FIG. 5 is a diagram showing a second example of processing results by each unit in the video processing device 100 in a table format.
  • Processing is performed using a fixed bit rate method that controls the encoding parameter values.
  • the quantization parameter is assigned a larger value as the video is more complex, ie, has more high frequency components, and a smaller value is assigned to the quantization parameter, as the video is simpler, ie, has fewer high frequency components.
  • the larger the quantization parameter the more quantization noise will occur, and the quality of the decoded video will deteriorate.
  • the target of quality determination by the quality determination unit 105 is the average quantization parameter P of a plurality of decoded videos, here a group of frames.
  • the threshold value for quality determination by the quality determination unit 105 is set to "10", and if the average quantization parameter P ⁇ 10, the determination result is "Q_G", and if P>10, the determination result is Suppose it is "Q_P".
  • the quality determination result is "Q_G"
  • the type of stream sent from the stream transmitter 104 is the input stream as it is. be.
  • FIG. 6 is a diagram showing a third example of processing results by each unit in the video processing device 100 in a table format.
  • the encoding unit of the user terminal 201 shown in FIG. 1 etc. performs control to avoid packet loss of the video stream as much as possible by varying the encoding bit rate based on the communication status of the communication network 203. Suppose there is.
  • the communication network between user A's user terminal 201 and video processing device 100 and the communication network between user B's user terminal 201 and video processing device 100 have a lower transmission band. It is assumed that the communication network between user C's user terminal 201 and the video processing device 100 has a large fluctuation in transmission band. In this example, it is assumed that the quality determination target by the quality determination unit 105 is the bit rate of the video encoded stream received from the user terminal of each user for the past one second.
  • bit rate (BR) Mbps
  • bit rate (BR) Mbps
  • bit rate (BR) Mbps
  • the quality judgment unit uses the bit rate (BR) (Mbps) of the video stream from the user terminal 201 of user B for the past 1 second. 105 is assumed to be 0.8 (Mbps).
  • the quality determination result is "Q_G"
  • the type of stream sent from the stream transmitter 104 is an input stream as is.
  • the quality determination result is "Q_P".
  • the type of stream sent out from the stream transmitting unit 104 is an input stream after image quality improvement processing, that is, a stream received by the stream receiving unit 101 is subjected to quality improvement processing by the quality improvement encoding unit 103. This is a stream.
  • the determination result by the quality determination unit 105 for the video stream received from the user terminal 201 of user A at times "T-2" and "T+1" is "Q_P".
  • the image quality improvement target user is user A.
  • the determination result by the quality determination unit 105 for the video stream received from user A's user terminal 201 and the determination result for the video stream received from user C's user terminal 201. Since the determination result by the quality determination unit 105 is "Q_P", the users A and C are the target users for image quality improvement.
  • FIG. 7 is a diagram showing a fourth example of processing results by each unit in the video processing device 100 in a table format.
  • the number of distributions that is, the number of simultaneous viewers, differs greatly between users A, B, and C.
  • users A, B, and C are performing live broadcasts at the same time, and the number of broadcasts varies greatly depending on the popularity of the broadcasters.
  • the quality judgment target by the quality judgment unit 105 is the number of encoded video streams distributed from the user terminal 201, and if this number is 50 or more, the judgment result is "Q_P", which is 50. If it is less than this, the determination result will be "Q_G".
  • the determination results by the quality determination unit 105 for the received video stream distributed from the user terminal 201 of user A are Since it is "Q_P", the user targeted for image quality improvement is "user A”.
  • FIG. 8 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment of the present invention.
  • the video processing device 100 is configured by, for example, a server computer or a personal computer, and includes a hardware processor 111A such as a CPU.
  • a program memory 111B, a data memory 112, an input/output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.
  • the communication interface 114 includes, for example, one or more wireless communication interface units, and enables information to be sent and received to and from a communication network.
  • a wireless interface for example, an interface adopting a low power wireless data communication standard such as a wireless LAN (Local Area Network) is used.
  • the input/output interface 113 is connected to an input device 200 and an output device 300 attached to the video processing apparatus 100 and used by a user or the like.
  • the input/output interface 113 receives operation data input by a user through an input device 200 such as a keyboard, touch panel, touchpad, mouse, etc., and outputs output data on a liquid crystal display or a mouse.
  • a process of outputting and displaying the image to an output device 300 including a display device using organic EL (Electro Luminescence) or the like is performed.
  • the input device 200 and the output device 300 may be a device built into the video processing device 100, or may be an input device of another information terminal that can communicate with the video processing device 100 via a communication network. Devices and output devices may also be used.
  • the program memory 111B is a non-temporary tangible storage medium, such as a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). It is used in combination with a nonvolatile memory such as a ROM (Read Only Memory), and stores programs necessary for executing various control processes and the like according to an embodiment.
  • a non-volatile memory such as a ROM (Read Only Memory)
  • ROM Read Only Memory
  • the data memory 112 is a tangible storage medium that uses a combination of the above-mentioned non-volatile memory and volatile memory such as RAM, and is acquired and created during various processes. It is used to store various types of data.
  • the video processing device 100 can be configured as a data processing device having each section shown in FIG. 1 and the like as a processing function section using software.
  • a storage device used as a working memory by each part of the video processing device 100 may be configured by using the data memory 112 shown in FIG. 8.
  • these configured storage areas are not essential configurations within the video processing device 100, and are, for example, external storage media such as a USB (Universal Serial Bus) memory, or a database server located in the cloud (cloud). It may also be an area provided in a storage device such as a database server.
  • the processing function units in each of the above units can be realized by causing the hardware processor 111A to read and execute a program stored in the program memory 111B. Note that some or all of these processing functions may be implemented in a variety of other formats, including integrated circuits such as application specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). It may be realized.
  • ASICs application specific integrated circuits
  • FPGAs field-programmable gate arrays
  • each embodiment can be applied to a magnetic disk (floppy (registered trademark) disk, hard disk) as a program (software means) that can be executed by a computer (computer). etc.), optical discs (CD-ROM, DVD, MO, etc.), semiconductor memories (ROM, RAM, Flash memory, etc.), and are stored in recording media, or transmitted and distributed via communication media. can be done.
  • the programs stored on the medium side also include a setting program for configuring software means (including not only execution programs but also tables and data structures) in the computer to be executed by the computer.
  • a computer that realizes this device reads a program recorded on a recording medium, and if necessary, constructs software means using a setting program, and executes the above-described processing by controlling the operation of the software means.
  • the recording medium referred to in this specification is not limited to one for distribution, and includes storage media such as a magnetic disk and a semiconductor memory provided inside a computer or in a device connected via a network.
  • the present invention is not limited to the above-described embodiments, and can be variously modified at the implementation stage without departing from the gist thereof.
  • each embodiment may be implemented in combination as appropriate, and in that case, the combined effect can be obtained.
  • the embodiments described above include various inventions, and various inventions can be extracted by combinations selected from the plurality of constituent features disclosed. For example, if a problem can be solved and an effect can be obtained even if some constituent features are deleted from all the constituent features shown in the embodiment, the configuration from which these constituent features are deleted can be extracted as an invention.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Multimedia (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

A video processing device according to one embodiment of the present invention comprises: a reception unit that receives a video stream; a determination unit that determines whether or not it is preferable to improve the quality of the video stream received by the reception unit; and a quality improvement processing unit that improves the quality of the video stream received by the reception unit when improving the quality of the video stream is determined to be preferable by the determination unit.

Description

映像処理装置、方法およびプログラムVideo processing device, method and program
　本発明の実施形態は、映像処理装置、方法およびプログラムに関する。 Embodiments of the present invention relate to a video processing device, method, and program.
　テレワーク（telework）におけるビデオ会議（video conference）、およびスマートフォン（smartphone）のアプリケーションプログラム（application program）（アプリ（App））を用いた、映像によるコミュニケーション（communication）の利用が増大している。スマートフォンのようなモバイル端末（mobile computer）（単に端末と称することがある）が利用される場合、電波状況によっては回線の帯域変動が起こるため、比較的高いビットレート（bit rate）での安定した通信が困難なことがある。 The use of video conferences in telework and video communication using smartphone application programs (Apps) is increasing. When a mobile computer (sometimes simply referred to as a device) such as a smartphone is used, the line bandwidth may fluctuate depending on the radio wave conditions, so it is difficult to maintain a stable bit rate at a relatively high bit rate. Communication may be difficult.
　仮に電波状況が良い状態でも、ビットレートを上げて高品質な映像通信を行なうには、端末による符号化演算処理に係る負荷が増大し、また、基地局へのデータ（data）の通信量が増大するといった、ユーザ（user）側のコスト（cost）増加に繋がってしまう。 Even if the radio wave conditions are good, increasing the bit rate and performing high-quality video communication will increase the load associated with encoding calculation processing on the terminal, and the amount of data communicated to the base station will increase. This leads to an increase in costs on the user side.
　一方、映像の解像度またはフレームレート（frame rate）を低くして映像品質を下げ、通信されるデータ量を低減させるといった手段もあるが、コミュニケーション自体の質に影響を与えてしまう問題が生じ得る。 On the other hand, there are methods to lower the video resolution or frame rate to lower the video quality and reduce the amount of data being communicated, but this may cause problems that affect the quality of the communication itself.
　近年では５Ｇ（5th Generation（第５世代移動通信システム））通信網と、この通信網に直結したエッジサーバ（edge server）とを連携させて、このエッジサーバに複雑な処理を行なわせることで、端末だけでなく、より、通信ネットワーク（network）上の比較的遠くに位置する、インターネット（internet）上のサーバでの処理量または通信量を分散または低減させるＭＥＣ（Multi-access Edge Computing）技術の商用化が進みつつある。 In recent years, by linking a 5G (5th Generation mobile communication system) communication network and an edge server directly connected to this communication network, and having this edge server perform complex processing, MEC (Multi-access Edge Computing) technology that distributes or reduces the amount of processing or communication not only at terminals but also at servers on the Internet that are located relatively far away on the communication network. Commercialization is progressing.
　このＭＥＣにおける処理分散が映像コミュニケーションに活用された事例が、非特許文献１および２に開示されている。この事例は、低ビットレートでアップロード（upload）された映像データである映像ストリーム（stream）（映像符号化ストリーム、または単にストリームと称することがある）の解像度を、エッジサーバに配置されたＧＰＵ（Graphics Processing Unit）サーバを利用して向上させて画質改善を行なった映像ストリームを生成する一例である。 Non-patent documents 1 and 2 disclose examples in which processing distribution in MEC is utilized for video communication. In this case, a GPU ( This is an example of generating a video stream with improved image quality using a Graphics Processing Unit (Graphics Processing Unit) server.
　しかしながら、ＧＰＵサーバでの画質改善は演算処理のコストの増大を招くため、ユーザの通信状況が比較的良好な場合には、常に画質改善処理を行なう必要はない。　
　また、ＭＥＣによる分散処理を行なうためには、基地局またはコアネットワーク（core network）に位置するゲートウェイ（gateway）などに配置されるエッジサーバの数が膨大になり、５Ｇ通信の事業者であるキャリア（carrier）としては、設置する計算機の資源を出来るだけ絞り込む、または有効に利用することが望ましい。 However, since improving the image quality on the GPU server increases the cost of arithmetic processing, it is not necessary to always perform the image quality improvement process when the user's communication status is relatively good.
In addition, in order to perform distributed processing using MEC, the number of edge servers installed at base stations or gateways located in the core network will be enormous, and carriers that operate 5G communications will As a carrier, it is desirable to narrow down or effectively use the resources of the installed computers as much as possible.
　そのため、ユーザ当たりの使用率、すなわちＧＰＵの処理時間を低下させ、より多くのユーザで共有して演算処理に用いるなどの、計算機資源の共有度合いを向上させることが望ましい。 Therefore, it is desirable to reduce the usage rate per user, that is, the processing time of the GPU, and to improve the degree of sharing of computer resources, such as sharing them among more users and using them for arithmetic processing.
　この発明は、上記事情に着目してなされたもので、その目的とするところは、映像ストリームの品質を適切に改善することができるようにした映像処理装置、方法およびプログラムを提供することにある。 The present invention was made in view of the above circumstances, and its purpose is to provide a video processing device, method, and program that can appropriately improve the quality of a video stream. .
　本発明の一態様に係る映像処理装置は、映像ストリームを受信する受信部と、前記受信部により受信した映像ストリームの品質の向上が望ましいか否かを判定する判定部と、前記映像ストリームの品質の向上が望ましいと前記判定部により判定されたときに、前記受信部により受信した映像ストリームの品質を向上させる品質向上処理部と、を備える。 A video processing device according to one aspect of the present invention includes: a receiving unit that receives a video stream; a determining unit that determines whether or not it is desirable to improve the quality of the video stream received by the receiving unit; and a quality improvement processing section that improves the quality of the video stream received by the reception section when the judgment section determines that it is desirable to improve the video stream.
　本発明の一態様に係る映像処理方法は、映像処理装置により行なわれる方法であって、前記映像処理装置の受信部により、映像ストリームを受信し、前記映像処理装置の判定部により、前記受信部により受信した映像ストリームの品質の向上が望ましいか否かを判定し、前記映像処理装置の品質向上処理部により、前記映像ストリームの品質の向上が望ましいと前記判定部により判定されたときに、前記受信部により受信した映像ストリームの品質を向上させる。 A video processing method according to one aspect of the present invention is a method performed by a video processing device, in which a video stream is received by a receiving unit of the video processing device, and a determining unit of the video processing device determines whether it is desirable to improve the quality of the received video stream, and when the determination unit determines that it is desirable to improve the quality of the video stream, the quality improvement processing unit of the video processing device determines whether the quality of the received video stream is desirable to improve. The quality of a video stream received by a receiving unit is improved.
　本発明によれば、映像ストリームの品質を適切に改善することができる。 According to the present invention, the quality of a video stream can be appropriately improved.
図１は、本発明の一実施形態に係る映像処理装置の第１の適用例を示す図である。FIG. 1 is a diagram showing a first application example of a video processing device according to an embodiment of the present invention. 図２は、本発明の一実施形態に係る映像処理装置の第２の適用例を示す図である。FIG. 2 is a diagram showing a second application example of the video processing device according to an embodiment of the present invention. 図３は、本発明の一実施形態に係る映像処理装置の第３の適用例を示す図である。FIG. 3 is a diagram showing a third application example of the video processing device according to an embodiment of the present invention. 図４は、映像処理装置内の各部による処理結果の第１の例を表形式で示す図である。FIG. 4 is a diagram showing a first example of processing results by each unit in the video processing device in a table format. 図５は、映像処理装置内の各部による処理結果の第２の例を表形式で示す図である。FIG. 5 is a diagram showing a second example of processing results by each unit in the video processing device in a table format. 図６は、映像処理装置内の各部による処理結果の第３の例を表形式で示す図である。FIG. 6 is a diagram showing a third example of processing results by each unit in the video processing device in a table format. 図７は、映像処理装置内の各部による処理結果の第４の例を表形式で示す図である。FIG. 7 is a diagram showing a fourth example of processing results by each unit in the video processing device in a table format. 図８は、本発明の一実施形態に係る映像処理装置のハードウエア（hardware）構成の一例を示すブロック図（block Diagram）である。FIG. 8 is a block diagram illustrating an example of the hardware configuration of a video processing device according to an embodiment of the present invention.
　以下、図面を参照しながら、この発明に係わる一実施形態を説明する。　
　本発明の一実施形態では、ユーザの通信状況に応じてＧＰＵサーバの処理を制御することで、第１に、一人当たりのＧＰＵサーバでの演算処理を低減して使用電力を削減し、第２に、ＧＰＵサーバ当たりの割り当てられるユーザを増やす効果が得られる。 Hereinafter, one embodiment of the present invention will be described with reference to the drawings.
In one embodiment of the present invention, by controlling the processing of the GPU server according to the communication status of the user, firstly, the calculation processing on the GPU server per person is reduced and power consumption is reduced; In addition, the effect of increasing the number of users assigned to each GPU server can be obtained.
　一方、サービス（service）の形態としては、常に画質改善処理を行なうオプション（option）を用意し、利用者の端末からは比較的低いビットレートで映像を送信しつつも、通信先には高画質映像を表示できるなど、柔軟な利用手段を提供できる。 On the other hand, as a form of service, we have an option that constantly performs image quality improvement processing, so that while the user's terminal is transmitting video at a relatively low bit rate, it is still sent to the communication destination in high quality. It can provide flexible usage methods such as the ability to display images.
　図１は、本発明の一実施形態に係る映像処理装置の第１の適用例を示す図である。　
　図１に示される映像処理装置１００は、ストリーム受信部１０１、品質向上符号化部１０３、およびストリーム送信部１０４を備える。 FIG. 1 is a diagram showing a first application example of a video processing device according to an embodiment of the present invention.
The video processing device 100 shown in FIG. 1 includes a stream receiving section 101, a quality improvement encoding section 103, and a stream transmitting section 104.
　ストリーム受信部１０１は、外部のユーザ端末２０１から映像ストリームを受信する。　
　品質判定部１０５は、ストリーム受信部１０１により受信したストリームの品質を測定し、測定結果を閾値と比較することで、ストリームの品質の良し悪しを判定する。　
　品質向上符号化部１０３は、映像ストリームの品質を改善し、映像ストリームへの再符号化処理を行なう。　
　ストリーム送信部１０４は、映像ストリームを外部ネットワークであるユーザ端末／サーバ２０２に送出する。ユーザ端末／サーバ２０２はユーザ端末またはサーバである。 The stream receiving unit 101 receives a video stream from an external user terminal 201.
The quality determining unit 105 determines whether the quality of the stream is good or bad by measuring the quality of the stream received by the stream receiving unit 101 and comparing the measurement result with a threshold value.
The quality improvement encoding unit 103 improves the quality of the video stream and performs re-encoding processing on the video stream.
Stream transmitter 104 transmits the video stream to user terminal/server 202, which is an external network. User terminal/server 202 is a user terminal or a server.
　図２は、本発明の一実施形態に係る映像処理装置の第２の適用例を示す図である。　
　図２に示される映像処理装置１００は、ストリーム受信部１０１、映像復号部１０２、品質向上符号化部１０３、およびストリーム送信部１０４を備える。 FIG. 2 is a diagram showing a second application example of the video processing device according to an embodiment of the present invention.
The video processing device 100 shown in FIG. 2 includes a stream receiving section 101, a video decoding section 102, a quality improvement encoding section 103, and a stream transmitting section 104.
　次に、図１および図２に示される映像処理装置１００内の各部の詳細について説明する。　
　一般的に、映像ストリームはパケット（packet）構造を有し、一定の短い時間ごと、もしくは一定のデータ量ごとに一つのパケットとして通信される。以下では、各処理がパケット単位で行われる例を説明するが、複数のパケットが一纏めに処理される構成でも良い。 Next, details of each part in the video processing device 100 shown in FIGS. 1 and 2 will be described.
Generally, a video stream has a packet structure, and is communicated as one packet every fixed short time period or every fixed amount of data. Although an example in which each process is performed in units of packets will be described below, a configuration in which a plurality of packets are processed at once may also be used.
　「ストリーム受信部１０１、映像復号部１０２」
　図１および図２に示されるストリーム受信部１０１は、ユーザ端末２０１から通信ネットワーク２０３を経由して通信された、符号化された映像ストリームのパケットを受信して、内部のバッファに格納する（図１、２の符号ａ参照）。 “Stream receiving unit 101, video decoding unit 102”
The stream receiving unit 101 shown in FIGS. 1 and 2 receives encoded video stream packets communicated from the user terminal 201 via the communication network 203, and stores them in an internal buffer (see FIG. (See symbol a in 1 and 2).
　図２に示される映像復号部１０２は、別途送受信している、通信セッション（session）の情報、および映像ストリームに含まれる符号化情報に基づき、ストリーム受信部１０１により受信した映像ストリームを画素値に復号し、復号の結果である映像データを品質向上符号化部１０３へ送出する（図２の符号ａ参照）。 The video decoding unit 102 shown in FIG. 2 converts the video stream received by the stream receiving unit 101 into pixel values based on communication session information and encoding information included in the video stream, which are transmitted and received separately. It decodes and sends the video data that is the result of the decoding to the quality improvement encoding unit 103 (see reference numeral a in FIG. 2).
　ストリーム受信部１０１は、受信したストリームの状況を基に、品質判定部１０５へ品質情報を渡す。品質情報は、例えば以下の（１）～（３）が挙げられる。品質判定部１０５に渡される品質情報の種類は一つでも良いし、複数の種類でも良い。 The stream receiving unit 101 passes quality information to the quality determining unit 105 based on the status of the received stream. Examples of quality information include (1) to (3) below. The quality information passed to the quality determination unit 105 may be of one type or may be of multiple types.
　（１）　通信路での伝送のエラー（error）によるパケットロスト（packet lost）の発生率
　（２）　ビットレート（符号化ビットレート、あるいはストリームの受信ビットレート）
　（３）　ストリームから復号あるいは別途重畳されて受信した、映像符号化時のビットレートの調整に用いられる量子化パラメータ（quantization parameter） (1) Occurrence rate of packet lost due to transmission errors on the communication channel (2) Bit rate (encoding bit rate or stream reception bit rate)
(3) Quantization parameter used to adjust the bit rate during video encoding, decoded from the stream or separately superimposed and received
　上記のパケットロストの発生率、および受信ビットレートは、時間平均で求められることが一般的であり、例えば過去１秒間の受信状況から平均値として算出される。 The above-mentioned packet loss occurrence rate and reception bit rate are generally calculated as a time average, and are calculated as an average value from the reception status of the past one second, for example.
　ストリーム受信部１０１は、品質判定部１０５からの指示に基づき、上記受信したパケットを、品質向上符号化部１０３を介さずに、そのままストリーム送信部１０４に渡す、または上記受信したパケットもしくは映像復号部１０２により復号した映像データを品質向上符号化部１０３に渡す。 Based on the instruction from the quality determining unit 105, the stream receiving unit 101 passes the received packet as is to the stream transmitting unit 104 without going through the quality improvement encoding unit 103, or passes the received packet or the video decoding unit The video data decoded by step 102 is passed to quality improvement encoding section 103.
　「品質判定部１０５」
　図１および図２に示される品質判定部１０５の内部メモリには、ストリームの品質の判定に用いられる閾値が記憶される。この閾値は、あらかじめ設定された固定値でも良いし、映像処理装置１００の使用中に何らかの手段により更新される可変値でも良い。 “Quality judgment unit 105”
The internal memory of the quality determination unit 105 shown in FIGS. 1 and 2 stores a threshold value used for determining the quality of a stream. This threshold value may be a fixed value set in advance, or may be a variable value that is updated by some means while the video processing device 100 is in use.
　品質判定部１０５は、ストリーム受信部１０１から取得した品質情報と、上記の閾値とを比較し、例えば、品質情報の値が閾値以上であるときは、ストリームの品質が比較的良い状態「Q_G（Quality good）」であると判定し、品質情報の値が閾値を下回ったときは、ストリームの品質が比較的悪い状態「Q_P（Quality poor）」であると判定することができる。 The quality determining unit 105 compares the quality information acquired from the stream receiving unit 101 with the above-mentioned threshold value, and, for example, when the value of the quality information is equal to or greater than the threshold value, the quality of the stream is relatively good. When it is determined that the stream quality is "Q_P (Quality good)" and the value of the quality information is less than the threshold value, it can be determined that the quality of the stream is relatively poor "Q_P (Quality poor)".
　上記のストリームの品質が比較的良い状態は、ストリームの品質の向上を特に要しない状態を意味する。また、上記のストリームの品質が比較的悪い状態は、ストリームの品質の向上を要する、または望ましい状態を意味する。 The above-mentioned state in which the quality of the stream is relatively good means a state in which there is no particular need to improve the quality of the stream. Furthermore, the above-mentioned state where the quality of the stream is relatively poor means a state where it is necessary or desirable to improve the quality of the stream.
　なお、上記のパケットロストの発生率などの値が低いときは品質が比較的良いとされるので、このとき、品質判定部１０５は、パケットロストの発生率などの値が品質情報の閾値を下回ったときは、ストリームの品質が上記「Q_G」であると判定し、パケットロストの発生率などの値が閾値以上であるときは、ストリームの品質が上記「Q_P」であると判定することができる。 Note that when the above-mentioned values such as the packet loss occurrence rate are low, the quality is considered to be relatively good, so in this case, the quality determination unit 105 determines whether the value such as the packet loss occurrence rate is below the threshold value of the quality information. When the quality of the stream is determined to be the above "Q_G", and when the value such as the occurrence rate of packet loss is equal to or higher than the threshold value, the quality of the stream is determined to be the above "Q_P". .
　また、品質判定部１０５は、ストリーム送信部１０４から取得した各ストリームの配信状況の値、例えば配信本数（配信数、または同時視聴者数などと称することがある）と、上記の閾値とを比較し、配信状況の値が閾値以下であるときは、ストリームの品質が上記「Q_G」であると判定し、配信状況の値が閾値を上回ったときは、ストリームの品質が上記「Q_P」であると判定することができる。　
　ここでは、上記と異なり、品質判定部１０５は、ストリーム送信部１０４から取得した各ストリームの重要度または人気具合などに応じてストリームの品質の良し悪しを判定している。例えば、配信本数が多い、すなわち人気が比較的高い映像ストリームは、画質を向上させて、より高品質の映像を見られるようにすることが望ましいとして、品質判定部１０５により、上記のようにストリームの品質が「Q_P」であると判定させる。 In addition, the quality determination unit 105 compares the value of the distribution status of each stream obtained from the stream transmission unit 104, for example, the number of distributions (sometimes referred to as the number of distributions or the number of simultaneous viewers) with the above threshold value. However, when the delivery status value is below the threshold, the quality of the stream is determined to be the above "Q_G", and when the delivery status value is above the threshold, the stream quality is determined to be the above "Q_P". It can be determined that
Here, unlike the above, the quality determining unit 105 determines whether the quality of the stream is good or bad depending on the importance or popularity of each stream acquired from the stream transmitting unit 104. For example, in the case of a video stream that is distributed in large numbers, that is, is relatively popular, it is desirable to improve the image quality so that higher quality video can be viewed. The quality of is determined to be "Q_P".
　上記処理により、ストリームの品質が上記「Q_G」であると判定した場合は、品質判定部１０５は、ストリーム受信部１０１に対し、当該ストリームのパケットをストリーム送信部１０４に直接送る指示を送る。　
　このとき、品質判定部１０５は、ストリーム送信部１０４に対し、ストリーム受信部１０１から上記のように直接送られたパケットを外部のユーザ端末／サーバ２０２に送出する指示を送る。　
　一方、ストリームの品質が上記「Q_P」であると判定した場合は、品質判定部１０５は、ストリーム受信部１０１に対し、当該パケットの映像データを品質向上符号化部１０３に送出する指示を送る。　
　このとき、品質判定部１０５は、ストリーム送信部１０４に対し、品質向上符号化部１０３から出力された、画質向上、すなわち品質向上後の符号化データを外部のユーザ端末／サーバ２０２に送出する指示を送る。 If the quality of the stream is determined to be the above-mentioned “Q_G” through the above process, the quality determining unit 105 sends an instruction to the stream receiving unit 101 to directly send packets of the stream to the stream transmitting unit 104.
At this time, the quality determining unit 105 sends an instruction to the stream transmitting unit 104 to transmit the packet directly sent as described above from the stream receiving unit 101 to the external user terminal/server 202.
On the other hand, if it is determined that the quality of the stream is the above-mentioned “Q_P”, the quality determining unit 105 sends an instruction to the stream receiving unit 101 to send the video data of the packet to the quality improvement encoding unit 103.
At this time, the quality determination unit 105 instructs the stream transmission unit 104 to improve the image quality, that is, to send the encoded data after quality improvement output from the quality improvement encoding unit 103 to the external user terminal/server 202. send.
　図３は、本発明の一実施形態に係る映像処理装置の第３の適用例を示す図である。　
　図３に示された構成は、外部の複数のユーザ端末２０１の各々からのストリームを映像処理装置１００により受信可能な構成、すなわち外部からの複数のストリームを受信可能な構成であり、映像処理装置１００には、ストリーム受信部１０１と映像復号部１０２の複数の組（図３の符号ａ）が設けられる。　
　ストリーム受信部１０１と映像復号部１０２の組の各々は、複数のユーザ端末２０１のいずれかからのストリームを受信および復号化することができる。 FIG. 3 is a diagram showing a third application example of the video processing device according to an embodiment of the present invention.
The configuration shown in FIG. 3 is a configuration in which the video processing device 100 can receive streams from each of a plurality of external user terminals 201, that is, a configuration in which multiple streams from the outside can be received. 100 is provided with a plurality of pairs of stream receiving sections 101 and video decoding sections 102 (symbol a in FIG. 3).
Each of the pair of stream receiving section 101 and video decoding section 102 can receive and decode a stream from any one of the plurality of user terminals 201.
　次に、ストリーム受信部１０１により受信されたストリームの数が複数であるときの制御について以下の（１）および（２）で説明する。　
　（１）　受信されストリームが単一のストリームである場合と同様に、品質判定部１０５は、ストリーム受信部１０１に対し、品質判定結果が「Q_P」であるストリームを品質向上符号化部１０３に送出する指示を送る。また、品質判定部１０５は、ストリーム受信部１０１に対し、品質判定結果が「Q_G」であるストリームをストリーム送信部１０４に直接送る指示を送る。 Next, control when a plurality of streams are received by the stream receiving unit 101 will be described in (1) and (2) below.
(1) As in the case where the received stream is a single stream, the quality determination unit 105 sends the stream whose quality determination result is “Q_P” to the quality improvement encoding unit 103 to the stream reception unit 101. send instructions to do so. Furthermore, the quality determining unit 105 sends an instruction to the stream receiving unit 101 to directly send a stream whose quality determination result is “Q_G” to the stream transmitting unit 104.
　（２）　品質判定部１０５は、ストリーム受信部１０１に対し、品質判定結果が「Q_P」である複数のストリームのうち、品質向上符号化部１０３で処理できる分量のストリームを当該品質向上符号化部１０３に送出する指示を送る。　
　一方で品質判定部１０５は、ストリーム受信部１０１に対し、品質判定結果が「Q_P」である複数のストリームのうち、品質向上符号化部１０３で処理できない分量のストリームをストリーム送信部１０４に直接送る指示を送る。 (2) The quality determining unit 105 requests the stream receiving unit 101 to select the streams that can be processed by the quality improvement encoding unit 103 from among the plurality of streams whose quality determination result is “Q_P”. 103 and sends an instruction to send it.
On the other hand, the quality determining unit 105 causes the stream receiving unit 101 to directly send to the stream transmitting unit 104 the streams that cannot be processed by the quality improvement encoding unit 103 among the multiple streams whose quality determination result is “Q_P”. Send instructions.
　「品質向上符号化部１０３」
　品質向上符号化部１０３は、ストリーム受信部１０１または映像復号部１０２から取得した映像データの品質改善を行なう。品質改善の処理は、例えば、映像の解像度を増加させる超解像度処理、あるいは粒子状のグレイン（grain）ノイズ、または符号化時の量子化が荒い、すなわち量子化パラメータが大きいときに発生するブロックノイズ（block noise）を除去するフィルタ（filter）処理などが挙げられる。　
　上記の超解像度処理の一手段としては、例えば米NVIDIA Corporationが提供する、ＡＩ（Artificial Intelligence）を使ったリアルタイム（real time）用の映像コミュニケーション用技術の「Maxine」などが挙げられる。これは、ディープラーニングモデル（deep learning model）が搭載されたＧＰＵを利用して画像の解像度を上げる技術であるが、品質改善の手段については、これに限らず他の手段が用いられても良い。 “Quality improvement encoding unit 103”
Quality improvement encoding section 103 improves the quality of video data obtained from stream reception section 101 or video decoding section 102. Quality improvement processing is, for example, super-resolution processing that increases the resolution of the video, grain noise, or block noise that occurs when quantization is rough during encoding, that is, when the quantization parameter is large. Examples include filter processing to remove block noise.
One example of the above-mentioned super-resolution processing is "Maxine," a real-time video communication technology using AI (Artificial Intelligence) provided by NVIDIA Corporation of the United States. This is a technology that uses a GPU equipped with a deep learning model to increase the resolution of images, but the quality improvement method is not limited to this and other methods may also be used. .
　品質向上符号化部１０３は、品質改善された映像データに対し、元のストリームと同一の符号化方式により再符号化処理を行ない、この再符号化処理により生成された符号化データをストリーム送信部１０４に渡す。再符号化処理が行なわれる際に、いくつかの符号化情報は、元の映像データと同じとなるように適宜制御される。この制御では、例えば、フレーム番号、または表示タイミング（timing）を表すタイムスタンプ（time stamp）などが用いられるが、これらに限られない。 The quality improvement encoding unit 103 performs re-encoding processing on the quality-improved video data using the same encoding method as the original stream, and transmits the encoded data generated by this re-encoding processing to the stream transmission unit. Pass it to 104. When re-encoding processing is performed, some encoded information is appropriately controlled so as to be the same as the original video data. This control uses, for example, a frame number or a time stamp indicating display timing, but is not limited to these.
　「ストリーム送信部１０４」
　ストリーム送信部１０４は、ストリーム受信部１０１から直接送られたストリームのパケットを外部のユーザ端末／サーバ２０２に送出する。　
　もしくは、ストリーム送信部１０４」は、品質向上符号化部１０３から送られた符号化データをパケット化して、外部のユーザ端末／サーバ２０２に送出する。上記パケット化の際には、ストリーム送信部１０４は、ユーザ端末／サーバ２０２において、パケット前後のパケットとの間で処理の齟齬が生じないように、いくつかのパケット情報を適宜制御する。この制御では、例えば、パケット番号などが用いられ得るが、これらに限られない。 “Stream transmission unit 104”
The stream transmitter 104 transmits the stream packets directly sent from the stream receiver 101 to the external user terminal/server 202.
Alternatively, the stream transmitter 104 packetizes the encoded data sent from the quality improvement encoder 103 and sends it to the external user terminal/server 202. During the packetization, the stream transmitting unit 104 appropriately controls some packet information so that there is no processing discrepancy between packets before and after the packet at the user terminal/server 202. This control may use, for example, a packet number, but is not limited to these.
　次に、映像処理装置１００内の各部による処理結果の具体的な例について説明する。図４は、映像処理装置１００内の各部による処理結果の第１の例を表形式で示す図である。　
　この例では、図１などに示される通信ネットワーク２０３の伝送帯域が変動し易い環境において、ユーザ端末２０１の図示しない符号化部が、通信ネットワーク２０３の通信状況に基づき符号化ビットレートを変動させて、映像ストリームのパケットロストを極力避けるような制御を行なっているとする。　
　この例での、品質判定部１０５による品質の判定対象は、ストリーム受信部１０１により受信された映像符号化ストリームのビットレートであるとする。 Next, a specific example of processing results by each unit in the video processing device 100 will be described. FIG. 4 is a diagram showing a first example of processing results by each unit in the video processing device 100 in a table format.
In this example, in an environment where the transmission band of the communication network 203 is likely to fluctuate as shown in FIG. , it is assumed that control is being performed to avoid packet loss of the video stream as much as possible.
In this example, it is assumed that the quality determination target by the quality determination unit 105 is the bit rate of the encoded video stream received by the stream reception unit 101.
　図４では、（１）各時刻における過去１秒間の映像ストリームのビットレート（過去１秒間のビットレートと称することがある）（Mbps）と、（２）このビットレートに基づく、品質判定部１０５による判定結果と、（３）ストリーム送信部１０４から送出されるストリームの種別、が示される。 In FIG. 4, (1) the bit rate of the video stream for the past 1 second at each time (sometimes referred to as the bit rate for the past 1 second) (Mbps), and (2) the quality determination unit 105 based on this bit rate. and (3) the type of stream sent from the stream transmitter 104.
　この例では、品質判定部１０５による品質の判定に係る閾値を1.0（Mbps）とし、上記過去１秒間のビットレート（BR）（Mbps）≧1.0であれば判定結果は「Q_G」であり、BR（Mbps）＜1.0であれば判定結果は「Q_P」であるとする。　
　この例では、品質の判定が秒単位で行なわれ、１秒間に含まれる単一または複数のパケットから得られる映像のフレームに対し、画質向上処理の適用または非適用が判定される。 In this example, the threshold value for quality determination by the quality determination unit 105 is set to 1.0 (Mbps), and if the bit rate (BR) for the past 1 second (Mbps) ≧1.0, the determination result is "Q_G" and BR If (Mbps)<1.0, the determination result is "Q_P".
In this example, the quality is determined on a second-by-second basis, and it is determined whether image quality improvement processing is to be applied to video frames obtained from a single or multiple packets included in one second.
　図４に示された例では、過去１秒間のビットレート（Mbps）が1.0以上のときは、品質判定部１０５による判定結果は「Q_G」であり、ストリーム送信部１０４から送出されるストリームの種別は、そのままの入力ストリーム、すなわちストリーム受信部１０１により受信された、そのままのストリームである。 In the example shown in FIG. 4, when the bit rate (Mbps) for the past one second is 1.0 or more, the determination result by the quality determining unit 105 is “Q_G”, and the type of stream sent from the stream transmitting unit 104 is is the raw input stream, that is, the raw stream received by the stream receiving unit 101.
　一方で、図４に示された例では、過去１秒間のビットレート（Mbps）が1.0未満のときは、品質判定部１０５による判定結果は「Q_P」であり、ストリーム送信部１０４から送出されるストリームの種別は、画質向上処理後の入力ストリーム、すなわちストリーム受信部１０１により受信されたストリームに対して品質向上符号化部１０３による品質向上の処理がなされたストリームである。 On the other hand, in the example shown in FIG. 4, when the bit rate (Mbps) for the past one second is less than 1.0, the determination result by the quality determining unit 105 is "Q_P", and the stream transmitting unit 104 sends out the determination result. The type of stream is an input stream that has been subjected to image quality improvement processing, that is, a stream that has been subjected to quality improvement processing by the quality improvement encoding unit 103 on a stream received by the stream reception unit 101.
　図５は、映像処理装置１００内の各部による処理結果の第２の例を表形式で示す図である。　
　この例では、図１などに示される通信ネットワーク２０３の伝送帯域の変動が比較的少ない環境において、ユーザ端末２０１の符号化部は、符号化ビットレートが、ほぼ一定の範囲内に収まるように量子化パラメータ値を制御する、固定ビットレート方式を用いた処理を行なう。 FIG. 5 is a diagram showing a second example of processing results by each unit in the video processing device 100 in a table format.
In this example, in an environment where there are relatively few fluctuations in the transmission band of the communication network 203 as shown in FIG. Processing is performed using a fixed bit rate method that controls the encoding parameter values.
　このとき量子化パラメータは、映像が複雑である、すなわち高周波成分が多いほど大きく、映像が単純である、すなわち高周波成分が少ないほど小さい値が割り当てられる。　
　一般的に、量子化パラメータが大きいほど、量子化ノイズの発生が多くなり、復号した映像の画質は低下する。 At this time, the quantization parameter is assigned a larger value as the video is more complex, ie, has more high frequency components, and a smaller value is assigned to the quantization parameter, as the video is simpler, ie, has fewer high frequency components.
Generally, the larger the quantization parameter, the more quantization noise will occur, and the quality of the decoded video will deteriorate.
　なお、符号化方式により、量子化パラメータの範囲は異なるため、ここでは量子化パラメータの値は「１」から「３１」の間であるとする。　
　この例では、品質判定部１０５による品質の判定対象は、復号した複数の映像、ここではフレーム群の平均量子化パラメータＰ、であるとする。
　図５では、（１）各フレーム群における平均量子化パラメータＰの値と、このビットレートに基づく、品質判定部１０５による判定結果と、（３）ストリーム送信部１０４から送出されるストリームの種別、が示される。 Note that since the range of the quantization parameter differs depending on the encoding method, it is assumed here that the value of the quantization parameter is between "1" and "31".
In this example, it is assumed that the target of quality determination by the quality determination unit 105 is the average quantization parameter P of a plurality of decoded videos, here a group of frames.
In FIG. 5, (1) the value of the average quantization parameter P in each frame group and the determination result by the quality determining unit 105 based on this bit rate, (3) the type of stream sent from the stream transmitting unit 104, is shown.
　この例では、品質判定部１０５による品質の判定に係る閾値を「１０」とし、平均量子化パラメータＰ≦１０であれば判定結果は「Q_G」であり、Ｐ＞１０であれば、判定結果は「Q_P」であるとする。 In this example, the threshold value for quality determination by the quality determination unit 105 is set to "10", and if the average quantization parameter P≦10, the determination result is "Q_G", and if P>10, the determination result is Suppose it is "Q_P".
　図５に示された例では、平均量子化パラメータＰが10以下のときは、品質の判定結果は「Q_G」であり、ストリーム送信部１０４から送出されるストリームの種別は、そのままの入力ストリームである。 In the example shown in FIG. 5, when the average quantization parameter P is 10 or less, the quality determination result is "Q_G", and the type of stream sent from the stream transmitter 104 is the input stream as it is. be.
　一方で、図５に示された例では、平均量子化パラメータＰが10を超えるときは、品質の判定結果は「Q_P」であり、ストリーム送信部１０４から送出されるストリームの種別は、画質向上処理後の入力ストリームである。 On the other hand, in the example shown in FIG. 5, when the average quantization parameter P exceeds 10, the quality determination result is "Q_P", and the type of stream sent from the stream transmitter 104 is "improved image quality". This is the input stream after processing.
　図６は、映像処理装置１００内の各部による処理結果の第３の例を表形式で示す図である。　
　この例では、図１などに示されるユーザ端末２０１の符号化部が、通信ネットワーク２０３の通信状況に基づき符号化ビットレートを変動させて、映像ストリームのパケットロストを極力避けるような制御をしているとする。 FIG. 6 is a diagram showing a third example of processing results by each unit in the video processing device 100 in a table format.
In this example, the encoding unit of the user terminal 201 shown in FIG. 1 etc. performs control to avoid packet loss of the video stream as much as possible by varying the encoding bit rate based on the communication status of the communication network 203. Suppose there is.
　この例での、ユーザＡのユーザ端末２０１と映像処理装置１００との間の通信ネットワーク、およびユーザＢのユーザ端末２０１と映像処理装置１００との間の通信ネットワークは、伝送帯域が低めではあるが変動が少なく、ユーザＣのユーザ端末２０１と映像処理装置１００の間の通信ネットワークは、伝送帯域の変動が大きいとする。　
　この例での品質判定部１０５による品質の判定対象は、各ユーザのユーザ端末から受信した映像符号化ストリームの過去１秒間のビットレートであるとする。 In this example, the communication network between user A's user terminal 201 and video processing device 100 and the communication network between user B's user terminal 201 and video processing device 100 have a lower transmission band. It is assumed that the communication network between user C's user terminal 201 and the video processing device 100 has a large fluctuation in transmission band.
In this example, it is assumed that the quality determination target by the quality determination unit 105 is the bit rate of the video encoded stream received from the user terminal of each user for the past one second.
　この例では、ユーザＡのユーザ端末２０１からの過去１秒間の映像ストリームのビットレート（BR）（Mbps）およびとユーザＣのユーザ端末２０１からの過去１秒間の映像ストリームのビットレート（BR）（Mbps）に対する品質判定部１０５による判定に係る閾値を1.0（Mbps）であるとする。 In this example, the bit rate (BR) (Mbps) of the video stream from user A's user terminal 201 in the past 1 second, and the bit rate (BR) (Mbps) of the video stream from user C's user terminal 201 in the past 1 second (Mbps). It is assumed that the threshold value related to the determination by the quality determining unit 105 for (Mbps) is 1.0 (Mbps).
　一方で、ユーザＢは自身の映像品質が多少低くても良いと考えているため、このユーザＢのユーザ端末２０１からの過去１秒間の映像ストリームのビットレート（BR）（Mbps）に対する品質判定部１０５による判定に係る閾値は0.8（Mbps）であるとする。 On the other hand, since user B thinks that it is okay even if his own video quality is somewhat low, the quality judgment unit uses the bit rate (BR) (Mbps) of the video stream from the user terminal 201 of user B for the past 1 second. 105 is assumed to be 0.8 (Mbps).
　図６では、（１）各時刻におけるユーザＡ、ＢおよびＣのユーザ端末からの映像ストリームの過去１秒間のビットレート、このビットレートに基づく、品質判定部１０５による判定結果、（２）上記判定結果に基づく、品質向上符号化部１０３による映像ストリームの画質向上の対象となるユーザ端末を用いるユーザ、が示される。 In FIG. 6, (1) the bit rate of the video stream from the user terminals of users A, B, and C for the past one second at each time, the determination result by the quality determination unit 105 based on this bit rate, and (2) the above determination. Based on the results, the users using the user terminals whose video streams are to be improved in quality by the quality improvement encoding unit 103 are shown.
　図６に示された例では、ユーザＡならびにＣに係る過去１秒間のビットレート（Mbps）が1.0以上のとき、またはユーザＢに係る過去１秒間のビットレート（Mbps）が0.8以上のときは、品質の判定結果は「Q_G」であり、ストリーム送信部１０４から送出されるストリームの種別は、そのままの入力ストリームである。 In the example shown in FIG. 6, when the bit rate (Mbps) for the past 1 second for users A and C is 1.0 or more, or when the bit rate (Mbps) for the past 1 second for user B is 0.8 or more, , the quality determination result is "Q_G", and the type of stream sent from the stream transmitter 104 is an input stream as is.
　一方で、図６に示された例では、ユーザＡならびにＣに係る過去１秒間のビットレート（Mbps）が1.0未満のとき、またはユーザＢに係る過去１秒間のビットレート（Mbps）が0.8未満のときは、品質の判定結果は「Q_P」である。ここでは、ストリーム送信部１０４から送出されるストリームの種別は、画質向上処理後の入力ストリーム、すなわちストリーム受信部１０１により受信されたストリームに対して品質向上符号化部１０３による品質向上の処理がなされたストリームである。 On the other hand, in the example shown in FIG. 6, when the bit rate (Mbps) for the past 1 second for users A and C is less than 1.0, or when the bit rate (Mbps) for the past 1 second for user B is less than 0.8. In this case, the quality determination result is "Q_P". Here, the type of stream sent out from the stream transmitting unit 104 is an input stream after image quality improvement processing, that is, a stream received by the stream receiving unit 101 is subjected to quality improvement processing by the quality improvement encoding unit 103. This is a stream.
　この図６に示された例では、時刻「T-2」、「T+1」において、ユーザＡのユーザ端末２０１から受信した映像ストリームに対する品質判定部１０５による判定結果が「Q_P」であるので、画質向上対象ユーザはユーザＡである。 In the example shown in FIG. 6, the determination result by the quality determination unit 105 for the video stream received from the user terminal 201 of user A at times "T-2" and "T+1" is "Q_P". , the image quality improvement target user is user A.
　また、図６に示された例では、時刻「T」において、ユーザＡのユーザ端末２０１から受信した映像ストリームに対する品質判定部１０５による判定結果、およびユーザＣのユーザ端末２０１から受信した映像ストリームに対する品質判定部１０５による判定結果が「Q_P」であるので、画質向上対象ユーザはユーザＡおよびユーザＣである。 Further, in the example shown in FIG. 6, at time "T", the determination result by the quality determination unit 105 for the video stream received from user A's user terminal 201, and the determination result for the video stream received from user C's user terminal 201. Since the determination result by the quality determination unit 105 is "Q_P", the users A and C are the target users for image quality improvement.
　図７は、映像処理装置１００内の各部による処理結果の第４の例を表形式で示す図である。　
　この例では、映像ストリームの出力先のユーザ端末２０２が多数あり、それぞれの端末の利用者は、ユーザＡ、ＢおよびＣのユーザ端末２０１からの映像ストリームを任意に選択して視聴しており、この配信数、すなわち同時視聴者数がユーザＡ、ＢおよびＣの間で大きく異なるとする。例えばユーザＡ、ＢおよびＣによるライブ配信が同時刻になされており、配信者の人気に応じた配信数が大きく異なることが挙げられる。 FIG. 7 is a diagram showing a fourth example of processing results by each unit in the video processing device 100 in a table format.
In this example, there are many user terminals 202 to which video streams are output, and users of each terminal arbitrarily select and view video streams from the user terminals 201 of users A, B, and C. Assume that the number of distributions, that is, the number of simultaneous viewers, differs greatly between users A, B, and C. For example, users A, B, and C are performing live broadcasts at the same time, and the number of broadcasts varies greatly depending on the popularity of the broadcasters.
　この例では、同時視聴者数が多い配信に係る画質を向上させて、より高品質の映像が見られるようなサービスを提供することができる。　
　この例での、品質判定部１０５による品質の判定対象は、ユーザ端末２０１から配信される映像符号化ストリームの本数であり、この本数が５０本以上ならば判定結果は「Q_P」であり、５０本未満であれば判定結果は「Q_G」となる。 In this example, it is possible to improve the image quality related to distribution with a large number of simultaneous viewers, and provide a service that allows higher quality video to be viewed.
In this example, the quality judgment target by the quality judgment unit 105 is the number of encoded video streams distributed from the user terminal 201, and if this number is 50 or more, the judgment result is "Q_P", which is 50. If it is less than this, the determination result will be "Q_G".
　図７では、（１）各時刻におけるユーザＡ、ＢおよびＣの各々のユーザ端末２０１からの映像ストリームの配信数と、配信数に基づく、品質判定部１０５による判定結果、（２）上記判定結果に基づく、品質向上符号化部１０３による映像ストリームの画質向上の対象となるユーザ端末を用いるユーザ、が示される。 In FIG. 7, (1) the number of distributions of video streams from the user terminals 201 of users A, B, and C at each time and the determination result by the quality determination unit 105 based on the number of distributions, and (2) the above determination result. A user using a user terminal whose image quality of a video stream is to be improved by the quality improvement encoding unit 103 based on the above is shown.
　図７に示された例では、時刻「T-1」、「T」および「T+1」において、ユーザＡのユーザ端末２０１から配信される受信した映像ストリームに対する品質判定部１０５による判定結果が「Q_P」であるので、画質向上対象ユーザは「ユーザＡ」である。 In the example shown in FIG. 7, at times "T-1", "T", and "T+1", the determination results by the quality determination unit 105 for the received video stream distributed from the user terminal 201 of user A are Since it is "Q_P", the user targeted for image quality improvement is "user A".
　図８は、本発明の一実施形態に係る映像処理装置のハードウエア構成の一例を示すブロック図である。　
　図８に示された例では、上記の実施形態に係る映像処理装置１００は、例えばサーバコンピュータ（server computer）またはパーソナルコンピュータにより構成され、ＣＰＵ等のハードウエアプロセッサ（hardware processor）１１１Ａを有する。そして、このハードウエアプロセッサ１１１Ａに対し、プログラムメモリ（program memory）１１１Ｂ、データメモリ（data memory）１１２、入出力インタフェース１１３及び通信インタフェース１１４が、バス（bus）１１５を介して接続される。 FIG. 8 is a block diagram showing an example of the hardware configuration of a video processing device according to an embodiment of the present invention.
In the example shown in FIG. 8, the video processing device 100 according to the above embodiment is configured by, for example, a server computer or a personal computer, and includes a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input/output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.
　通信インタフェース１１４は、例えば１つ以上の無線の通信インタフェースユニットを含んでおり、通信ネットワークとの間で情報の送受信を可能にする。無線インタフェースとしては、例えば無線ＬＡＮ（Local Area Network）などの小電力無線データ通信規格が採用されたインタフェースが使用される。 The communication interface 114 includes, for example, one or more wireless communication interface units, and enables information to be sent and received to and from a communication network. As the wireless interface, for example, an interface adopting a low power wireless data communication standard such as a wireless LAN (Local Area Network) is used.
　入出力インタフェース１１３には、映像処理装置１００に付設される、ユーザなどにより用いられる入力デバイス（device）２００および出力デバイス３００が接続される。　
　入出力インタフェース１１３は、キーボード（keyboard）、タッチパネル（touch panel）、タッチパッド（touchpad）、マウス（mouse）等の入力デバイス２００を通じてユーザなどにより入力された操作データを取り込むとともに、出力データを液晶または有機ＥＬ（Electro Luminescence）等が用いられた表示デバイスを含む出力デバイス３００へ出力して表示させる処理を行なう。なお、入力デバイス２００および出力デバイス３００には、映像処理装置１００に内蔵されたデバイスが使用されてもよく、また、通信ネットワークを介して映像処理装置１００と通信可能である他の情報端末の入力デバイスおよび出力デバイスが使用されても良い。 The input/output interface 113 is connected to an input device 200 and an output device 300 attached to the video processing apparatus 100 and used by a user or the like.
The input/output interface 113 receives operation data input by a user through an input device 200 such as a keyboard, touch panel, touchpad, mouse, etc., and outputs output data on a liquid crystal display or a mouse. A process of outputting and displaying the image to an output device 300 including a display device using organic EL (Electro Luminescence) or the like is performed. Note that the input device 200 and the output device 300 may be a device built into the video processing device 100, or may be an input device of another information terminal that can communicate with the video processing device 100 via a communication network. Devices and output devices may also be used.
　プログラムメモリ１１１Ｂは、非一時的な有形の記憶媒体として、例えば、ＨＤＤ（Hard Disk Drive）またはＳＳＤ（Solid State Drive）等の随時書込みおよび読出しが可能な不揮発性メモリ（non-volatile memory）と、ＲＯＭ（Read Only Memory）等の不揮発性メモリとが組み合わせて使用されたもので、一実施形態に係る各種制御処理等を実行する為に必要なプログラムが格納されている。 The program memory 111B is a non-temporary tangible storage medium, such as a non-volatile memory that can be written to and read from at any time, such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). It is used in combination with a nonvolatile memory such as a ROM (Read Only Memory), and stores programs necessary for executing various control processes and the like according to an embodiment.
　データメモリ１１２は、有形の記憶媒体として、例えば、上記の不揮発性メモリと、ＲＡＭ等の揮発性メモリ（volatile memory）とが組み合わせて使用されたもので、各種処理が行なわれる過程で取得および作成された各種データが記憶される為に用いられる。 The data memory 112 is a tangible storage medium that uses a combination of the above-mentioned non-volatile memory and volatile memory such as RAM, and is acquired and created during various processes. It is used to store various types of data.
　本発明の一実施形態に係る映像処理装置１００は、ソフトウエア（software）による処理機能部として、図１などに示される各部を有するデータ処理装置として構成され得る。 The video processing device 100 according to an embodiment of the present invention can be configured as a data processing device having each section shown in FIG. 1 and the like as a processing function section using software.
　映像処理装置１００の各部によるワークメモリ（working memory）などとして用いられる記憶装置は、図８に示されたデータメモリ１１２が用いられることで構成され得る。ただし、これらの構成される記憶領域は映像処理装置１００内に必須の構成ではなく、例えば、ＵＳＢ（Universal Serial Bus）メモリなどの外付け記憶媒体、又はクラウド（cloud）に配置されたデータベースサーバ（database server）等の記憶装置に設けられた領域であっても良い。 A storage device used as a working memory by each part of the video processing device 100 may be configured by using the data memory 112 shown in FIG. 8. However, these configured storage areas are not essential configurations within the video processing device 100, and are, for example, external storage media such as a USB (Universal Serial Bus) memory, or a database server located in the cloud (cloud). It may also be an area provided in a storage device such as a database server.
　上記の各部における処理機能部は、いずれも、プログラムメモリ１１１Ｂに格納されたプログラムを上記ハードウエアプロセッサ１１１Ａにより読み出させて実行させることにより実現され得る。なお、これらの処理機能部の一部または全部は、特定用途向け集積回路（ＡＳＩＣ（Application Specific Integrated Circuit））またはＦＰＧＡ（Field-Programmable Gate Array）などの集積回路を含む、他の多様な形式によって実現されても良い。 The processing function units in each of the above units can be realized by causing the hardware processor 111A to read and execute a program stored in the program memory 111B. Note that some or all of these processing functions may be implemented in a variety of other formats, including integrated circuits such as application specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). It may be realized.
　また、各実施形態に記載された手法は、計算機（コンピュータ）に実行させることができるプログラム（ソフトウエア手段）として、例えば磁気ディスク（フロッピー（登録商標）ディスク（Floppy disk）、ハードディスク（hard disk）等）、光ディスク（optical disc）（ＣＤ－ＲＯＭ、ＤＶＤ、ＭＯ等）、半導体メモリ（ＲＯＭ、ＲＡＭ、フラッシュメモリ（Flash memory）等）等の記録媒体に格納し、また通信媒体により伝送して頒布され得る。なお、媒体側に格納されるプログラムには、計算機に実行させるソフトウエア手段（実行プログラムのみならずテーブル（table）、データ構造も含む）を計算機内に構成させる設定プログラムをも含む。本装置を実現する計算機は、記録媒体に記録されたプログラムを読み込み、また場合により設定プログラムによりソフトウエア手段を構築し、このソフトウエア手段によって動作が制御されることにより上述した処理を実行する。なお、本明細書でいう記録媒体は、頒布用に限らず、計算機内部あるいはネットワークを介して接続される機器に設けられた磁気ディスク、半導体メモリ等の記憶媒体を含むものである。 In addition, the method described in each embodiment can be applied to a magnetic disk (floppy (registered trademark) disk, hard disk) as a program (software means) that can be executed by a computer (computer). etc.), optical discs (CD-ROM, DVD, MO, etc.), semiconductor memories (ROM, RAM, Flash memory, etc.), and are stored in recording media, or transmitted and distributed via communication media. can be done. Note that the programs stored on the medium side also include a setting program for configuring software means (including not only execution programs but also tables and data structures) in the computer to be executed by the computer. A computer that realizes this device reads a program recorded on a recording medium, and if necessary, constructs software means using a setting program, and executes the above-described processing by controlling the operation of the software means. Note that the recording medium referred to in this specification is not limited to one for distribution, and includes storage media such as a magnetic disk and a semiconductor memory provided inside a computer or in a device connected via a network.
　なお、本発明は、上記実施形態に限定されるものではなく、実施段階ではその要旨を逸脱しない範囲で種々に変形することが可能である。また、各実施形態は適宜組み合わせて実施してもよく、その場合組み合わせた効果が得られる。更に、上記実施形態には種々の発明が含まれており、開示される複数の構成要件から選択された組み合わせにより種々の発明が抽出され得る。例えば、実施形態に示される全構成要件からいくつかの構成要件が削除されても、課題が解決でき、効果が得られる場合には、この構成要件が削除された構成が発明として抽出され得る。 Note that the present invention is not limited to the above-described embodiments, and can be variously modified at the implementation stage without departing from the gist thereof. Moreover, each embodiment may be implemented in combination as appropriate, and in that case, the combined effect can be obtained. Furthermore, the embodiments described above include various inventions, and various inventions can be extracted by combinations selected from the plurality of constituent features disclosed. For example, if a problem can be solved and an effect can be obtained even if some constituent features are deleted from all the constituent features shown in the embodiment, the configuration from which these constituent features are deleted can be extracted as an invention.
　　１００…映像処理装置
　　１０１…ストリーム受信部
　　１０２…映像復号部
　　１０３…品質向上符号化部
　　１０４…ストリーム送信部
　　１０５…品質判定部 100...Video processing device 101...Stream receiving section 102...Video decoding section 103...Quality improvement encoding section 104...Stream transmitting section 105...Quality determining section

Claims (8)

  1. 　映像ストリームを受信する受信部と、
    　前記受信部により受信した映像ストリームの品質の向上が望ましいか否かを判定する判定部と、
    　前記映像ストリームの品質の向上が望ましいと前記判定部により判定されたときに、前記受信部により受信した映像ストリームの品質を向上させる品質向上処理部と、
    　を備える映像処理装置。     a receiving unit that receives a video stream;
    a determination unit that determines whether it is desirable to improve the quality of the video stream received by the reception unit;
    a quality improvement processing unit that improves the quality of the video stream received by the reception unit when the determination unit determines that it is desirable to improve the quality of the video stream;
    An image processing device comprising:
  2. 　前記受信部は、符号化された映像ストリームを受信し、
    　前記受信部により受信された、前記符号化された映像ストリームを復号する復号部をさらに備え、
    　前記判定部は、前記復号部により復号された映像ストリームの品質の向上が望ましいか否かを判定する、
    　請求項１に記載の映像処理装置。     The receiving unit receives an encoded video stream,
    further comprising a decoding unit that decodes the encoded video stream received by the receiving unit,
    The determination unit determines whether it is desirable to improve the quality of the video stream decoded by the decoding unit.
    The video processing device according to claim 1.
  3. 　前記映像ストリームの品質は、
    　　前記受信部による前記映像ストリームの受信時のパケットロストの発生率、前記受信部により受信した映像ストリームのビットレート、および前記映像ストリームが符号化されるときの量子化パラメータの少なくとも１つを含む、
    　請求項１に記載の映像処理装置。     The quality of the video stream is
    including at least one of a packet loss occurrence rate when receiving the video stream by the receiving unit, a bit rate of the video stream received by the receiving unit, and a quantization parameter when the video stream is encoded;
    The video processing device according to claim 1.
  4. 　前記受信部は、複数の端末から前記映像ストリームを受信し、
    　前記判定部は、複数の前記映像ストリームの各々の品質の向上が望ましいか否かを判定し、
    　前記品質向上処理部は、
    　　前記判定部により品質の向上が望ましいと判定された複数の映像ストリームがあるときに、前記判定された複数の映像ストリームのうち処理可能な分量の映像ストリームの品質を向上させる、
    　請求項１に記載の映像処理装置。     The receiving unit receives the video stream from a plurality of terminals,
    The determination unit determines whether it is desirable to improve the quality of each of the plurality of video streams,
    The quality improvement processing unit is
    When there are a plurality of video streams whose quality is determined to be desirable to improve by the determination unit, improving the quality of a processable amount of video streams among the determined plurality of video streams;
    The video processing device according to claim 1.
  5. 　前記受信部は、複数の外部端末の各々から送信された前記映像ストリームを受信し、
    　前記判定部は、複数の外部端末の各々から送信された前記映像ストリームの同時視聴者数に基づいて、前記映像ストリームの品質の向上が望ましいか否かを判定する、
    　請求項１に記載の映像処理装置。     The receiving unit receives the video stream transmitted from each of a plurality of external terminals,
    The determination unit determines whether it is desirable to improve the quality of the video stream based on the number of simultaneous viewers of the video stream transmitted from each of a plurality of external terminals.
    The video processing device according to claim 1.
  6. 　映像処理装置により行なわれる方法であって、
    　前記映像処理装置の受信部により、映像ストリームを受信し、
    　前記映像処理装置の判定部により、前記受信部により受信した映像ストリームの品質の向上が望ましいか否かを判定し、
    　前記映像ストリームの品質の向上が望ましいと前記判定部により判定されたときに、前記映像処理装置の品質向上処理部により、前記受信部により受信した映像ストリームの品質を向上させる、
    　映像処理方法。     A method performed by an image processing device, the method comprising:
    receiving a video stream by a receiving unit of the video processing device;
    a determination unit of the video processing device determines whether it is desirable to improve the quality of the video stream received by the reception unit;
    When the determining unit determines that it is desirable to improve the quality of the video stream, a quality improvement processing unit of the video processing device improves the quality of the video stream received by the receiving unit;
    Video processing method.
  7. 　前記受信部は、符号化された映像ストリームを受信し、
    　前記映像処理装置の復号部により、前記受信部により受信された、前記符号化された映像ストリームを復号し、
    　前記判定部は、前記復号部により復号された映像ストリームの品質の向上が望ましいか否かを判定する、
    　請求項６に記載の映像処理方法。     The receiving unit receives an encoded video stream,
    decoding the encoded video stream received by the receiving unit by a decoding unit of the video processing device;
    The determination unit determines whether it is desirable to improve the quality of the video stream decoded by the decoding unit.
    The video processing method according to claim 6.
  8. 　請求項１乃至５のいずれか１項に記載の映像処理装置の前記各部としてプロセッサを機能させる映像処理プログラム。 A video processing program that causes a processor to function as each of the units of the video processing device according to claim 1.
PCT/JP2022/028052 2022-07-19 2022-07-19 Video processing device, method, and program WO2024018525A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/028052 WO2024018525A1 (en) 2022-07-19 2022-07-19 Video processing device, method, and program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/028052 WO2024018525A1 (en) 2022-07-19 2022-07-19 Video processing device, method, and program

Publications (1)

Publication Number Publication Date
WO2024018525A1 true WO2024018525A1 (en) 2024-01-25

Family

ID=89617390

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/028052 WO2024018525A1 (en) 2022-07-19 2022-07-19 Video processing device, method, and program

Country Status (1)

Country Link
WO (1) WO2024018525A1 (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004153620A (en) * 2002-10-31 2004-05-27 Kyocera Corp Communication system, radio communication terminal, data distribution device, and communication method
JP2009081753A (en) * 2007-09-27 2009-04-16 Hitachi Ltd Radio communication system, transmitter, and receiver
JP2010093650A (en) * 2008-10-09 2010-04-22 Nec Corp Terminal, image display method, and program
WO2016031912A1 (en) * 2014-08-29 2016-03-03 シャープ株式会社 Control information generating device, transmission device, reception device, television receiver, video signal transmission system, control program, and recording medium
CN111586412A (en) * 2020-05-06 2020-08-25 华为技术有限公司 High-definition video processing method, master device, slave device and chip system
CN113115067A (en) * 2021-04-19 2021-07-13 脸萌有限公司 Live broadcast system, video processing method and related device
JP2022122466A (en) * 2021-02-10 2022-08-23 ソフトバンク株式会社 Communication system, communication device, and program

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2004153620A (en) * 2002-10-31 2004-05-27 Kyocera Corp Communication system, radio communication terminal, data distribution device, and communication method
JP2009081753A (en) * 2007-09-27 2009-04-16 Hitachi Ltd Radio communication system, transmitter, and receiver
JP2010093650A (en) * 2008-10-09 2010-04-22 Nec Corp Terminal, image display method, and program
WO2016031912A1 (en) * 2014-08-29 2016-03-03 シャープ株式会社 Control information generating device, transmission device, reception device, television receiver, video signal transmission system, control program, and recording medium
CN111586412A (en) * 2020-05-06 2020-08-25 华为技术有限公司 High-definition video processing method, master device, slave device and chip system
JP2022122466A (en) * 2021-02-10 2022-08-23 ソフトバンク株式会社 Communication system, communication device, and program
CN113115067A (en) * 2021-04-19 2021-07-13 脸萌有限公司 Live broadcast system, video processing method and related device

Similar Documents

Publication Publication Date Title
US10194188B1 (en) Bitrate selection for video streaming
US8902967B2 (en) Systems and methods for distributed media stream transcoding and sharing
US20110150100A1 (en) Progressive shape based encoding of video content within a swarm environment
US8842159B2 (en) Encoding processing for conferencing systems
US8964851B2 (en) Dual-mode compression of images and videos for reliable real-time transmission
US20140241419A1 (en) Multi-stream optimization
WO2020220902A1 (en) Method and apparatus for distributing transmission parameters of video resources
US20170103577A1 (en) Method and apparatus for optimizing video streaming for virtual reality
US20140189064A1 (en) Method and system for adaptive video transmission
KR20120013179A (en) Method and system for progressive rate adaptation for uncompressed video communication in wireless systems
WO2015196590A1 (en) Method and apparatus for playing desktop cloud video
CN110113306B (en) Method and network device for distributing data
US20230096562A1 (en) Method and system for transmitting and reproducing video of dynamic bitrate with a plurality of channels
KR20140056296A (en) Techniques for dynamic switching between coded bitstreams
WO2019114690A1 (en) Precoding method and apparatus in multi-antenna system
US20220239920A1 (en) Video processing method, related apparatus, storage medium, and program product
Chen et al. Enhancing quality of experience for collaborative virtual reality with commodity mobile devices
US9838463B2 (en) System and method for encoding control commands
WO2024018525A1 (en) Video processing device, method, and program
WO2022237427A1 (en) Video processing method and apparatus, device, and storage medium
US11277460B1 (en) Video communications network with value optimization
CN112153322A (en) Data distribution method, device, equipment and storage medium
Lv et al. Unequal error protection for 360 VR video based on expanding window fountain codes
Li et al. Dante: Enabling fov-aware adaptive fec coding for 360-degree video streaming
US11765220B2 (en) Method, system, and computer program product for streaming

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22951914

Country of ref document: EP

Kind code of ref document: A1