JP6224516B2 - Encoding method and encoding program - Google Patents

Description

  The present invention relates to an encoding method and an encoding program for encoding video.

  In recent years, video having more pixels than video that can be displayed on HD (High Definition) television, which is currently mainstream, such as 4K video, 8K video, and panoramic video, has appeared. If you try to display these images on a device such as a general HD TV or a smartphone or tablet with a panel with a smaller pixel count, the original high-definition image will be reduced. In general, video display methods have been used, such as displaying, in which the fineness and content of the video of the original video source is lost.

  On the other hand, even with a terminal having a smaller number of pixels than the number of pixels that the video source has, as a viewing method that does not lose the definition and content of the original video, video partial distribution technology such as interactive panorama has appeared (for example, Non-patent document 1).

  In the method of Non-Patent Document 1, it is possible to decode and present a tile region having the most appropriate resolution according to the position and angle of view selected by a viewer using a device such as a tablet, a smartphone, or a controller. FIG. 7 is an explanatory diagram showing a method of encoding a plurality of tile areas in a rectangular image of Non-Patent Document 1. As shown in FIG. 7, in Non-Patent Document 1, an image is divided into a plurality of resolutions (resolution [1], resolution [2],..., Resolution [n]) in advance, and these multiple resolution images are further divided in advance. It is divided into tile areas of a predetermined size and encoded at a bit rate determined in advance for each resolution. Then, the tile area having the most appropriate resolution is decoded and presented according to the position and angle of view selected by the device. The viewer can change the viewing position and size by interactively selecting the position and angle of view. For this reason, not only the fineness of the video is lost, but also the content of the video itself is not lost.

Hideaki Kimata, Daisuke Ochi, Akio Kameda, Hajime Noto, Katsuhiko Fukazawa, Akira Kojima, "Mobile and Multi-device Interactive Panorama Video Distribution System," IEEE GCCE 2012, 2012.

  However, when encoding a general video by dividing it into tiled areas including such a technique as interactive panorama, it is general to encode all tiled areas at a constant bit rate. Or video that encodes according to the amount of edge of the image in the tiled area, or encodes the amount of edge of the image of the tiled area near or near the target tiled area It is common to determine the bit rate ignoring the content. In the conventional encoding, the bit rate is generally determined in consideration of only video features such as edges in an image.

  Since people have different preferences, it goes without saying that even if they watch the same video, the positions they see vary. It can be said that there are positions that are often seen by people (popular positions) and positions that are not so depending on the video. If a tiled area in a video is encoded by a uniform method as in the prior art, there is a problem that a high bit rate is assigned to an unpopular position. On the other hand, there is a problem that inefficient bit rate allocation may be performed such that a low bit rate is allocated to a popular position.

  The present invention has been made in view of such circumstances, and provides an encoding method and an encoding program capable of performing efficient bit rate allocation according to video content when encoding a tiled area of a video. With the goal.

  The present invention provides an encoding method for encoding a video, a tile dividing step for dividing the video into tiled region videos, a first encoding step for encoding the tiled region videos, and the encoded tiled steps. A distribution frequency storing step for storing a distribution frequency, which is a frequency at which the region video is distributed in response to a user's viewing request, and a bit for calculating a bit rate for the tiled region video by multiplying the distribution frequency by a predetermined bit rate The method includes a rate calculating step and a second encoding step for re-encoding the tiled area video according to the calculated bit rate.

  The present invention relates to an encoding method for encoding a video, an attribute value calculating step for calculating a value of a predetermined attribute relating to the video, a tile dividing step for dividing the video into tiled region images, and the tiled region A bit rate calculating step of calculating a bit rate for the tiled region image by multiplying a ratio of the attribute value included in the image by a predetermined bit rate; and an encoding for encoding the tiled region image according to the calculated bit rate And a step.

  The present invention relates to an encoding method for encoding a video, an attribute value calculating step for calculating a value of a predetermined attribute relating to the video, a tile dividing step for dividing the video into tiled region images, and the tiled region A first encoding step for encoding a video; and a distribution frequency storage step for storing a distribution frequency, which is a frequency of distributing the encoded tiled region video in response to a user's viewing request, for each value of the predetermined attribute; A distribution ratio calculating step of calculating a distribution ratio of the tiled area video corresponding to the predetermined attribute value, and generating the tiled area video from a newly input video. The bit rate assigned to the region video is multiplied by the distribution rate corresponding to the attribute value of the tiled region video by a predetermined bit rate. Characterized in that it has a bit rate calculation step of leaving, a second encoding step of encoding a tiled area image newly input the image in accordance with the bit rate calculated.

  The present invention is characterized in that the bit rate calculating step has a plurality of predetermined bit rates, and the second encoding step encodes the tiled region video for each of the plurality of bit rates.

  The present invention is an encoding program for causing a computer to execute the encoding method.

  According to the present invention, it is possible to perform an efficient bit rate allocation according to video content when encoding a tiled area of a video.

1 is a block diagram showing an overall configuration of a system according to an embodiment of the present invention. It is a figure which shows the example which implement | achieved the system shown in FIG. 1 with the actual apparatus. It is explanatory drawing which shows the operation | movement of the encoding process in the encoding apparatus shown in FIG. 1 by 1st Embodiment. It is a figure which shows the processing operation of the encoding apparatus shown in FIG. 1 by 2nd Embodiment. It is a figure which shows the processing operation of the encoding apparatus shown in FIG. 1 by 3rd Embodiment. It is a block diagram which shows the structure of the system which implement | achieved real-time video delivery using WebRTC combining the 1st-3rd embodiment. It is explanatory drawing which shows the method of encoding a some tile area | region in the rectangular image of a nonpatent literature 1. FIG.

<First Embodiment>
Hereinafter, an encoding apparatus according to a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the overall configuration of the system of the embodiment. In this figure, reference numeral 1 denotes an image composed of any device capable of capturing an image, such as a general video camera including a 4K / 8K camera, a panoramic image photographing all-around camera, and a plurality of video cameras. It is a shooting part. Reference numeral 2 denotes a video storage unit that holds a video shot by the video imaging unit 1. Reference numeral 3 denotes a video compression unit that encodes video at the determined bit rate. The video compression unit 3 also performs multi-resolution video source and tile area division.

  Reference numeral 4 denotes a compressed video storage unit that stores the video encoded by the video compression unit 3. Reference numeral 5 denotes a video distribution unit that distributes the requested tile area. The number of tiled areas distributed is recorded as a log. Reference numeral 6 denotes a bit rate storage unit that stores video features / contents of tiled regions encoded in the past and bit rates assigned thereto.

  Reference numeral 7 denotes the distribution frequency (viewing frequency) of the tiled area obtained from the distribution log in the video distribution unit 5, the video feature / video content of the tiled area in the video compression unit 3, and the past stored in the bit rate storage unit 6. The bit rate determination unit determines the bit rate of each tiled region of the video input to the video compression unit 3 based on the video feature of the tiled region, the bit rate value corresponding to the video content, and the like. The operation for calculating the bit rate assigned to each tile area will be described later. Reference numeral 8 denotes a network capable of information communication.

  Reference numeral 9 denotes a video reception unit that receives a tile area distributed from the video distribution unit 5. Reference numeral 10 denotes a video buffer unit that stores the video received by the video receiving unit 9 until a predetermined unit (time) is obtained. The video buffer unit 10 is used to absorb fluctuations in time required for arrival of video data due to path fluctuations between the video distribution unit 5 and the video reception unit 9 in handling video. Reference numeral 11 denotes a video development unit that decodes the encoded video data delivered from the video reception unit 9.

  Reference numeral 12 denotes a video rendering unit that renders video data decoded by the video development unit 11 in a form that can be viewed by the user. Here, it is configured by a display device such as a general monitor, a three-dimensional monitor, a projector, an HMD (head mounted display), a smartphone, or a tablet. Reference numeral 13 denotes a user input unit that inputs a position and an angle of view desired by the user. The user input unit 13 includes not only input devices such as a controller, a mouse, a keyboard, a touch interface of a smartphone or a tablet, but also input from sensors such as an orientation sensor and an acceleration sensor. Reference numeral 14 denotes a video requesting unit that requests a necessary tile area in response to an input from the user input unit 13.

  Next, an example in which the system shown in FIG. 1 is realized by an actual apparatus will be described with reference to FIG. FIG. 2 is a diagram showing an example in which the system shown in FIG. 1 is realized by an actual device. First, the video photographing unit 1 is composed of an all-sky camera. The video storage unit 2 is constituted by a storage. The video compression unit 3, the bit rate determination unit 6, and the bit rate storage unit 7 are composed of an encode PC (personal computer) 20. The compressed video storage unit 4 and the video distribution unit 5 are configured by a distribution server 30. The network 8 uses the Internet.

  The video request unit 14, the video reception unit 9, the video buffer unit 10, and the video development unit 11 are configured by a client PC 40 to which an HMD (head mounted display) is connected. The video requesting unit 14, the video receiving unit 9, and the video developing unit 11 are realized by an application 50 on the client PC 40. The video buffer unit 9 is realized by a memory mounted on the client PC 40.

  The video drawing unit 12 and the user input unit 13 are composed of a head mounted display 60. The video drawing unit 12 is realized by a display equipped with a head mounted display 60. The user input unit 13 is realized by a gyro equipped with a head mounted display 60.

  In addition, as a form of the system, each role may operate on different physical systems, or a plurality of roles may operate on the same system. In addition, regardless of whether the roles are wired or wireless, there is no limitation on the form such as being connected by a network or being connected by a bus in the same system.

  Next, the operation of the encoding process in the encoding apparatus shown in FIG. 1 will be described with reference to FIG. FIG. 3 is an explanatory diagram showing the operation of the encoding process in the encoding apparatus shown in FIG. The initial bit rate value, the overall bit rate value, the viewing frequency, the minimum bit rate value, the bit rate update interval, and the like illustrated in FIG. 3 are merely examples, and are not limited thereto.

  FIG. 3A shows an example of the initial value of the bit rate assigned by the bit rate determining unit 7. The numerical value is the initial bit rate, and the unit is kbps. It is assumed that 2000 kbps is determined in advance. Divided into tiled areas, all encoded at the same bit rate. FIG. 3A illustrates one resolution.

  FIG. 3B shows the number of distributions for each tiled region after a predetermined time has elapsed, for example, one hour later, that is, one hour. The numerical value indicates the number of tiles distributed in one hour from the initial value and the percentage. The calculation method of the percentage is, for example, calculating how much the target tile is distributed in one hour in a frame including the target tile-shaped region, or all the tile-shaped regions from the past in the same video. As long as the viewing frequency can be expressed as a ratio, such as calculating the number of distributions from the past of the target tile-shaped region, the calculation method is not limited.

When 2000 kbps is distributed according to the viewing frequency (ratio), it is as shown in FIG. The numerical value is a bit rate and the unit is kbps. However, the minimum bit rate was 10 kbps. An equation for calculating the bit rate is as shown in equation (1).
Allocated bit rate = (2000−120) × viewing frequency (ratio) (1)
In the formula (1), 120 is the minimum bit rate.

  Finally, the video compression unit 3 re-encodes at the bit rate determined by the bit rate determination unit 7. Thereafter, the processing shown in FIGS. 3B and 3C is executed at predetermined time intervals, and re-encoding is continued.

  Although the ratio is used as the viewing frequency, the bit rate may be increased / decreased according to the absolute number of distributions. Therefore, it is not necessary to determine the overall bit rate value in advance. Also, if you prepare multiple bit rate amounts from large to small and save all the bit rate sets for each tiled area calculated by the total bit rate amount, the user's line quality will be improved. When it is understood, it is possible to deliver a set with a bit rate as low as possible for a slow line and a fast set for a fast line.

  According to the first embodiment, the following effects can be obtained. According to the user's viewing frequency, encoding of tiled areas of video that has been performed uniformly until now is assigned a higher bit rate as the position is often viewed, and if it is a position that has not been viewed Since the bit rate can be suppressed as much as possible, the bit rate of the entire video can be saved (reduced) while maintaining the user's quality of experience.

  Note that saving the bit rate has the effect of saving the network bandwidth at the time of video distribution, saving the storage area at the time of saving the video, saving the computer resources of the encoder at the time of video encoding, and saving the encoding time.

Second Embodiment
Next, an encoding apparatus according to a second embodiment of the present invention will be described. FIG. 4 is a diagram showing a processing operation of the encoding apparatus shown in FIG. 1 according to the second embodiment. The maximum bit rate value and the minimum bit rate value assigned to each tile area illustrated in FIG. 4 are merely examples, and the present invention is not limited thereto.

  In FIG. 4A, one frame of one resolution of an arbitrary video is illustrated as being divided into tiled areas. It is assumed that the face of a person moving in the video can be recognized using openCV or the like using a known technique and is recognized in advance.

  FIG. 4B shows the ratio (attribute value) of the area occupied by the face image in each tile area. FIG. 4C shows the bit rate assigned to each tiled area according to the face area ratio calculated by “200 kbps × ratio”. Note that 200 kbps is an example. The numerical values in the table are bit rate examples, and the unit is kbps. However, the minimum bit rate was 10 kbps.

  Finally, the video compression unit 3 encodes according to the bit rate determined by the bit rate determination unit 7 as shown in FIG.

  Although the ratio of the area of the face image was given as an example, the ratio of the area occupied by a person area or a specific object, the ratio of brightness (when all the pixels of the tile area are the brightest and each of the tile areas) Ratio of pixel brightness sum), ratio of edge amount (ratio of sum of edge amount of each pixel in tiled area when all pixels in tiled area are edges), ratio of motion vector amount (tile) The ratio of the sum of the motion vector amounts of each pixel in the tiled area) and the ratio of salientity (all the pixels in the tiled area are all salient) And the ratio of the sum of the saliency of each pixel in the tiled area) and the ratio of the color intensity (when all the pixels in the tiled area are of a predetermined color and each pixel in the tiled area) Of may even ratio), and the like of the sum of the color intensity.

  The viewing frequency processing described in the first embodiment may be used together. That is, a higher bit rate may be allocated as it is popular and includes more face areas (examples). Also, if you prepare multiple bit rate amounts from large to small and save all the bit rate sets for each tiled area calculated by the total bit rate amount, the user's line quality will be improved. When it is understood, it is possible to deliver a set with a bit rate as low as possible for a slow line and a fast set for a fast line.

  According to the second embodiment, bit rate allocation is performed according to a position that is often seen in an image, that is, a position where a subject is reflected by person tracking or the like and a ratio including many salient areas obtained by image analysis. By changing, it is possible to save (reduce) the bit rate of the entire video while maintaining the user's quality of experience.

  Note that saving the bit rate has the effect of saving the network bandwidth at the time of video distribution, saving the storage area at the time of saving the video, saving the computer resources of the encoder at the time of video encoding, and saving the encoding time.

<Third Embodiment>
Next, an encoding apparatus according to a third embodiment of the present invention will be described. FIG. 5 is a diagram showing a processing operation of the encoding apparatus shown in FIG. 1 according to the third embodiment. FIG. 5A illustrates an average of luminance values in each tiled region of an arbitrary video by dividing one frame of one resolution into tiled regions. The average brightness (here, 256 levels) can be calculated using a known technique such as openCV, and is calculated in advance.

  FIG. 5B shows the measurement of the viewing frequency of each tiled area of the video every hour (the number of times of viewing for any one hour). The numerical value in the lower row indicates the viewing ratio within one frame as a target. This measurement is performed for all frames and all resolutions.

  A database of the number of times of viewing for each luminance as shown in FIG. This database corresponds to a detector that knows how many tiled areas having a certain luminance have been seen in the past. One database in FIG. 5C is configured for a plurality of distribution videos.

The numerical value in the upper part of FIG. 5D is a value obtained by calculating the luminance value (attribute value) of the tiled region in an arbitrary one resolution and one frame of a new video. The numerical values in the lower stage are bit rates predicted from past luminance values when the total is 2000 kbps, the unit is kbps, and the lowest bit rate is 10 kbps. The bit rate determining unit 7 calculates the bit rate to be assigned to each tile using the equation (2).
Allocated bit rate = 2000 kbps × viewing ratio at each luminance on the database (2)

  Although only one frame of a certain video is shown this time, the database is actually created based on the luminance value and the viewing frequency calculated over all tile regions of all the frames of the video. Since the feature value of the image such as the brightness value shown here is often determined in its size and maximum / minimum values (for example, 256 levels), if the feature value is divided by a predetermined width, Since the tiled area having an unknown luminance value of a new video always falls within some width, it is possible to determine the bit rate to be assigned.

  The initial bit rate of a new video is determined with reference to a past database. After the initial value is determined, the number of times of viewing is registered in the database based on the viewing frequency of each tiled area. . While sequentially updating the database, the bit rate assignment may be changed based on the latest database information not only for new videos but also for existing videos at an arbitrary timing.

  Here, the average brightness in each tiled area is used as the image feature, but the index of the salientity value, the edge amount, the motion vector amount, etc., which are also the image feature amount, is not limited. Further, the face area as in the second embodiment may be used.

  Also, if you prepare multiple bit rate amounts from large to small and save all the bit rate sets for each tiled area calculated by the total bit rate amount, the user's line quality will be improved. When it is understood, it is possible to deliver a set with a bit rate as low as possible for a slow line and a fast set for a fast line. The total bit rate amount, database update interval, and minimum bit rate used here are not limited to this.

  According to the third embodiment, by accumulating the video content and the viewing frequency of each tiled region, and by rotating the learning cycle such as how much the video feature is viewed and building the database, Efficient bit rate assignment is possible even when the characteristics of a new video or frequently seen part whose viewing frequency is unknown are changed. Also, by holding tiled areas with multiple sets of bitrates with different overall bitrates, the tiled areas that are commonly seen on any quality line are high bitrates, which are rarely seen A region can present a low bit rate, and can efficiently allocate a bit rate within a limited bandwidth.

<Fourth embodiment>
Next, an encoding apparatus according to a fourth embodiment of the present invention will be described. In the first to third embodiments, the description has been made in the case where VOD (video on demand) is applied. However, the same can be implemented in real-time communication of multiple persons such as a video conference.

  FIG. 6 is a block diagram illustrating a configuration of a system that realizes real-time video distribution using WebRTC by combining the first to third embodiments. In FIG. 6, the server 21 has a database of video features and viewing frequency inside and distributes it to each of the PCs 22, 23 and 24. The PCs 22, 23, and 24 perform from imaging to encoding / distribution according to the database. However, the video from the cameras 25, 26, and 27 connected to the PCs 22, 23, and 24 may be collected and encoded / distributed by the server 21. Further, in the communication between the server 21 and the PCs 22, 23, 24, a known technique such as WebRTC may be used.

  In the above description, each tile area adjacent to the tile area has not been described as having an overlapping area (a portion overlapping each other). However, the tile area may have an overlapping area by a predetermined amount.

  As described above, when encoding a tiled area of a video, the video content is ignored (or people often see it) by avoiding inefficient bit rate allocation that ignores video content or considers only video features. The bit rate is changed and the encoding is performed efficiently by changing the bit rate according to the listening position) and the actual viewing frequency of the person.

  Even when the user's viewing tendency changes (such as popular subjects change depending on the time) by periodically feeding back the video content and viewing frequency to the encoder and re-encoding it regularly, it is always efficient. Dynamic bit rate allocation was added.

  In addition, the video content (video features such as the amount of edges and saliency) and the encoding result (assigned bit rate) stored in each tiled area are stored, and a new video (not sure of the viewing frequency) is encoded. At this time, the bit rate to be assigned is automatically determined from past encoding results.

  With this configuration, encoding of tiled areas of video that has been performed uniformly until now is assigned a higher bit rate according to the user's viewing frequency, and the higher the bit rate is assigned, the more it was not seen. Since the bit rate can be suppressed as the position is increased, the bit rate of the entire video can be saved (reduced) while maintaining the user's quality of experience.

  In addition, the position that is often seen in the video, that is, the position where the subject is shown by person tracking or the salient obtained by video analysis (salient, in the image processing, the salientity map) is attracting attention from the user. By changing the bit rate allocation according to the ratio that includes many areas (there is a method to clarify such a part), the bit rate of the entire video can be saved (reduced) while maintaining the user's quality of experience as well. it can.

  In addition, by accumulating the video contents and encoding results of each tiled area, and by rotating a learning cycle such as assigning what bit rate when there is any video feature, new video that does not know the viewing frequency etc. Efficient bit rate allocation is possible even when the characteristics of commonly seen parts change.

  Note that saving the bit rate has the effect of saving the network bandwidth at the time of video distribution, saving the storage area at the time of saving the video, saving the computer resources of the encoder at the time of video encoding, and saving the encoding time.

  You may make it implement | achieve the encoding apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  When encoding a tiled area of a video, it can be applied to an application in which it is indispensable to perform efficient bit rate allocation according to the video content.

  DESCRIPTION OF SYMBOLS 1 ... Video imaging | photography part, 2 ... Video storage part, 3 ... Video compression part, 4 ... Compressed video storage part, 5 ... Video distribution part, 6 ... Bit rate storage part, 7: Bit rate determination unit, 8 ... Network, 9 ... Video reception unit, 10 ... Video buffer unit, 11 ... Video development unit, 12 ... Video drawing unit, 13 ...・ User input part, 14 ... Video request part

Claims (5)

An encoding method for encoding video,
A tile dividing step of dividing the video into tiled area videos;
A first encoding step for encoding the tiled region image;
A distribution frequency storage step of storing a distribution frequency that is a frequency of distributing the encoded tiled region video according to a user's viewing request;
A bit rate calculating step of calculating a bit rate for the tiled region image by multiplying the distribution frequency by a predetermined bit rate;
And a second encoding step of re-encoding the tiled area video according to the calculated bit rate. An encoding method for encoding video,
An attribute value calculating step for calculating a value of a predetermined attribute relating to the video;
A tile dividing step of dividing the video into tiled area videos;
A bit rate calculating step of calculating a bit rate for the tiled region image by multiplying a predetermined bit rate by a ratio of the value of the attribute included in the tiled region image;
An encoding method comprising: an encoding step of encoding the tiled area image according to the calculated bit rate. An encoding method for encoding video,
An attribute value calculating step for calculating a value of a predetermined attribute relating to the video;
A tile dividing step of dividing the video into tiled area videos;
A first encoding step for encoding the tiled region image;
A distribution frequency storage step of storing, for each value of the predetermined attribute, a distribution frequency that is a frequency of distributing the encoded tiled region video according to a user's viewing request;
A distribution ratio calculating step for calculating a ratio at which the tiled area video corresponding to the value of the predetermined attribute is distributed;
When generating the tiled area image from the newly input image, the bit rate assigned to the tiled area image is multiplied by the distribution rate corresponding to the attribute value of the tiled area image. A bit rate calculation step to calculate
And a second encoding step for encoding the tiled area video of the video newly input according to the calculated bit rate. The bit rate calculating step includes a plurality of predetermined bit rates,
It said second encoding step, the encoding method according to any one of claims 1 or claim 3, characterized in that encoding the tiled area image for each of the plurality of bit rates.   The encoding program for making a computer perform the encoding method of any one of Claim 1 to 4.

Images