JP2015142376A - Image decoding device, image transmission device, image processing system, image decoding method and image transmission method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image decoding device capable of efficiently processing an encoded image.SOLUTION: The image decoding device includes: a request unit 1103 for requesting transmission of an image to an image transmission device; a transmission unit 1102 for transmitting to the image transmission device, range information indicating a range, in a part of the image, selected by operation of a user; a reception unit 1104 for receiving a part of the image from the image transmission device by receiving the entire image from the image transmission device in the case where the image belongs to a first type in the state where the range in a part of the image is being selected, and by receiving only one or more tiles, corresponding to the range in a part of the image, for which the reference to the outside of a range corresponding to each tile in inter-plane prediction is prohibited, among a plurality of tiles included in the image in the case where the image belongs to a second type; and a decoding unit 1105 for decoding the entire or partial image received from the image transmission device.

Description

  The present invention relates to an image decoding device that receives an encoded image from an image transmission device and decodes an image received from the image transmission device.

  Currently, a High Efficiency Video Coding (HEVC) system is being studied as a new image codec standard (see Non-Patent Document 1). Since the HEVC method enables transmission / reception of high-quality images at a low bit rate as compared with the conventional method, it is assumed that data including images will be further transmitted / received in the future.

JCTVC-N1005, “High Efficiency Video Coding (HEVC) Range extension text specification draft 4”

  In recent years, in processing of data including images, there are an increasing number of services that accompany images with information related to images such as subtitles, program information, advertisements, commercials, and the like, and that enhance user convenience. However, such services can handle unnecessary information.

  Therefore, the present invention provides an image decoding apparatus and the like that can efficiently process an encoded image.

  An image decoding device according to an aspect of the present invention is an image decoding device that receives an encoded image from an image transmission device and decodes the image received from the image transmission device, the image transmission device A request unit for requesting transmission of the image, and range information indicating a range that is a partial range of the image and is selected according to a user's operation from the entire range of the image. In a state where a transmission unit to transmit and a partial range of the image are selected, (i) when the image belongs to the first type, the entire image is received from the image transmission device; (ii) When the image belongs to the second type, among the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and out of the range corresponding to each tile in inter-plane prediction Reference to is prohibited By receiving only one or more tiles, the receiving unit receives a part of the image from the image transmitting device, and the receiving unit decodes the whole or a part of the image received from the image transmitting device. A decoding unit.

  Note that these comprehensive or specific aspects may be realized by a system, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM. The present invention may be realized by any combination of an integrated circuit, a computer program, and a recording medium.

  An image decoding device or the like according to one embodiment of the present invention can efficiently process an encoded image.

FIG. 1 is a diagram illustrating a reference relationship between blocks in a tile. FIG. 2 is a diagram illustrating a block reference relationship in the restriction tile. FIG. 3 is a block diagram illustrating a configuration example of the image processing system according to the first embodiment. FIG. 4 is a block diagram illustrating a configuration example of the image decoding apparatus according to the first embodiment. FIG. 5 is a diagram illustrating an example of syntax indicating motion compensation restriction according to the first embodiment. FIG. 6 is a diagram illustrating an example of a bitstream in which syntax indicating motion compensation restriction according to Embodiment 1 is stored. FIG. 7 is a diagram illustrating an example of a processing flow of the image processing system according to the first embodiment. FIG. 8 is a diagram illustrating an example of image data selection processing according to the first embodiment. FIG. 9 is a diagram illustrating a first allocation example for a plurality of tiles in an image according to the first embodiment. FIG. 10 is a diagram illustrating a second assignment example for a plurality of tiles in the image according to the first embodiment. FIG. 11 is a diagram illustrating a third allocation example for a plurality of tiles in an image according to the first embodiment. FIG. 12 is a diagram illustrating a fourth assignment example for a plurality of tiles in the image according to the first embodiment. FIG. 13 is a diagram illustrating an example of timing for changing the assignment to a plurality of tiles in an image according to the first embodiment. FIG. 14 is a diagram illustrating an example of an operation for changing assignment to a plurality of tiles in an image according to the first embodiment. FIG. 15 is a diagram illustrating an example of a processing flow for changing assignment to a plurality of tiles in an image according to the first embodiment. FIG. 16 is a diagram illustrating an example of an operation of scrambling a part of a plurality of tiles in an image according to the first embodiment. FIG. 17 is a block diagram illustrating a configuration example of a processing unit of the server according to the first embodiment. FIG. 18 is a diagram illustrating a display example of the first mode according to the second embodiment. FIG. 19 is a diagram illustrating a display example of the second mode according to the second embodiment. FIG. 20 is a diagram illustrating a display example of the third mode according to the second embodiment. FIG. 21 is a schematic diagram illustrating a configuration example of a broadcasting system according to the fourth embodiment. FIG. 22 is a block diagram illustrating a configuration example of a television according to Embodiment 4. FIG. 23 is a diagram illustrating an example of a processing flow of the broadcasting system according to the fourth embodiment. FIG. 24 is a diagram illustrating an example of a processing flow of the image processing system according to the fifth embodiment. FIG. 25 is a diagram showing an example (continued) of the processing flow of the image processing system according to the fifth embodiment. FIG. 26 is a diagram illustrating an example of the timing at which the entire display and the partial display according to the fifth embodiment are switched. FIG. 27 is a diagram showing an example of partial display according to the fifth embodiment. FIG. 28 is a diagram showing an example of the entire display according to the fifth embodiment. FIG. 29 is a block diagram illustrating an example of a characteristic configuration according to the fifth embodiment. FIG. 30 is a diagram illustrating an example of a processing flow of characteristic operations according to the fifth embodiment. FIG. 31 is an overall configuration diagram of a content supply system that implements a content distribution service. FIG. 32 is an overall configuration diagram of a digital broadcasting system. FIG. 33 is a block diagram illustrating a configuration example of a television. FIG. 34 is a block diagram illustrating a configuration example of an information reproducing / recording unit that reads and writes information from and on a recording medium that is an optical disk. FIG. 35 is a diagram showing an example of the structure of a recording medium that is an optical disk. FIG. 36A is a diagram illustrating an example of a mobile phone. FIG. 36B is a block diagram illustrating a configuration example of a mobile phone. FIG. 37 is a diagram showing a structure of multiplexed data. FIG. 38 is a diagram schematically showing how each stream is multiplexed in the multiplexed data. FIG. 39 is a diagram showing in more detail how the video stream is stored in the PES packet sequence. FIG. 40 is a diagram showing the structure of TS packets and source packets in multiplexed data. FIG. 41 shows the data structure of the PMT. FIG. 42 shows an internal structure of multiplexed data information. FIG. 43 shows the internal structure of stream attribute information. FIG. 44 is a diagram showing steps for identifying video data. FIG. 45 is a block diagram illustrating a configuration example of an integrated circuit that implements the moving picture coding method and the moving picture decoding method according to each embodiment. FIG. 46 is a diagram showing a configuration for switching the drive frequency. FIG. 47 is a diagram illustrating steps for identifying video data and switching between driving frequencies. FIG. 48 is a diagram showing an example of a look-up table in which video data standards are associated with drive frequencies. FIG. 49A is a diagram illustrating an example of a configuration for sharing a module of a signal processing unit. FIG. 49B is a diagram illustrating another example of a configuration for sharing a module of a signal processing unit.

(Knowledge that is the basis of the present invention)
The present inventor has found problems regarding an image transmission device, an image decoding device, and the like. This will be specifically described below.

  H. In the ITU-T standard called 26x or the ISO / IEC standard called MPEG-x, a picture is divided into a plurality of rectangular units called tiles as shown in FIG. Each tile cannot refer to another tile area in the screen (in the picture). For example, when the second tile is an encoding target or decoding target tile, the second tile cannot refer to the first tile.

  However, in the case of image signals that are different in time, the current block in the encoding target or decoding target tile can refer to a different area from the encoding target or decoding target tile. For example, the mth picture in FIG. 1 corresponds to the image signal at time t−1, and the nth picture corresponds to the image signal at time t. These are temporally different image signals. In this case, the encoding target or decoding target block C in the encoding target or decoding target tile can refer to a region different from the region of the second tile.

  Here, a tile or block or the like refers to another tile or another block or the like means that another tile or another block or the like is referred to when the tile or block or the like is processed. FIG. 1 shows an example in which the reference region R belonging to the third tile of the mth picture at time t−1 is used as a reference image when processing the block C belonging to the second tile of the nth picture at time t.

  On the other hand, when such a reference is allowed, there are the following problems. For example, when the screen size (picture size) is very large, there is a method in which a plurality of tile areas are processed in parallel by different LSIs. However, as described above, when reference to image signals at different times is allowed, it may be necessary to write image signals generated by different LSIs into one large memory and share the memory. In this case, it may be necessary to synchronize separate LSIs. For this reason, the processing amount for the synchronization processing may increase.

  Non-Patent Document 1 describes a method for describing information indicating that reference to image signals having different times is restricted in an encoded stream. In this case, as illustrated in FIG. 2, the processing target block C refers to the reference region R in the same tile region (second tile) with respect to temporally different image signals (m-th picture). Thereby, when a plurality of LSIs perform parallel processing on a plurality of tile areas, it is not necessary to output all the reference images to the shared memory.

  Hereinafter, such tiles are referred to as MV restricted tiles or restricted tiles. Further, the referenceable area is not limited to the same tile area. For example, the reference area R may be designated in an area including the first tile and the second tile of the m-th picture.

  By using such an MV restricted tile, it is possible to perform encoding or decoding without making a partial area of the image dependent on other areas.

  Therefore, the inventors considered that transmission / reception and decoding of images can be performed in tile units.

  For example, an image decoding device according to an aspect of the present invention is an image decoding device that receives an encoded image from an image transmission device and decodes the image received from the image transmission device, wherein the image A request unit for requesting transmission of the image to the transmission device, and range information indicating a range of the partial range of the image selected in accordance with a user's operation from the entire range of the image. In a state where a transmission unit to be transmitted to the device and a partial range of the image are selected, (i) when the image belongs to the first type, the entire image is received from the image transmission device; ii) When the image belongs to the second type, among the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and corresponding to each tile in inter prediction Reference out of range A reception unit that receives a part of the image from the image transmission device by receiving only one or more prohibited tiles, and the whole or a part of the image received from the image transmission device by the reception unit The decoding part which decodes.

  Thereby, the image decoding apparatus can switch between decoding of the entire image and decoding of a part of the image according to the type of the image. Therefore, the image decoding apparatus can efficiently process the encoded image.

  In addition, for example, the image decoding device may further include a display control unit that displays all or part of the image decoded by the decoding unit on the first display device.

  Thereby, the image decoding apparatus can switch between displaying the entire image and displaying a part of the image.

  For example, when the receiving unit receives a part of the image, the receiving unit receives the part of the image according to the stored range information, and the decoding unit decodes the part of the image. The display control unit decodes a part of the received image according to the stored range information, and the display control unit receives and decodes according to the stored range information when displaying the part of the image. A part of the image may be displayed.

  Accordingly, the image decoding apparatus can receive, decode, and display a part of the image according to the range information stored in the image decoding apparatus or the image transmission apparatus.

  Further, for example, when the display control unit displays a part of the image after displaying the whole of another image, the display control unit displays the part of the image received and decoded according to the stored range information. May be.

  Thereby, the image decoding apparatus can switch the display from the entire display of another image to the partial display of the image.

  In addition, for example, when the display control unit further stops the display of the part of the image after stopping the display of the part of the image, the image received and decoded according to the stored range information A part of may be displayed.

  As a result, the image decoding apparatus can resume displaying a part of the image in accordance with the range information stored in the image decoding apparatus or the image transmission apparatus.

  Further, for example, the display control unit may further display a part of the image received and decoded according to the stored range information on a second display device different from the first display device.

  Thereby, the image decoding apparatus can take over the display format in one display apparatus to another display apparatus.

  For example, when the range information is not stored, the transmission unit transmits the range information to the image transmission device, and the reception unit stores the range information, and the image When a part is received, the part of the image may be received according to the stored range information without the range information being transmitted to the image transmission apparatus by the transmission unit.

  Thereby, the image decoding apparatus can receive, decode, and display a part of the image, for example, according to the range information transmitted and stored from the other image decoding apparatus.

  In addition, for example, the image decoding apparatus further includes a selection unit that selects a partial range of the image according to an operation of the user, and the transmission unit indicates the range selected by the selection unit. Range information may be transmitted to the image transmission device.

  Thereby, the image decoding apparatus can appropriately select a portion to be received and decoded in accordance with a user operation.

  Further, for example, in a state where a partial range of the image is selected, the reception unit receives the entire image when the image belongs to the first type recommended to be displayed entirely. When the image belongs to the second type whose display is not recommended, a part of the image may be received.

  Accordingly, the image decoding apparatus can receive the entire image when the entire display is recommended, and can receive a part of the image when the entire display is not recommended. Therefore, the image decoding apparatus can appropriately receive and decode an image.

  In addition, for example, in a state where a partial range of the image is selected, the reception unit receives the entire image when the image belongs to the first type that is a commercial, and the image is a commercial. If the image belongs to the second type different from the above, a part of the image may be received.

  As a result, the image decoding apparatus may receive the entire image in a commercial in which the entire image is properly displayed, and may receive a part of the image in the main part in which the entire image may not be displayed. it can.

  In addition, for example, a flag indicating whether the image belongs to the first type or the second type is given to the image, and the receiving unit performs the whole or one of the images according to the flag. May be received.

  Thereby, the image decoding apparatus can receive the whole or part of the image according to the flag given to the image.

  For example, the image transmission device according to an aspect of the present invention is an image transmission device that transmits an encoded image to the image decoding device, and receives a request for transmitting the image from the image decoding device. A reception unit that receives from the image decoding device range information indicating a range of a part of the image of the entire range of the image, the range being selected according to a user operation, In a state where a part of the range is selected, (i) when the image belongs to the first type, the entire image is transmitted to the image decoding device, and (ii) the image belongs to the second type One or more tiles corresponding to a partial range of the image among a plurality of tiles included in the image, and one or more tiles for which reference to the outside of the range corresponding to each tile is prohibited in inter-plane prediction Only tiles By signal, and a transmitter that transmits a portion of the image to the image decoding apparatus.

  Thereby, the image transmission apparatus can switch between transmission of the entire image and transmission of a part of the image according to the type of the image. Therefore, the image transmission apparatus can efficiently process the encoded image.

  In addition, for example, in a state where a partial range of the image is selected, the transmission unit transmits the entire image when the image belongs to the first type recommended to be displayed entirely. When the image belongs to the second type whose display is not recommended, a part of the image may be transmitted.

  Accordingly, the image transmission apparatus can transmit the entire image when the display of the entire image is recommended, and can transmit a part of the image when the display of the entire image is not recommended. Therefore, the image transmission apparatus can appropriately transmit an image.

  In addition, for example, in a state where a partial range of the image is selected, the transmission unit transmits the entire image when the image belongs to the first type that is a commercial, and the image is a commercial. If the image belongs to the second type different from, a part of the image may be transmitted.

  As a result, the image transmission apparatus may transmit the entire image in a commercial where the entire display of the image is appropriate, and may transmit a part of the image in the main part in which the entire image may not be displayed. it can.

  In addition, for example, a flag indicating whether the image belongs to the first type or the second type is given to the image, and the transmission unit performs the whole or one of the images according to the flag. May be transmitted.

  Thereby, the image transmission apparatus can transmit the whole or a part of the image according to the flag given to the image.

  Note that these comprehensive or specific aspects may be realized by a system, method, integrated circuit, computer program, or non-transitory recording medium such as a computer-readable CD-ROM. The present invention may be realized by any combination of an integrated circuit, a computer program, or a recording medium.

  Hereinafter, embodiments will be specifically described with reference to the drawings. It should be noted that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

  Moreover, the same code | symbol is attached | subjected to the same element and description may be abbreviate | omitted.

(Embodiment 1)
In the first embodiment, an image processing system that decodes an image from a bitstream obtained by dividing an image into a plurality of tiles and encoded, an image decoding method, and a bitstream generation method will be described.

  In particular, in the first embodiment, the bitstream includes information for enabling parallel processing of encoding or decoding. For example, the image is divided into a plurality of areas each called a tile. Information that can specify motion prediction or motion compensation restriction that a bitstream refers only to pixels of one or more corresponding tiles in image signals at the same or different times during motion prediction processing or motion compensation processing. including.

[Description of configuration]
FIG. 3 shows a configuration example of the image processing system 100 according to the first embodiment. The image processing system 100 illustrated in FIG. 3 includes a decoding device (image decoding device) 200 and a server (image transmission device) 300. The decoding device 200 includes a processing unit 210, a storage unit 220, a communication unit 230, a control unit 240, and a display unit 250. The server 300 includes a processing unit 310, a storage unit 320, a communication unit 330, and a control unit 340. The storage unit 320 stores the image streams A, B, and C in an encoded state.

  Note that the storage unit 220 of the decoding device 200 and the storage unit 320 of the server 300 may be an external memory or the like, or may be an external memory connected via a wired or wireless network. Currently, the server 300 has memories arranged in various countries in the world that hold similar data so that images are distributed from the nearest location according to the position of the decoding device 200 using GPS or the like. There are many.

  Similarly, the other components may be connected to each other in a wired or wireless manner, and may execute the entire processing by hardware or software, and may not be physically integrated into one device.

  In addition, each unit may be configured by one or more processors, one or more circuits, one or more memories, or any combination thereof, or may be a general-purpose circuit that executes software that performs processing of each unit. .

  FIG. 4 is a block diagram illustrating a configuration example of the decoding device 200 illustrated in FIG. FIG. 4 particularly shows a configuration example of the processing unit 210 of the decoding device 200. The decoding device 200 illustrated in FIG. 4 includes an entropy decoding unit 201, an inverse quantization unit 202, an inverse transform unit 203, a block memory 204, a frame memory 205, an intra prediction unit 206, and an inter prediction unit 207.

  The entropy decoding unit 201, the inverse quantization unit 202, the inverse transform unit 203, the intra prediction unit 206, and the inter prediction unit 207 are included in the processing unit 210. The block memory 204 and the frame memory 205 may be included in the storage unit 220 or may be components different from the storage unit 220.

  The processing unit 210 receives a bit stream that is an encoded image signal acquired by the communication unit 230 from the outside.

  The entropy decoding unit 201 performs entropy decoding on the bitstream. The inverse quantization unit 202 performs inverse quantization on the coefficient obtained by entropy decoding. The inverse transform unit 203 performs inverse frequency transform on the coefficient that has been subjected to inverse quantization. A frame (picture) composed of blocks obtained by the inverse frequency transform is output to the display unit 250.

  The block memory 204 stores a block obtained by inverse frequency conversion. The frame memory 205 stores a frame composed of blocks obtained by inverse frequency conversion. The intra prediction unit 206 performs intra prediction with reference to the blocks stored in the block memory 204. The inter prediction unit 207 performs inter prediction with reference to the frame stored in the frame memory 205.

  FIG. 5 is a diagram illustrating an example of syntax indicating motion compensation restriction. The bitstream includes the syntax shown in FIG.

  FIG. 6 is a diagram illustrating an example of a bitstream in which the syntax illustrated in FIG. 5 is stored. The syntax shown in FIG. 5 is written in a header called supplementary enhancement information (SEI) as shown in FIG. The SEI is a header for storing auxiliary information that is commonly used for subsequent encoded image data. However, of course, the syntax shown in FIG. 5 may be described in headers called SPS and PPS, or other headers.

  The processing unit 210 writes necessary data to the block memory 204 and the frame memory 205, or reads necessary data from these memories. Then, the entropy decoding unit 201, the inverse quantization unit 202, the inverse transform unit 203, the intra prediction unit 206, and the inter prediction unit 207 in the processing unit 210 are H.264. A decoding process as defined by 26x is performed to decode an image from the bitstream.

  Based on an instruction from the processing unit 210 or the control unit 240, the communication unit 230 transmits a bitstream transmission request to the outside and receives the bitstream from the outside.

  In response to such a request from the decoding device 200, the server 300 reads the corresponding bitstream from the storage unit 320 based on the control performed by the control unit 340, and transmits the image data processed by the processing unit 310 to the communication unit. The data is transmitted from 330 to the decoding device 200. For example, the server 300 receives the request for the image stream A from the decoding device 200 and transmits the image stream A to the decoding device 200.

[Description of operation]
FIG. 7 is a diagram illustrating a processing flow of the operation of the image processing system 100 according to the first embodiment.

  The decoding apparatus 200 transmits a request for an image that the user wants to view to the external server 300 (S501). Currently, a processing device that receives a request and a server that sends an image to the decoding device 200 in accordance with an instruction from the processing device are often separate, and companies that operate each are also arranged. Countries are often different. Here, for the sake of simplicity, the description will be made assuming that the external server 300 includes the apparatus that receives the request and the server that sends the image.

  The decoding apparatus 200 can receive a more appropriate image by sending additional information to the server 300 in addition to the request. The additional information includes, for example, position information, user information, performance information, and the like. In the case where the server 300 has several images with different formats or the like for images showing the same content, the additional information is for the server 300 to transmit a more appropriate image to the decoding device 200 among the several images. Information.

  The server 300 receives the image request (S502), and further determines whether there is additional information (S503).

  When there is additional information (Yes in S503), the server 300 selects image data to be transmitted based on the additional information (S504). When there is no additional information (No in S503), the server 300 selects image data to be transmitted based on a predetermined condition (S505). Here, the image data includes a main image corresponding to a request from the decoding device 200 and sub-information including text or an image.

  The server 300 transmits a bit stream including the selected image data to the decoding device 200 (S506). The decoding device 200 receives the bit stream (S507). Then, the decoding device 200 decodes the encoded image included in the bit stream and displays the decoded image (S508).

[Detailed explanation of selection processing]
FIG. 8 is a diagram showing an example of the image data selection process (S503 to S505) shown in FIG.

  In the determination processing (S601 to S603) in FIG. 8, it is determined whether or not the additional information includes performance information, position information, and user information. These determination processes may be performed in any order, and may be performed in parallel. Note that these determination processes are merely examples. Other procedures including one or more steps of selecting sub-information based on some information may be used for the image data selection process.

  The determination process may not be performed on all of the performance information, the position information, and the user information. Information other than these may be used for image data selection processing, or sub-information may be selected based on performance information.

  In FIG. 8, for example, when it is determined that the additional information includes performance information (Yes in S601), a standard that can be decoded by the decoding device 200 based on the performance information, or the decoding device 200 among the standards. The profile or level to which is supported is determined. Examples of standards are MPEG-2, MPEG-4 Visual, H.264. H.264 / AVC or H.264 H.265 / HEVC. The server 300 selects one of a plurality of types of main images stored in the storage unit 320 of the server 300 based on the determination result (S604).

  When the decoding apparatus 200 supports a plurality of standards or a plurality of profiles, the server 300 further converts a plurality of main images into a higher quality and more efficient standard or profile based on the situation such as the transmission path. One compliant main image may be selected. Alternatively, the server 300 uses a distribution system (for example, MPEG-DASH or the like) that sends information linked to each candidate of the main image so that the decoding apparatus 200 can select the main image based on the situation such as the transmission path. The information may be distributed.

  If it is determined that the performance information is not included (No in S601), the server 300 selects a predetermined main image, for example, a main image that conforms to the most basic standard or profile (S605).

  If it is determined that position information is included in the additional information (Yes in S602), the server 300 selects sub-information based on the country or region indicated by the position information as the position of the decoding device 200 (S606). This sub-information is information indicating, for example, the local weather, a radio program or a television program broadcast in the area, or a grocery store or restaurant in the vicinity of the area in the language of the area.

  When it is determined that the position information is not included in the additional information (No in S602), the server 300 may not select the sub information based on the position or may select the predetermined sub information. . For example, when the location information cannot be acquired and the optimal language is unknown, the server 300 may select general news written in English as sub information.

  When it is determined that the user information is included in the additional information (Yes in S603), the server 300 selects the sub information based on the user information. For example, the server 300 selects the sub information based on the gender or age of the user (S607). Alternatively, the server 300 may acquire information related to a network of close friends of the user via the SNS, and select the sub information based on the user's preference estimated from the information.

  For example, if the user likes the car, the server 300 selects the news regarding the latest product or the advertisement of the car-related goods related to the exterior or interior of the car as the sub-information.

  When it is determined that the user information is not included in the additional information (No in S603), the server 300 may not select the sub information based on the user information, or may select the predetermined sub information. Also good. For example, the server 300 may select information related to the main image such as goods or movie information related to the main image as the sub-information.

  As described above, the server 300 selects or generates one or more sub-information as a plurality of sub-information candidates.

[Transmission method of main image and sub information]
In the transmission process (S506) of FIG. 7, an image including the main image and sub information selected as described above is constructed using a plurality of tiles, and a bitstream including the constructed image is transmitted to the decoding device 200. To do. Basically, each of these multiple tiles is an MV restricted tile, but these multiple tiles may include tiles that are not MV restricted tiles.

  FIG. 9 is a diagram illustrating a first allocation example for a plurality of tiles in an image according to the first embodiment. As illustrated in FIG. 9, a tile is a rectangular region obtained by dividing one image (picture) by vertical or horizontal straight lines.

  For example, as shown in FIG. 9, one image is divided into a first tile and a second tile. Then, the processing unit 310 performs H.264 processing. In accordance with H.265 / HEVC, the selected main image is encoded as the first tile, and the selected sub-information is encoded as the second tile. At that time, the processing unit 310 encodes each of the first tile and the second tile as an MV restricted tile. FIG. 9 shows an example in which, for example, weather and news are selected as sub-information, and they are encoded in a form that is arranged side by side.

  FIG. 10 is a diagram illustrating a second assignment example for a plurality of tiles in the image according to the first embodiment. FIG. 10 shows a configuration example of a main image and sub information different from the example of FIG. Here, the first tile and the fifth to eighth tiles correspond to the sub-information, and the second to fourth tiles correspond to the main image.

  Information related to the main image, such as program information of the main image, is included in the first tile, and an advertisement is included in the fifth tile. Further, the sub information is divided into sixth to eighth tiles so as to correspond to the main image divided into the second to fourth tiles.

  As shown in FIG. 10, when each of the second to fourth tiles corresponding to the main image is associated with a specific person or a microphone, object coding is used as the audio codec method. May be. Then, when the sound of a specific position or microphone can be reproduced at the time of decoding, the decoding device 200 may reproduce sound corresponding to the image displayed on the tile. Further, a link to the sound adjusted so that the sound is mainly used may be pasted on the corresponding sub-information tile.

  FIG. 11 is a diagram illustrating a third allocation example for a plurality of tiles in an image according to the first embodiment. FIG. 11 illustrates an example in which the main image is divided into a first tile and a second tile, and the sub information is divided into a third tile and a fourth tile. In this example, two types of sub-information, weather and news, are encoded with different tiles.

  FIG. 12 is a diagram illustrating a fourth assignment example for a plurality of tiles in the image according to the first embodiment. FIG. 12 shows an example in which some tiles of the main image are encoded by replacing with sub-information. Specifically, FIG. 12 shows an example in which the main image is divided into the first tile to the 35th tile, and the seventh tile and the 29th tile to the 32nd tile are replaced with the sub information. Yes. In this example, an MV restriction tile is used for the seventh tile in accordance with the time or timing for displaying each sub-information.

  FIG. 13 is a diagram illustrating an example of timing for changing the assignment to a plurality of tiles in an image according to the first embodiment. Specifically, in FIG. 13, the tile division format is switched for an I picture close to a predetermined time.

For example, when weather information is displayed on the seventh tile for 5 seconds every 10 seconds, the MV restriction tile is used for the seventh tile of the picture for 5 seconds, and the weather information is embedded in a part of the main image. . In FIG. 13, the picture is divided so that the seventh tile is constructed at I ₀ . The division format is maintained for the pictures up to I ₁ . Weather information at I ₁ is returned to the main image, limitations and division type is changed to the initial state. Of course, the present invention is not limited to this, and the weather information may be returned to the main image in a state where the division format and the restriction are continued.

  When the division format is changed while the user is viewing, or when the allocation format between the main image and the sub information is changed, the server 300 informs the decoding device 200 that the format has been changed. Notice. For example, when using a distribution system such as MPEG-DASH, the server 300 changes the MPD file and notifies that the MPD file has been changed.

Thereafter, the tile is divided again so that the seventh tile is constructed with the I picture close to the time when 10 seconds have passed (I _{3 in} FIG. 13). Then, the server 300 replaces a part of the main image with the sub information, and encodes the image using the MV restriction.

  FIG. 14 is a diagram illustrating an example of an operation for changing assignment to a plurality of tiles in an image according to the first embodiment. Specifically, in FIG. 14, the sub-information displayed in the 29th to 32nd tiles is returned to a part of the main image and displayed on the 7th tile at the next timing based on an instruction from the user. An example is shown in which the sub-information is moved to the position of the first tile. Here, the next timing is, for example, the timing of the I picture after the instruction.

  FIG. 15 is a diagram illustrating an example of a processing flow for changing assignment to a plurality of tiles in an image according to the first embodiment. For example, as in the example of FIG. 15, the decoding device 200 sends a request for changing the position for displaying the sub information from the seventh tile to the first tile to the server 300 via the communication unit 230 based on an instruction from the user. Transmit (S801 to S803). Upon receiving the request, the server 300 transmits a bit stream in which the tile column is changed so that the sub information is displayed on the first tile at the next appropriate timing.

  Then, the decoding device 200 receives the changed bit stream (S801). Then, the decoding device 200 decodes the encoded image included in the changed bitstream, and displays the decoded image (S804). When the user performs an operation for deleting the display of the sub information (Yes in S802), the decoding apparatus 200 requests a display change corresponding to the deletion of the sub information (S803). In this case, the server 300 transmits a bit stream in which the sub information is returned to a part of the main image.

  In FIG. 15, the decoding device 200 sends a display change request to the server 300. When a distribution system that can switch the received stream according to the situation, such as MPEG-DASH, is used, the decoding device 200 does not send a request and selects an appropriate one from a plurality of options transmitted from the server 300. You may choose an image. Further, the decoding device 200 may first select a default image. When the server 300 has information to be notified, such as emergency disaster information, the server 300 may transmit the information so that the information is selected.

  Of course, even when such a distribution system is used, if there is no appropriate option, the decryption apparatus 200 may request the server 300 to create an appropriate option.

  In FIG. 15, the decoding apparatus 200 receives a new stream at the next timing after the display change request. The decoding device 200 may display, for example, one of the following two on the tile to be changed in display until reception and decoding of a new stream are completed. That is, the decoding apparatus 200 may perform any one of (1) continuation of the current display and (2) display of a predetermined background image.

  In the case of (1), since no new processing occurs in the decoding device 200, the display is stable. On the other hand, the user may feel that the response is bad because the display is not changed despite the request for movement or deletion.

  Therefore, the decoding device 200 may display (1) when the period until the reception and decoding of a new stream is completed is short. The decoding apparatus 200 should change the operation so that the display of (2) is performed when the period until the reception and decoding of the new stream is over a certain period.

  In the case of (2), the decoding device 200 can visually notify the user that the request has been accepted by displaying the background image. Furthermore, in the case of (2), the processing is also simple. The background image is gray, a logo, a pattern image, an image stored in advance in the memory, or a tile portion of the main image that is most recently decoded and held in the memory. The background image is not limited to a still image.

  For example, the decoding apparatus 200 may change the sub information from an image to text data, reduce the area, change the sub information to other sub information, and the like. You may do it based on. In a case where the data amount of the sub information is large and processing is heavy, or when the sub information of the moving image flickers to prevent concentration on the main image, the decoding apparatus 200 changes the sub information to text data. Thus, a state suitable for viewing the main image can be constructed.

  Note that the decoding apparatus 200 can accept the user's instruction by selecting the movement source using the remote controller and instructing the movement destination to move the sub information with an arrow key or the like. If the decryption apparatus 200 has a touch panel, the user can instruct the destination from the source by a touch operation such as swipe, or the deletion can be instructed by swiping from the inside of the screen to the outside of the screen. The device 200 can accept a user instruction.

  Similarly, the decoding apparatus 200 can accept a movement or deletion instruction by using information on a motion or gesture detected by an acceleration sensor, a depth sensor, or the like. Here, an example in which the user instructs movement or deletion is shown. However, when displaying sub-information such as emergency disaster information to be notified on the decoding device 200, the server 300 may select such sub-information by default.

  As shown in FIGS. 9 to 12, when a part of the main image is replaced with sub information and the sub information is encoded as a tile in a picture, the sub information includes four corners of the screen, left and right upper and lower edges, and the like. It is better to arrange in a region that does not hinder viewing of the main image.

  However, when a main image provided for a fee is distributed in a trial period, tiles of sub information such as advertisements may be arranged in an area that prevents viewing of the main image. Thereby, the decoding device 200 can act so that the viewer wants to view a more appropriate main image. Thus, the decryption apparatus 200 enables a content distribution service that collects an appropriate copyright fee from the user.

  FIG. 16 shows an example of an operation of scrambling a part of a plurality of tiles in an image according to the first embodiment. For example, in FIG. 16, first, all tiles of the image correspond to the main image. Then, some tiles may be gradually changed to MV restricted tiles at the timing when the I picture is displayed. In the changed tile, sub information, a gray image, a scrambled image, or a main image that is rough or blurred and is difficult to view may be displayed.

  When the user views the main image, for example, the tile is selected using the above-described UI, and an appropriate viewing fee is paid through the electronic payment system, so that a display that prevents viewing by the tile is displayed. It may be canceled. The user may select a viewing area, and the viewing fee may be determined based on the size of the area. Tiles for displaying sub-information may be arranged in an unselected area, and an advertisement or the like may always be displayed.

  As described above, the decoding device 200 receives the transmitted bit stream, reads the standard to which the bit stream complies, from the header, executes the decoding process corresponding to the standard, and displays the decoded image on the display unit 250. indicate.

  On the other hand, the server 300 holds a plurality of bitstreams corresponding to a plurality of patterns having the same number of tiles and different contents so as to be able to respond to the request from the decoding device 200 as described above. For example, when using MPEG-Dash, the server 300 may transmit a media presentation description (MPD) file including segment information linked to a plurality of bitstreams to the decoding device 200.

[effect]
In the first embodiment, as described above, the image is divided into a plurality of tiles, and the selected main image and the selected sub-information are encoded into separate tiles. As a result, an image accompanied with appropriate information is transmitted to the decoding apparatus 200 in the form of one stream. This also allows the server 300 to transmit the sub information to be transmitted, and the decoding device 200 can receive appropriate sub information.

  As described above, the image processing system 100 according to the first embodiment can transmit and receive an image accompanied with appropriate information.

[Supplement]
In the image coding in the transmission process (S506) of FIG. If the data is pre-encoded with H.265 / HEVC, the server 300 may combine them. Then, the server 300 rewrites the header portion so as to include the total number of tiles obtained by adding the number of tiles as the number of tiles in the image, so that a bit stream including the main image and the sub information is constructed.

  On the other hand, for example, the main image is H.264. When the sub information is text data, JPEG data, or the like, the image including the main image and the sub information is entirely H.264. It may be re-encoded to comply with H.265 / HEVC.

  Both main image and sub information The server 300 may re-encode the image including the main image and the sub information so as not to refer to the outside of the screen even if the data is encoded in advance with H.265 / HEVC. For example, if the tile located at the end of the main image includes a block that refers to the outside of the screen (outside the picture), the tile of the sub information may be referred to simply by rewriting the header portion. Therefore, the server 300 may re-encode an image including the main image and the sub information so as not to refer to the outside of the screen.

  At that time, when the sub information is large data including a moving image and a small tile that makes it difficult to view the moving image is prepared for the sub information, the selected sub information and the prepared area are In some cases, it is not suitable. In this case, the server 300 may convert the sub information. Specifically, the server 300 may extract the text data from the moving image in the sub information, replace the sub information with the text data, and encode the main image and the sub information.

  FIG. 17 is a block diagram illustrating a configuration example of the processing unit 310 of the server 300 illustrated in FIG. The above encoding process is performed by the processing unit 310 having a configuration as shown in FIG. The processing unit 310 illustrated in FIG. 17 includes a subtraction unit 301, a transform quantization unit 302, an entropy coding unit 303, an inverse quantization inverse transform unit 304, an addition unit 305, a deblocking filter unit 306, an intra prediction unit 307, A motion compensation unit 308, a motion detection unit 309, and a memory 311 are provided.

  The subtraction unit 301 subtracts the value indicated by the prediction signal from the value indicated by the input signal, and outputs a prediction error. The transform quantization unit 302 performs frequency conversion and quantization on the prediction error, and outputs a quantization coefficient. The entropy encoding unit 303 performs entropy encoding on the quantized coefficient and outputs an encoded signal. The inverse quantization inverse transform unit 304 performs inverse quantization and inverse frequency transform on the quantization coefficient. Adder 305 adds the prediction error obtained by inverse quantization and inverse frequency conversion and the value indicated by the prediction signal, and outputs a pixel value.

  The deblocking filter unit 306 removes block distortion of an image composed of pixel values. The intra prediction unit 307 performs intra prediction with reference to the pixel value in the encoding target picture. The motion compensation unit 308 performs inter prediction with reference to pixel values in a picture different from the encoding target picture. The motion detection unit 309 performs motion detection based on the input signal and the picture stored in the memory 311 and outputs motion data. The memory 311 stores the reconstructed picture by adding the prediction error and the value indicated by the prediction signal.

  Here, as the memory 311, the storage unit 320 included in the server 300 may be used, or a memory unique to the processing unit 310 may be used.

  In addition, the decoding apparatus 200 is H.264. When the H.265 / HEVC is not supported, the server 300 uses a transmission method in which each of the main image and the sub information is encoded by a corresponding encoding method, and each packet is transmitted in a separate stream. It doesn't matter. In this case, the decoding apparatus 200 may display the main image and the sub information so that the main image corresponds to the entire screen and the sub information is superimposed on a partial area of the main image. Further, in this transmission method, data loss may be prevented by frequently transmitting packets including sub-information.

  H. When H.265 / HEVC is used, the priority may be set in advance so that information that is considered to have higher priority, for example, sub-information such as typhoon information rather than advertisement, is frequently embedded with priority. . Furthermore, a request for excluding the display of specific information from the decoding apparatus 200 may be received, and the selection operation may be controlled so that the information is not selected in subsequent images.

(Embodiment 2)
The second embodiment shows another example of the decoding method, the display method, and the management method in the image processing system 100 having the same configuration as that of the first embodiment. Also in the second embodiment, the bitstream includes information for designating motion prediction or motion compensation restrictions.

  In the second embodiment, the decoding device 200 displays one or more modes on the display unit 250 as one or more options based on the user's operation or automatically when the user starts viewing the main image. To do. The mode corresponds to a screen (picture) division format. That is, the decoding device 200 displays on the display unit 250 how the screen is divided into a plurality of tiles for each of one or more modes.

  18 to 20 show display examples in each mode. In FIGS. 18 to 20, the area displayed by shading is an area where the main image is mainly displayed. The user selects one mode from one or more modes. The decoding device 200 receives from the server 300 a bitstream including the main image and the sub information in the division format corresponding to the selected mode.

  In the second embodiment, in the decoding device 200 which is a terminal where the user views the main image, the display area is divided into a plurality of tiles according to the user's preference. The main image can be viewed with a desired tile. For example, the decoding device 200 superimposes the sub information on a part of the main image so that another application such as e-mail or a game comes to the front. The decryption apparatus 200 may execute a plurality of applications simultaneously.

  In addition, by selecting the mode at the start of viewing, occurrence of a change request for the display position of the sub information is suppressed during viewing. Therefore, the server 300 can transmit the originally held stream to the decoding device 200 based on one or more modes without changing the encoding format.

  The decoding device 200 displays each mode so that the user can easily confirm which mode is suitable for viewing. For example, for each mode, the decoding apparatus 200 represents the tile boundary with a broken line or the like, represents the boundary between the main image area and the sub information area with a thick line, or changed the color between the main image area and the sub information area. You may change the ring.

  In the above, the user selects a mode. However, the server 300 may determine an initial mode corresponding to tile division and assignment based on the content of the main image or the environment such as the degree of congestion of the distribution network. Then, the decoding device 200 may display the main image and the sub information in the initial mode.

  Thereby, the server 300 can allow the user to view the main image and the sub-information with more suitable division and assignment according to the content of the main image. In addition, inappropriate deletion or modification of sub-information to be viewed such as emergency disaster information is suppressed. Furthermore, the decoding device 200 may display the latest MPD and accept a mode reselection from the user when the MPD is changed.

(Embodiment 3)
The third embodiment shows another example of the decoding method, the display method, and the management method in the image processing system 100 having the same configuration as that of the first embodiment. Also in the third embodiment, the bitstream includes information for designating motion prediction or motion compensation restrictions.

  As described in the first and second embodiments, when the display method of the main image and the sub information is changed by the user's operation, the decoding device 200 or the server 300 holds the user's operation or viewing mode information. You may keep it. Then, the decoding device 200 may display an image by receiving and decoding a bitstream for displaying more appropriate sub-information based on the held information.

  For example, although the user performs an operation to delete or reduce other advertisements, the user performs an operation to enlarge a car advertisement in reverse, or selects a tile on which a car advertisement is displayed and displays it in an MPD file or the like. Or you may access the URL. In this case, the decoding device 200 may request the server 300 to generate a bitstream that displays many car advertisements. Alternatively, the server 300 may select such a bitstream from the beginning.

(Embodiment 4)
The fourth embodiment shows another example of an image processing system, an image decoding method, and a bitstream generation method that decode an image from a bitstream that is encoded by dividing an image into a plurality of tiles.

  In particular, in Embodiment 4, the bitstream includes information for enabling parallel processing of encoding or decoding. For example, the image is divided into a plurality of areas each called a tile. Information that can specify motion prediction or motion compensation restriction that a bitstream refers only to pixels of one or more corresponding tiles in image signals at the same or different times during motion prediction processing or motion compensation processing. including.

[Description of configuration]
FIG. 21 shows a configuration example of the broadcast system 110 according to the fourth embodiment. Each of television 400, set-top box (STB) 460, reader / recorder 470, playback device 600, mobile phone (smart phone) 700, and car navigation system 800 corresponds to decoding device 200 of the first embodiment. The monitor 450 and the monitor 650 correspond to the display unit 250 of the first embodiment. The recording medium 430, the recording medium 431, and the recording medium 432 correspond to the storage unit 220 of the first embodiment.

  Broadcast station 500 corresponds to server 300 in the first embodiment. One of television 400, STB 460, reader / recorder 470, playback device 600, mobile phone (smart phone) 700, and car navigation system 800 may be included in broadcast system 110 as a device that receives broadcasts. That is, all of these may not be included in the broadcast system 110 as a device that receives the broadcast.

  The broadcast station 500 transmits multiplexed data obtained by multiplexing music data encoded with encoded video data to the satellite 501 by radio waves. The satellite 501 receiving the radio wave from the broadcasting station 500 transmits the radio wave for broadcasting, and the home antenna 503 capable of receiving the satellite broadcasting receives the radio wave for broadcasting. Multiplexed data may be transmitted via cable 502. A device such as the television (receiver) 400 or the STB 460 decodes the received multiplexed data, displays an image on the monitor 450, and outputs sound from a speaker.

  The antenna 802 of the car 801 may receive radio waves. In this case, the car navigation system 800 displays an image similarly to the television 400.

  A reader / recorder 470 having processing units 210 and 310 (decoding processing unit and encoding processing unit) as shown in FIGS. 4 and 17 is received by recording media 431 and 432 such as HDD, DVD, BD, or SD. The multiplexed data thus recorded is recorded and reproduced. In this case, the decoded video and audio are output via the monitor 450 or the speaker. Also, other devices or other systems can reproduce video and audio via recording media 431 and 432 on which multiplexed data is recorded.

  4 is mounted in the STB 460 connected to the cable 502 for cable television or the antenna 503 for satellite / terrestrial broadcasting, and an image is displayed on the monitor 450 of the television 400 via the STB 460. May be displayed. In addition, the processing unit 210 as illustrated in FIG.

  FIG. 22 is a block diagram illustrating a configuration example of the television 400. The tuner 401, the modulation / demodulation unit 402, and the multiplexing / demultiplexing unit 403 receive the multiplexed stream, demodulate it, and separate it into audio data and video data. Audio signal and video data are decoded by a signal processing unit 420 including an audio signal processing unit 404 and a video signal processing unit 405, and the audio data and video data are output from the speaker 480 and the display unit 450 in the output unit 409. .

  The television 400 includes a power supply circuit unit 411 that supplies power to each unit of the television 400. In addition, the television 400 includes an LSI 900, and some components are included in the LSI 900.

  Furthermore, the television 400 includes an interface unit 417 for the external remote controller 490 and the like to control the television 400 and the television 400 to control external peripheral devices. The interface unit 417 includes an operation input unit 412, a bridge 413, a slot unit 414, a driver 415, and a modem 416. A plurality of other constituent elements also perform operations for reception, decoding, and display in cooperation based on control from the control unit 410.

  Control unit 410, signal processing unit 420, and display unit (monitor) 450 correspond to control unit 240, processing unit 210, and display unit 250 of the first embodiment, respectively. The recording medium 431, the recording medium 432, the buffer 433, the buffer 434, the buffer 435, and the buffer 436 correspond to the storage unit 220 of the first embodiment.

[Description of operation]
FIG. 23 shows an example of a processing flow of the broadcasting system 110 according to the fourth embodiment. First, the broadcast station 500 broadcasts a plurality of image streams corresponding to a plurality of contents without receiving a request from the television 400 (S1301). Each of the plurality of image streams is encoded by using, for example, one main image corresponding to one channel and the sub-information described in Embodiment 1 using a plurality of tiles that divide the screen (picture). Stream.

  In Embodiment 1, the sub information is selected from a plurality of candidates based on the additional information from decoding apparatus 200. In Embodiment 4, the sub information is selected in advance from a plurality of candidates.

  Here, the sub information is encoded as a tile. The sub-information mainly includes information with high priority as shown in the first embodiment. The sub information may include information in which low priority information is summarized as a headline. The detailed content of the sub-information may be transmitted as an additional separate stream in association with the tile.

  The television 400 confirms an image stream corresponding to the viewing target content (S1302). Then, the television 400 determines whether there are a plurality of image streams corresponding to the viewing target content (S1303). For example, the television 400 determines whether or not there are a plurality of candidates corresponding to a plurality of different encoding standards or a plurality of encoding levels with respect to the main image. Also, the television 400 determines whether there are a plurality of candidates for the sub information encoded as tiles.

  When there are a plurality of candidates for the main image (Yes in S1303), television 400 selects a candidate that can be received and decoded with high image quality and high efficiency as the main image. When there are a plurality of candidates for the sub information, the television 400 selects the sub information from the plurality of candidates based on the position information, user information, or the like held by the television 400. For example, the selection of sub information is performed in the same manner as in the example shown in the first embodiment. Accordingly, the television 400 selects an image stream based on the unique information (S1304).

  If it is determined that there are not a plurality of image streams (No in S1303), the television 400 selects a receivable image stream (S1305). Then, the television 400 decodes the image stream selected in the selection process (S1304, S1305), and displays the decoded image stream on the monitor 450 (S1306).

[effect]
In this way, the broadcast station 500 divides the screen (picture) into a plurality of tiles, and encodes the selected main image and the selected sub information as separate tiles.

  Thus, the television 400 can receive an image accompanied with appropriate information in the form of one stream. Accordingly, the broadcast station 500 can transmit the sub information so that the sub information is not altered into an unintended format by the apparatus or system that receives or relays the sub information. Therefore, the television 400 can receive appropriate sub-information.

  As described above, television 400 in Embodiment 4 can receive an image accompanied with appropriate information.

[Supplement]
The selection method, encoding method, decoding method, transmission method and display method shown in the first to third embodiments can be similarly applied to the fourth embodiment.

  For example, the image quality of the main image may be changed, the viewing channel may be switched, a CM may be inserted, or the user may directly instruct the change of sub information. When the MPEG2 system is used, the change of tiles constituting the content may be notified using program information or event information. Then, the television 400 may accept reselection of tiles (sub-information and the like) with the change as a trigger.

  One content is broadcast in association with one channel. However, if the image size is large, such as 8k4k, one content may be broadcast on a plurality of channels. In addition, there is a possibility that broadcasting is used for transmission / reception of part of an image and communication is used for transmission / reception of part of an image. That is, there is a possibility that a plurality of transmission paths are used for image transmission / reception.

  Note that there may be one candidate (one main image) in one channel with respect to the main image, and one channel has different encoding standards or levels with respect to the main image. There may be multiple candidates.

(Embodiment 5)
The fifth embodiment shows another example of an image processing system that decodes an image from a bitstream that is encoded by dividing an image into a plurality of tiles, an image decoding method, and a bitstream generation method.

  In particular, also in Embodiment 5, the bitstream includes information for enabling parallel processing of encoding or decoding. For example, the image is divided into a plurality of areas each called a tile. Information that can specify motion prediction or motion compensation restriction that a bitstream refers only to pixels of one or more corresponding tiles in image signals at the same or different times during motion prediction processing or motion compensation processing. including.

[Overview]
When the HEVC method is used, large and high-definition image data such as 8k4k can be transmitted and received. However, a mobile terminal that receives image data may not be suitable for displaying a large image. In addition, the user may be interested in only a part of a large image and may want to view a part of the image in detail. Furthermore, such a region may change over time.

  In addition, there is image data that is less harmful even when only a part of the image is viewed. On the other hand, there are some image data in which it is difficult to grasp the contents of an image unless the entire image is viewed.

  In the fifth embodiment, a partial range of an image is selected, and appropriate decoding and appropriate display are performed. More specifically, there is content in which image data suitable for clipping and viewing a part of an image, such as a soccer game, and image data for viewing the entire image, such as CM, are mixed. When received and viewed, a display without a sense of incongruity is performed.

[Description of configuration]
Since the configuration according to the fifth embodiment is the same as that of the first embodiment (FIGS. 3 and 4), description thereof is omitted.

[Description of operation]
24 and 25 are diagrams showing a processing flow of the operation of the image processing system 100 according to the fifth embodiment.

  The decoding device 200 transmits a request for an image to be viewed to the server 300 (S1401). More specifically, the decoding device 200 requests the server 300 to transmit an image stream (image sequence) composed of a plurality of images to be viewed. The server 300 receives an image request (S1402).

  Further, the server 300 determines whether or not the selection range (selection position) that is the range of the area to be reproduced in the image has been saved (S1403). When the selection range is not stored (No in S1403), the server 300 requests the decoding device 200 to transmit the selection range (S1403). The decryption apparatus 200 receives the request for the selection range (Yes in S1405), and transmits the selection range to the server 300 (S1406). The server 300 receives the selection range and saves the received selection range (S1407).

  Next, the server 300 determines whether or not the image to be transmitted in the image stream is the entire display target image (S1408). The entire display target image is an image that should be displayed as a whole. When the image is the entire display target image (Yes in S1408), the server 300 selects the entire image regardless of the selection range specified by the decoding device 200 (S1409).

  On the other hand, when the image to be transmitted is not the entire display target image (No in S1408), the server 300 acquires the selection range saved in the saving process (S1407) or the previously saved selection range (S1410). Then, the server 300 selects a part of the image according to the acquired selection range (S1411).

  Then, the server 300 transmits a bit stream including all or part of the selected image to the decoding device 200 (S1412). The decoding device 200 receives the bit stream transmitted from the server 300 (S1413).

  Here, the bit stream to be transmitted and received is generated by encoding an image using the MV restricted tile, as in the first embodiment. As shown in FIG. 2, the processing target block in the MV restricted tile refers to the reference area within the tile for the temporally different images (that is, for other temporally different pictures). .

  Therefore, for example, when the selection range is the second tile in FIG. 2, the entire bit stream corresponding to the entire image may not be transmitted / received. In this case, only the part corresponding to the second tile in the bit stream may be transmitted and received. The decoding device 200 can decode and display an image in the second tile based on a portion corresponding to the second tile in the bitstream.

  Next, the decoding device 200 decodes an image from the bit stream. Then, the decoding device 200 resizes (enlarges or reduces) the decoded image so as to fit the display size of the decoding device 200 and displays it (S1414).

  When receiving an instruction to stop playback from the user (Yes in S1415), the decoding apparatus 200 transmits a playback stop request to the server 300 (S1416). In other cases (No in S1415), the decoding apparatus 200 repeats reception, decoding, and display (S1413 and S1414) until display of all the images in the image stream is completed (S1418).

  The server 300 repeats selection and transmission (S1408 to S1412) until a reproduction stop request is received from the decoding device 200 or until transmission of all images in the image stream is completed (S1417).

[effect]
With the above operation, in the case of the entire display target image, the reproduction of the image is automatically switched to the reproduction of the entire image regardless of the selection range. This improves the convenience for the user.

  More specifically, when selecting all or part of an image, the server 300 determines whether the image is an entire display target image. When the image is the entire display target image, the server 300 selects the entire image. When the image is not the entire display target image, the server 300 selects a part of the image according to the selection range designated and stored by the user or the selection range designated in the past reproduction. Thereby, the selection is automatically switched. Hereinafter, a more specific example will be described with reference to FIGS.

  FIG. 26 is a diagram illustrating an example of the timing at which the entire display and the partial display are switched. Specifically, FIG. 26 shows an example of an image stream selected by the user. In this example, the image stream has three CMs (advertisement parts) between four scenes (main part). The images included in the first main part to the fourth main part are not entire display target images, and the images included in the first advertisement part to the third advertising part are entire display target images.

  FIG. 27 is a diagram illustrating an example of partial display. Specifically, FIG. 27 shows an example of a part of the image in the first main part of the image stream designated by the user, and the range of the six tiles at the upper left of the entire image as shown in FIG. 12 is selected. An example is shown.

  FIG. 28 is a diagram illustrating an example of the entire display. Specifically, FIG. 28 shows an example of an image of the first advertisement part. The image of the first advertisement part may be divided into a plurality of tiles or may not be divided. Hereinafter, the operation of the image processing system 100 will be described with reference to FIGS.

  First, based on a user operation, the decoding device 200 selects the image stream shown in FIG. 26 and requests the server 300 for the selected image stream. In addition, the decoding device 200 receives a selection range from the user and transmits the selection range to the server 300 so that the part shown in FIG.

  The server 300 transmits a bit stream indicating a part of the image as shown in FIG. 27 for the first main part of FIG. In addition, the server 300 transmits a bit stream indicating the entire image as illustrated in FIG. 28 when the processing target image is switched to the image of the first advertisement portion. The decoding device 200 displays an image obtained by decoding the bitstream. The image shown in FIG. 27 or 28 may not match the size of the display unit 250 of the decoding device 200. In that case, the decoding device 200 displays the image in an enlarged or reduced manner.

  The contents of the CM (advertisement part) are often grasped by looking at the entire CM image. Therefore, the entire image is displayed in the CM. Conversely, in other scenes (main part), only a part of the image is displayed according to the display size of the decoding device 200 or the user's request. Thereby, the reproduction of a part of the image and the reproduction of the whole image are automatically switched without an instruction from the user. This improves the convenience for the user.

  When the image is not the entire display target image, only a bit stream indicating a part of the image is transmitted / received. Therefore, there is a possibility that a block in an image that is not the entire display target image is not decoded. Further, the selection range of a part of the image changes according to an instruction from the decoding device 200 to the server 300 or a selection range that has already been saved. Therefore, the block in the image that is the entire display target image is encoded without referring to the block in the image that is not the entire display target image.

  For example, when the image in FIG. 12 is not the entire display target image, the decoding location (decoding range) changes according to an instruction from the decoding device 200 or a stored selection range. Therefore, depending on the situation, only the first tile may be decoded, and only the second tile may be decoded. Thereafter, when the entire display target image as shown in FIG. 28 is decoded, the decoded portion of the image shown in FIG. 12 differs depending on the situation. References to images are prohibited.

  In addition, the selection range in the image initially designated by the user may be stored in the server 300. Then, the saved selection range may be called and used when the advertisement part is switched to the main part. Thereby, the selection range does not have to be designated again by the user every time the reproduction of the advertisement part is finished.

  Specifically, in the example of FIG. 26, the server 300 reads the selection range designated by the user first when the image is switched from the first advertisement portion to the second main portion, and the server 300 reads the first selection range according to the selection range. The bit stream of the two main parts is transmitted. Similarly, the server 300 transmits the entire image in the second advertisement part, transmits a part of the image in the third part, transmits the entire image in the third part, and transmits the entire part in the third part. Send part of the image.

  That is, the server 300 acquires a selection range that is first designated by the user. Thereafter, the server 300 automatically switches between reproduction of a part of the image and reproduction of the entire image without newly acquiring a selection range. This improves the convenience for the user. In addition, the processing amount for transmitting / receiving information of the selected range from the decoding device 200 to the server 300 is reduced.

  Also, according to the user's operation, when the image reproduction is temporarily stopped and after a while, the image reproduction is resumed, or when the reproduction of the same image is started with another decoding device, Re-selection of the selection range is omitted. Hereinafter, these operations will be specifically described with reference to FIG.

  When receiving the image request, the server 300 determines whether or not the selection range is stored (S1403). When the selection range is stored (Yes in S1403), the server 300 does not request the selection range from the decoding device 200. That is, if another decoding device has specified a selection range and has reproduced the same image in the past, the server 300 does not request the selection range because the selection range is stored.

  For example, with respect to the first main part of the image stream of FIG. 26, the reproduction of a part of the image as shown in FIG. 27 is started on the mobile phone by the user's operation, and then the first main part by the user's operation. Playback is temporarily stopped in the middle of.

  Next, when the reproduction is resumed on the personal computer by the user's operation, the server 300 calls the saved selection range and transmits a bit stream indicating a part of the image to the personal computer according to the saved selection range. Then, on the personal computer, the reproduction of a part of the image as shown in FIG. 27 is started without receiving an instruction of the selection range from the user.

  Thereby, the repetition of designation | designated of the selection range is suppressed. Therefore, convenience for the user is improved. In addition, the processing amount for transmitting / receiving the information of the selected range from the decoding device 200 to the server 300 is reduced.

[Supplement]
Note that the server 300 determines whether the image is the entire display target image (S1408 in FIG. 25). Whether the image is the entire display target image may be explicitly described in SEI or PPS in the bitstream.

  For example, for each image, a flag indicating whether or not the image is the entire display target image may be included in the bitstream. This flag may be a flag indicating whether or not display of the entire image is recommended, or may be a flag indicating whether or not display of a part of the image is prohibited. Specifically, when the flag indicates that the display of the entire image is recommended or the display of a part of the image is prohibited, the server 300 determines that the image is the entire display target image. . Otherwise, the server 300 determines that the image is not the entire display target image.

  Further, for example, in order to reduce the amount of code or the amount of processing for explicitly describing whether or not the image is the entire display target image, the server 300 extracts the features of the image, and based on the extracted features Thus, it may be determined whether or not the image is the entire display target image.

  For the same purpose, when the image is not divided into a plurality of tiles, when the number of tiles is small, or when the division format is different from the previous scene, the server 300 displays the image as a whole display target image. It may be determined that The contents of the image that is the entire display target image and the image that is not the entire display target image are greatly different. Therefore, the tile division format often changes when these images are switched. Therefore, such a change in the division format may be used for determining whether or not the image is the entire display target image.

  In the example of FIG. 26, it is determined that the CM image is the entire display target image. The determination of whether or not the image is the entire display target image is not limited to such determination. For example, an emergency news image, a drama image, a low-resolution image, or the like may be determined as the entire display target image. Further, for example, a sport image, a concert image, or the like may be determined not to be an entire display target image.

  Whether or not the image is a whole display target image may be freely set by an image creator or a provider. By setting the entire image to be shown to the user as the entire display target image, the image is displayed in accordance with the intention of the creator or provider of the image.

  In the above, the selection range is specified at the start of reproduction. However, the selection range may be changed during playback in accordance with a user operation or a display device. In that case, the newly designated selection range is saved again.

  Further, in the above, when the selection range is not stored, the server 300 requests the selection range from the decoding device 200 (S1403, S1404). However, when the selection range is not stored, first, reproduction may be started using the entire range of the image as the selection range. Thereafter, the selection range may be changed according to an instruction of the selection range from the user. Also, playback may be started using a default selection range.

  Thereby, processing at the start of reproduction is reduced. In addition, when the display of the entire image or the default selection range is in accordance with the user's request, the designation of a new selection range can be omitted.

  The server 300 may store an image selection range for each user. For example, the server 300 stores a selection range for each user ID. When the same image is reproduced by the same user, the server 300 may call a selection range corresponding to the user. Thereby, even when the server 300 is accessed by a plurality of users and an image is reproduced, the reproduction of the image is started in the selection range designated in the past by each user. Therefore, convenience for the user is improved.

  The selection range designated in the past may be a range defined by absolute coordinates with respect to the image, or may be a range defined by other criteria. For example, when one idol is selected in a live video of an idol group, an area where the idol is displayed for each image may be a selection range.

  Moreover, the selection range may be shared between users. Playback may be started using a selection range specified by another user. That is, even if the selection range is not specified by the user himself, the selection range specified by another user may be used. Thereby, it is possible to use an appropriate selection range selected by another user without trouble.

  For example, the image processing system 100 may specify a user's hobby or career based on information such as an SNS account, and recommend a selection range specified by another user who has a similar hobby or career. .

  In the above description, the selection range is stored in the server 300. However, the selection range may be stored in a device different from the server 300 or may be stored in the decoding device 200.

  For example, when the selection range is stored in the decoding device 200, the decoding device 200 calls the stored selection range and transmits the selection range to the server 300 when starting playback according to the user's operation. Also good. When the selection range is stored in another device, the decoding device 200 may read the selection range from another device and transmit the selection range to the server 300. Alternatively, the server 300 may read the selection range from another device when receiving the request.

  Here, the reproduction of the image is shown, but when a part of the image is displayed, the decoding apparatus 200 may decode only the sound related to the part depending on the content of the image.

  For example, when an individual microphone is assigned to each person in an image in which many persons are captured, and the selection range is a specific person, the decoding device 200 decodes only a signal from the microphone corresponding to the specific person. May be. In this case, when the image is switched to the entire display target image, the decoding target sound is also switched at the same time so as to decode the entire sound associated with the entire image.

[Conclusion]
A characteristic configuration and a characteristic operation of the image processing system shown in the fifth embodiment will be described below.

  FIG. 29 is a block diagram illustrating a characteristic configuration of the image processing system according to the fifth embodiment as an example. An image processing system 1000 illustrated in FIG. 29 includes an image decoding device 1100 and an image transmission device 1200. The image decoding device 1100 and the image transmission device 1200 correspond to the decoding device 200 and the server 300 in FIG.

  The image decoding device 1100 includes a selection unit 1101, a transmission unit 1102, a request unit 1103, a reception unit 1104, a decoding unit 1105, a display control unit 1106, a display device 1107, and a storage unit 1110. In particular, the selection unit 1101, the display control unit 1106, the display device 1107, the storage unit 1110, and the like may not be included in the image decoding device 1100. For example, these may be included in another device or may not be used in the image processing system 1000.

  The selection unit 1101, the request unit 1103, and the decryption unit 1105 mainly correspond to the processing unit 210 in FIG. The transmission unit 1102 and the reception unit 1104 mainly correspond to the communication unit 230 in FIG. The display control unit 1106 mainly corresponds to the control unit 240 in FIG. The display device 1107 mainly corresponds to the display unit 250 in FIG. The storage unit 1110 mainly corresponds to the storage unit 220 in FIG.

  The image transmission apparatus 1200 includes a reception unit 1201, a request reception unit 1202, a transmission unit 1203, and a storage unit 1210. In particular, the storage unit 1210 may not be included in the image transmission device 1200. For example, the storage unit 1210 may be included in another device or may not be used in the image processing system 1000.

  The reception unit 1201 and the transmission unit 1203 mainly correspond to the communication unit 330 in FIG. The request reception unit 1202 mainly corresponds to the processing unit 310 in FIG. The storage unit 1210 mainly corresponds to the storage unit 320 in FIG.

  FIG. 30 is a diagram illustrating a processing flow of characteristic operations of the image processing system 1000 illustrated in FIG. 29 as an example. The image processing system 1000 processes the encoded image according to the operation shown in FIG.

  The image decoding apparatus 1100 included in the image processing system 1000 receives the encoded image from the image transmission apparatus 1200 and decodes the image received from the image transmission apparatus 1200 according to the operation shown in FIG. To do. In addition, the image transmission device 1200 included in the image processing system 1000 transmits the encoded image to the image decoding device 1100 according to the operation illustrated in FIG.

  Specifically, first, in the image decoding apparatus 1100, the selection unit 1101 selects a partial range of the image out of the entire range of the image in accordance with a user operation (S1101). For example, the selection unit 1101 receives user input, and selects a partial range of the image according to the received input. This operation may be omitted if a partial range of the image has already been selected.

  The selection unit 1101 may notify each unit of the image decoding device 1100 of a partial range of the image via the storage unit 1110. The storage unit 1110 stores range information indicating a partial range of the image. The storage unit 1110 may store the range information in a storage unit inside the image decoding device 1100, or may store the range information in a storage unit outside the image decoding device 1100. For example, the storage unit 1110 may store the range information in the image transmission device 1200 via the transmission unit 1102.

  Next, the request unit 1103 requests the image transmission apparatus 1200 to transmit an image (S1102). For example, the request unit 1103 transmits a signal for requesting image transmission to the image transmission device 1200 via the transmission unit 1102.

  Next, in the image transmission device 1200, the request reception unit 1202 receives a request for image transmission from the image decoding device 1100 (S1201). For example, the request reception unit 1202 receives a signal for requesting transmission of an image from the image decoding device 1100 via the reception unit 1201.

  Next, in the image decoding device 1100, the transmission unit 1102 transmits range information indicating a partial range of the image to the image transmission device 1200 (S1103). More specifically, the range indicated by the range information is a range selected by the selection unit 1101 according to a user operation.

  Next, in the image transmission device 1200, the reception unit 1201 receives range information from the image decoding device 1100 (S1202). The storage unit 1210 may store the range information received by the reception unit 1201. In addition, the storage unit 1210 may store the range information in a storage unit inside the image transmission apparatus 1200 or may store the range information in a storage unit outside the image transmission apparatus 1200.

  Next, in the image transmission device 1200, the transmission unit 1203 transmits the entire image or a part of the image to the image decoding device 1100 in a state where a partial range of the image is selected (S1203).

  Specifically, when the image belongs to the first type, the transmission unit 1203 transmits the entire image to the image decoding device 1100. The transmission unit 1203 transmits a part of the image to the image decoding device 1100 when the image belongs to the second type. More specifically, when the image belongs to the second type, the transmission unit 1203 is one or more tiles corresponding to a partial range of the image among a plurality of tiles included in the image, and the inter-surface prediction is performed. Only one or more tiles for which reference to the outside of the range corresponding to each tile is prohibited are transmitted.

  For example, the first type is a type for which the whole display is recommended, and the second type is a type for which the whole display is not recommended. Also, for example, the first type is a commercial and the second type is different from a commercial. For example, a flag indicating whether the image belongs to the first type or the second type may be given to the image. Then, the transmission unit 1203 may transmit all or part of the image according to the flag.

  One or more tiles for which reference to the outside of the range corresponding to each tile in the inter-plane prediction is prohibited corresponds to an MV restricted tile. The range corresponding to each tile is a range determined for each tile, and is basically the range of each tile.

  Next, in the image decoding apparatus 1100, the reception unit 1104 receives the whole or a part of the image from the image transmission apparatus 1200 in a state where a partial range of the image is selected (S1104).

  Specifically, the reception unit 1104 receives the entire image from the image transmission device 1200 when the image belongs to the first type. In addition, when the image belongs to the second type, the reception unit 1104 receives a part of the image from the image transmission device 1200. More specifically, when the image belongs to the second type, the reception unit 1104 is one or more tiles corresponding to a partial range of the image among a plurality of tiles included in the image, and inter-surface prediction is performed. Only one or more tiles for which reference to the outside of the range corresponding to each tile is prohibited are received.

  For example, a flag indicating whether the image belongs to the first type or the second type is given to the image, and the receiving unit 1104 may receive all or part of the image according to the flag.

  Next, the decoding unit 1105 decodes all or part of the image received from the image transmission device 1200 by the receiving unit 1104 (S1105).

  Next, the display control unit 1106 displays all or part of the image decoded by the decoding unit 1105 on the display device 1107 (S1106). The whole or part of the decoded image may be stored in the storage unit 1110 as data. In this case, the display may be omitted.

  Accordingly, the image transmission device 1200 can switch between transmission of the entire image and transmission of a part of the image according to the type of image. Further, the image decoding apparatus 1100 can switch between decoding of the entire image and decoding of a part of the image according to the type of the image. Therefore, the image transmission device 1200 and the image decoding device 1100 can efficiently process the encoded image.

  For example, as described above, the range information may be stored in the image decoding device 1100, the image transmission device 1200, or the like. Then, the receiving unit 1104 of the image decoding device 1100 may receive a part of the image according to the stored range information. Then, the decoding unit 1105 of the image decoding device 1100 may decode a part of the received image according to the stored range information. Then, the display control unit 1106 of the image decoding device 1100 may display a part of the image received and decoded according to the stored range information.

  Further, for example, the transmission unit 1102 of the image decoding device 1100 transmits the range information to the image transmission device 1200 when the range information is not stored, and the range information is transmitted to the image transmission device 1200 when the range information is stored. You do not have to send to. The receiving unit 1104 of the image decoding device 1100 stores the range information, and when receiving a part of the image, the range information is stored without being transmitted to the image transmitting device 1200 by the transmitting unit 1102. A part of the image may be received according to the range information.

  Further, for example, when displaying a part of an image after displaying the whole of another image, the display control unit 1106 may display a part of the image received and decoded according to the stored range information. . In addition, for example, when the display control unit 1106 resumes displaying a part of the image after stopping displaying the part of the image, the display control unit 1106 displays the part of the image received and decoded according to the stored range information. May be.

  For example, the display control unit 1106 may display a part of an image received and decoded according to the stored range information on another display device different from the display device 1107.

(Other variations)
The configuration according to the present invention is not limited to the above-described plurality of embodiments. The configuration according to the present invention may be in the form shown below.

  (1) Each of the above-described devices (each of an image decoding device and an image transmission device) is specifically a computer system including a microprocessor, ROM, RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. is there.

  A computer program is stored in the RAM or the hard disk unit. Each device achieves its functions by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions for the computer in order to achieve a predetermined function.

  (2) A part or all of the plurality of constituent elements constituting each of the above devices may be configured by one system LSI (Large Scale Integration). The system LSI is a super multifunctional LSI manufactured by integrating a plurality of components on one chip, and specifically, a computer system including a microprocessor, a ROM, a RAM, and the like. .

  A computer program is stored in the RAM. The system LSI achieves its functions by the microprocessor operating according to the computer program.

  (3) A part or all of a plurality of components constituting each of the above devices may be configured as an IC card that can be attached to and detached from each device or a single module. The IC card or module is a computer system that includes a microprocessor, ROM, RAM, and the like.

  The IC card or the module may include the super multifunctional LSI described above. The IC card or the module achieves its function by the microprocessor operating according to the computer program. The IC card or module may have tamper resistance.

  (4) The configuration according to the present invention may be the method described above. The configuration according to the present invention may be a computer program that implements this method by a computer, or a digital signal that is configured by a computer program.

  The configuration according to the present invention may be a computer-readable recording medium on which a computer program or a digital signal is recorded. The computer-readable recording medium is, for example, a flexible disk, a hard disk, a CD-ROM, an MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray (registered trademark) Disc), or a semiconductor memory. . The configuration according to the present invention may be a digital signal recorded on these recording media.

  The configuration according to the present invention may be a configuration in which a computer program or a digital signal is transmitted via an electric communication line, a wireless communication line, a wired communication line, a network represented by the Internet, a data broadcast, or the like.

  The configuration according to the present invention may be a computer system including a microprocessor and a memory. The memory may store the above computer program, and the microprocessor may operate according to the computer program.

  Further, the program or digital signal may be recorded on a recording medium and transferred, or transferred via a network or the like. The configuration according to the present invention may be implemented by another independent computer system.

  (5) In the above embodiment, an image decoding method, an image decoding device, an image transmission method, an image transmission device, and the like are mainly described. Of course, an image encoding method and an image encoding apparatus for generating a bit stream that enables these may be used. The image encoding device and the image transmission device may be the same device. Some devices or circuits that do some processing may be in different countries. As a whole, if a system according to the present invention is constructed, the system is included in the scope of the present invention.

  (6) The above embodiment and the above modification may be combined.

  In each of the above embodiments, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

  In other words, the image decoding device and the image transmitting device include a processing circuit and a storage device (Storage) that is electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit, and executes processing using a storage device. Further, when the processing circuit includes a program execution unit, the storage device stores a software program executed by the program execution unit.

  Here, the software that realizes the image decoding device of each of the above embodiments is a program as follows.

  That is, this program is an image decoding method for receiving an encoded image from an image transmission device and decoding the image received from the image transmission device, to the computer, wherein the image is transmitted to the image transmission device. A request step for requesting transmission of the image, and transmission for transmitting, to the image transmission device, range information indicating a range of a part of the image selected from the entire range of the image and selected according to a user operation. And in a state where a partial range of the image is selected, (i) if the image belongs to the first type, the entire image is received from the image transmitting device; and (ii) the image is In the case of belonging to the second type, among the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and a category corresponding to each tile in inter-surface prediction. A reception step of receiving a part of the image from the image transmission device by receiving only one or more tiles whose reference to outside is prohibited, and the image received from the image transmission device in the reception step And a decoding step for decoding all or part of the image decoding method.

  Further, this program is an image transmission method for transmitting an encoded image to an image decoding device to a computer, a request receiving step for receiving a request for transmission of the image from the image decoding device, and the entire image A receiving step of receiving from the image decoding device range information indicating a range of a part of the image selected in accordance with a user operation, and a range of the part of the image are selected. (I) when the image belongs to the first type, the entire image is transmitted to the image decoding device, and (ii) when the image belongs to the second type, a plurality of images included in the image Among these tiles, only one or more tiles corresponding to a part of the range of the image and for which reference to the outside of the range corresponding to each tile is prohibited in the inter prediction is prohibited. By sending it may be a program for executing the image transmission method comprising a transmission step of transmitting a part of the image to the image decoding apparatus.

  Each component may be a circuit as described above. These circuits may constitute one circuit as a whole, or may be separate circuits. Each component may be realized by a general-purpose processor or a dedicated processor.

  Moreover, another component may perform the process which a specific component performs. In addition, the order in which the processes are executed may be changed, or a plurality of processes may be executed in parallel. The image processing apparatus may include an image decoding apparatus and an image transmission apparatus.

  In addition, the image processing system, the image transmission device, the image transmission method, the image decoding device, and the image decoding method described above are respectively an image distribution system, an image distribution device, an image distribution method, an image reception device, and It may be expressed as an image receiving method.

  As described above, the image decoding device and the image transmission device according to one or a plurality of modes have been described based on the embodiment, but the present invention is not limited to this embodiment. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in this embodiment, and forms constructed by combining components in different embodiments are also within the scope of one or more aspects. May be included.

(Embodiment 6)
By recording a program for realizing the configuration of the moving image encoding method (image encoding method) or the moving image decoding method (image decoding method) shown in each of the above embodiments on a storage medium, each of the above embodiments It is possible to easily execute the processing shown in the form in the independent computer system. The storage medium may be any medium that can record a program, such as a magnetic disk, an optical disk, a magneto-optical disk, an IC card, and a semiconductor memory.

  Furthermore, application examples of the moving picture coding method (picture coding method) and the moving picture decoding method (picture decoding method) shown in the above embodiments and a system using the same will be described. The system has an image encoding / decoding device including an image encoding device using an image encoding method and an image decoding device using an image decoding method. Other configurations in the system can be appropriately changed according to circumstances.

  FIG. 31 is a diagram showing an overall configuration of a content supply system ex100 that implements a content distribution service. A communication service providing area is divided into desired sizes, and base stations ex106, ex107, ex108, ex109, and ex110, which are fixed wireless stations, are installed in each cell.

  The content supply system ex100 includes a computer ex111, a PDA (Personal Digital Assistant) ex112, a camera ex113, a mobile phone ex114, and a game machine ex115 via the Internet ex101, the Internet service provider ex102, the telephone network ex104, and the base stations ex106 to ex110. Etc. are connected.

  However, the content supply system ex100 is not limited to the configuration as shown in FIG. 31, and may be connected by combining any of the elements. In addition, each device may be directly connected to the telephone network ex104 without going from the base station ex106, which is a fixed wireless station, to ex110. In addition, the devices may be directly connected to each other via short-range wireless or the like.

  The camera ex113 is a device capable of shooting a moving image such as a digital video camera, and the camera ex116 is a device capable of shooting a still image and moving image such as a digital camera. The mobile phone ex114 is a GSM (registered trademark) (Global System for Mobile Communications) system, a CDMA (Code Division Multiple Access) system, a W-CDMA (Wideband-Code Division Multiple Access) system, or LTE (Long Term Evolution). It may be a system, a HSPA (High Speed Packet Access) mobile phone, a PHS (Personal Handyphone System), or the like.

  In the content supply system ex100, the camera ex113 and the like are connected to the streaming server ex103 through the base station ex109 and the telephone network ex104, thereby enabling live distribution and the like. In live distribution, content that is shot by a user using the camera ex113 (for example, music live video) is encoded as described in each of the above embodiments (that is, in one aspect of the present invention). Functions as an image encoding device), and transmits it to the streaming server ex103. On the other hand, the streaming server ex103 stream-distributes the content data transmitted to the requested client. Examples of the client include a computer ex111, a PDA ex112, a camera ex113, a mobile phone ex114, and a game machine ex115 that can decode the encoded data. Each device that receives the distributed data decodes the received data and reproduces it (that is, functions as an image decoding device according to one embodiment of the present invention).

  Note that the captured data may be encoded by the camera ex113, the streaming server ex103 that performs data transmission processing, or may be shared with each other. Similarly, the decryption processing of the distributed data may be performed by the client, the streaming server ex103, or may be performed in common with each other. In addition to the camera ex113, still images and / or moving image data captured by the camera ex116 may be transmitted to the streaming server ex103 via the computer ex111. The encoding process in this case may be performed by any of the camera ex116, the computer ex111, and the streaming server ex103, or may be performed in a shared manner.

  These encoding / decoding processes are generally performed by the computer ex111 and the LSI ex500 included in each device. The LSI ex500 may be configured as a single chip or a plurality of chips. It should be noted that moving image encoding / decoding software is incorporated into some recording medium (CD-ROM, flexible disk, hard disk, etc.) that can be read by the computer ex111 and the like, and encoding / decoding processing is performed using the software. May be. Furthermore, when the mobile phone ex114 is equipped with a camera, moving image data acquired by the camera may be transmitted. The moving image data at this time is data encoded by the LSI ex500 included in the mobile phone ex114.

  The streaming server ex103 may be a plurality of servers or a plurality of computers, and may process, record, and distribute data in a distributed manner.

  As described above, in the content supply system ex100, the client can receive and reproduce the encoded data. Thus, in the content supply system ex100, the information transmitted by the user can be received, decrypted and reproduced by the client in real time, and personal broadcasting can be realized even for a user who does not have special rights or facilities.

  In addition to the example of the content supply system ex100, as shown in FIG. 32, the digital broadcasting system ex200 also includes at least the video encoding device (video encoding device) or video decoding of each of the above embodiments. Any of the devices (image decoding devices) can be incorporated. Specifically, in the broadcast station ex201, multiplexed data obtained by multiplexing music data and the like on video data is transmitted to a communication or satellite ex202 via radio waves. This video data is data encoded by the moving image encoding method described in each of the above embodiments (that is, data encoded by the image encoding apparatus according to one aspect of the present invention). Receiving this, the broadcasting satellite ex202 transmits a radio wave for broadcasting, and this radio wave is received by a home antenna ex204 capable of receiving satellite broadcasting. The received multiplexed data is decoded and reproduced by an apparatus such as the television (receiver) ex300 or the set top box (STB) ex217 (that is, functions as an image decoding apparatus according to one embodiment of the present invention).

  Also, a reader / recorder ex218 that reads and decodes multiplexed data recorded on a recording medium ex215 such as a DVD or a BD, or encodes a video signal on the recording medium ex215 and, in some cases, multiplexes and writes it with a music signal. It is possible to mount the moving picture decoding apparatus or moving picture encoding apparatus described in the above embodiments. In this case, the reproduced video signal is displayed on the monitor ex219, and the video signal can be reproduced in another device or system using the recording medium ex215 on which the multiplexed data is recorded. Alternatively, a moving picture decoding apparatus may be mounted in a set-top box ex217 connected to a cable ex203 for cable television or an antenna ex204 for satellite / terrestrial broadcasting and displayed on the monitor ex219 of the television. At this time, the moving picture decoding apparatus may be incorporated in the television instead of the set top box.

  FIG. 33 is a diagram illustrating a television (receiver) ex300 that uses the video decoding method and the video encoding method described in each of the above embodiments. The television ex300 obtains or outputs multiplexed data in which audio data is multiplexed with video data via the antenna ex204 or the cable ex203 that receives the broadcast, and demodulates the received multiplexed data. Alternatively, the modulation / demodulation unit ex302 that modulates multiplexed data to be transmitted to the outside, and the demodulated multiplexed data is separated into video data and audio data, or the video data and audio data encoded by the signal processing unit ex306 Is provided with a multiplexing / demultiplexing unit ex303.

  The television ex300 also decodes the audio data and the video data, or encodes the information, the audio signal processing unit ex304, the video signal processing unit ex305 (the image encoding device or the image according to one embodiment of the present invention) A signal processing unit ex306 that functions as a decoding device), a speaker ex307 that outputs the decoded audio signal, and an output unit ex309 that includes a display unit ex308 such as a display that displays the decoded video signal. Furthermore, the television ex300 includes an interface unit ex317 including an operation input unit ex312 that receives an input of a user operation. Furthermore, the television ex300 includes a control unit ex310 that performs overall control of each unit, and a power supply circuit unit ex311 that supplies power to each unit. In addition to the operation input unit ex312, the interface unit ex317 includes a bridge unit ex313 connected to an external device such as a reader / recorder ex218, a recording unit ex216 such as an SD card, and an external recording unit such as a hard disk. A driver ex315 for connecting to a medium, a modem ex316 for connecting to a telephone network, and the like may be included. Note that the recording medium ex216 is capable of electrically recording information by using a nonvolatile / volatile semiconductor memory element to be stored. Each part of the television ex300 is connected to each other via a synchronous bus.

  First, a configuration in which the television ex300 decodes and reproduces multiplexed data acquired from the outside by the antenna ex204 or the like will be described. The television ex300 receives a user operation from the remote controller ex220 or the like, and demultiplexes the multiplexed data demodulated by the modulation / demodulation unit ex302 by the multiplexing / demultiplexing unit ex303 based on the control of the control unit ex310 having a CPU or the like. Furthermore, in the television ex300, the separated audio data is decoded by the audio signal processing unit ex304, and the separated video data is decoded by the video signal processing unit ex305 using the decoding method described in each of the above embodiments. The decoded audio signal and video signal are output from the output unit ex309 to the outside. At the time of output, these signals may be temporarily stored in the buffers ex318, ex319, etc. so that the audio signal and the video signal are reproduced in synchronization. Also, the television ex300 may read multiplexed data from recording media ex215 and ex216 such as a magnetic / optical disk and an SD card, not from broadcasting. Next, a configuration in which the television ex300 encodes an audio signal or a video signal and transmits the signal to the outside or to a recording medium will be described. The television ex300 receives a user operation from the remote controller ex220 and the like, encodes an audio signal with the audio signal processing unit ex304, and converts the video signal with the video signal processing unit ex305 based on the control of the control unit ex310. Encoding is performed using the encoding method described in (1). The encoded audio signal and video signal are multiplexed by the multiplexing / demultiplexing unit ex303 and output to the outside. When multiplexing, these signals may be temporarily stored in the buffers ex320, ex321, etc. so that the audio signal and the video signal are synchronized. Note that a plurality of buffers ex318, ex319, ex320, and ex321 may be provided as illustrated, or one or more buffers may be shared. Further, in addition to the illustrated example, data may be stored in the buffer as a buffer material that prevents system overflow and underflow, for example, between the modulation / demodulation unit ex302 and the multiplexing / demultiplexing unit ex303.

  In addition to acquiring audio data and video data from broadcasts, recording media, and the like, the television ex300 has a configuration for receiving AV input of a microphone and a camera, and performs encoding processing on the data acquired from them. Also good. Here, the television ex300 has been described as a configuration capable of the above-described encoding processing, multiplexing, and external output, but these processing cannot be performed, and only the above-described reception, decoding processing, and external output are possible. It may be a configuration.

  In addition, when reading or writing multiplexed data from a recording medium by the reader / recorder ex218, the decoding process or the encoding process may be performed by either the television ex300 or the reader / recorder ex218, The reader / recorder ex218 may share with each other.

  As an example, FIG. 34 shows a configuration of an information reproducing / recording unit ex400 when data is read from or written to an optical disk. The information reproducing / recording unit ex400 includes elements ex401, ex402, ex403, ex404, ex405, ex406, and ex407 described below. The optical head ex401 irradiates a laser spot on the recording surface of the recording medium ex215 that is an optical disk to write information, and detects information reflected from the recording surface of the recording medium ex215 to read the information. The modulation recording unit ex402 electrically drives a semiconductor laser built in the optical head ex401 and modulates the laser beam according to the recording data. The reproduction demodulator ex403 amplifies the reproduction signal obtained by electrically detecting the reflected light from the recording surface by the photodetector built in the optical head ex401, separates and demodulates the signal component recorded on the recording medium ex215, and is necessary To play back information. The buffer ex404 temporarily holds information to be recorded on the recording medium ex215 and information reproduced from the recording medium ex215. The disk motor ex405 rotates the recording medium ex215. The servo control unit ex406 moves the optical head ex401 to a predetermined information track while controlling the rotational drive of the disk motor ex405, and performs a laser spot tracking process. The system control unit ex407 controls the entire information reproduction / recording unit ex400. In the reading and writing processes described above, the system control unit ex407 uses various types of information held in the buffer ex404, and generates and adds new information as necessary. The modulation recording unit ex402, the reproduction demodulation unit This is realized by recording / reproducing information through the optical head ex401 while operating the ex403 and the servo control unit ex406 in a coordinated manner. The system control unit ex407 includes, for example, a microprocessor, and executes these processes by executing a read / write program.

  In the above, the optical head ex401 has been described as irradiating a laser spot. However, the optical head ex401 may be configured to perform higher-density recording using near-field light.

  FIG. 35 shows a schematic diagram of a recording medium ex215 that is an optical disk. Guide grooves (grooves) are formed in a spiral shape on the recording surface of the recording medium ex215, and address information indicating the absolute position on the disc is recorded in advance on the information track ex230 by changing the shape of the groove. This address information includes information for specifying the position of the recording block ex231 that is a unit for recording data, and the recording block is specified by reproducing the information track ex230 and reading the address information in a recording or reproducing apparatus. Can do. Further, the recording medium ex215 includes a data recording area ex233, an inner peripheral area ex232, and an outer peripheral area ex234. The area used for recording user data is the data recording area ex233, and the inner circumference area ex232 and the outer circumference area ex234 arranged on the inner or outer circumference of the data recording area ex233 are used for specific purposes other than user data recording. Used. The information reproducing / recording unit ex400 reads / writes encoded audio data, video data, or multiplexed data obtained by multiplexing these data with respect to the data recording area ex233 of the recording medium ex215.

  In the above description, an optical disk such as a single-layer DVD or BD has been described as an example. However, the present invention is not limited to these, and an optical disk having a multilayer structure and capable of recording other than the surface may be used. Also, an optical disc with a multi-dimensional recording / reproducing structure, such as recording information using light of different wavelengths in the same place on the disc, or recording different layers of information from various angles. It may be.

  In the digital broadcasting system ex200, the car ex210 having the antenna ex205 can receive data from the satellite ex202 and the like, and the moving image can be reproduced on the display device such as the car navigation ex211 that the car ex210 has. The configuration of the car navigation ex211 may be, for example, a configuration in which a GPS receiving unit is added in the configuration illustrated in FIG. 33, and the same may be considered for the computer ex111, the mobile phone ex114, and the like.

  FIG. 36A is a diagram illustrating the mobile phone ex114 using the video decoding method and the video encoding method described in the above embodiment. The mobile phone ex114 includes an antenna ex350 for transmitting and receiving radio waves to and from the base station ex110, a camera unit ex365 capable of capturing video and still images, a video captured by the camera unit ex365, a video received by the antenna ex350, and the like Is provided with a display unit ex358 such as a liquid crystal display for displaying the decrypted data. The mobile phone ex114 further includes a main body unit having an operation key unit ex366, an audio output unit ex357 such as a speaker for outputting audio, an audio input unit ex356 such as a microphone for inputting audio, a captured video, In the memory unit ex367 for storing encoded data or decoded data such as still images, recorded audio, received video, still images, mails, or the like, or an interface unit with a recording medium for storing data A slot ex364 is provided.

  Further, a configuration example of the mobile phone ex114 will be described with reference to FIG. 36B. The mobile phone ex114 has a power supply circuit part ex361, an operation input control part ex362, and a video signal processing part ex355 with respect to a main control part ex360 that comprehensively controls each part of the main body including the display part ex358 and the operation key part ex366. , A camera interface unit ex363, an LCD (Liquid Crystal Display) control unit ex359, a modulation / demodulation unit ex352, a multiplexing / demultiplexing unit ex353, an audio signal processing unit ex354, a slot unit ex364, and a memory unit ex367 are connected to each other via a bus ex370. ing.

  When the end call and the power key are turned on by a user operation, the power supply circuit ex361 starts up the mobile phone ex114 in an operable state by supplying power from the battery pack to each unit.

  The cellular phone ex114 converts the audio signal collected by the audio input unit ex356 in the voice call mode into a digital audio signal by the audio signal processing unit ex354 based on the control of the main control unit ex360 having a CPU, a ROM, a RAM, and the like. Then, this is subjected to spectrum spread processing by the modulation / demodulation unit ex352, digital-analog conversion processing and frequency conversion processing are performed by the transmission / reception unit ex351, and then transmitted via the antenna ex350. The mobile phone ex114 also amplifies the received data received via the antenna ex350 in the voice call mode, performs frequency conversion processing and analog-digital conversion processing, performs spectrum despreading processing by the modulation / demodulation unit ex352, and performs voice signal processing unit After being converted into an analog audio signal by ex354, this is output from the audio output unit ex357.

  Further, when an e-mail is transmitted in the data communication mode, the text data of the e-mail input by operating the operation key unit ex366 of the main unit is sent to the main control unit ex360 via the operation input control unit ex362. The main control unit ex360 performs spread spectrum processing on the text data in the modulation / demodulation unit ex352, performs digital analog conversion processing and frequency conversion processing in the transmission / reception unit ex351, and then transmits the text data to the base station ex110 via the antenna ex350. . In the case of receiving an e-mail, almost the reverse process is performed on the received data and output to the display unit ex358.

  When transmitting video, still images, or video and audio in the data communication mode, the video signal processing unit ex355 compresses the video signal supplied from the camera unit ex365 by the moving image encoding method described in the above embodiments. Encode (that is, function as an image encoding device according to an aspect of the present invention), and send the encoded video data to the multiplexing / demultiplexing unit ex353. The audio signal processing unit ex354 encodes the audio signal picked up by the audio input unit ex356 while the camera unit ex365 images a video, a still image, etc., and sends the encoded audio data to the multiplexing / separating unit ex353. To do.

  The multiplexing / demultiplexing unit ex353 multiplexes the encoded video data supplied from the video signal processing unit ex355 and the encoded audio data supplied from the audio signal processing unit ex354 by a predetermined method, and is obtained as a result. The multiplexed data is subjected to spread spectrum processing by the modulation / demodulation unit (modulation / demodulation circuit unit) ex352, digital-analog conversion processing and frequency conversion processing by the transmission / reception unit ex351, and then transmitted via the antenna ex350.

  Decode multiplexed data received via antenna ex350 when receiving video file data linked to a homepage, etc. in data communication mode, or when receiving e-mail with video and / or audio attached Therefore, the multiplexing / separating unit ex353 separates the multiplexed data into a video data bit stream and an audio data bit stream, and performs video signal processing on the video data encoded via the synchronization bus ex370. The encoded audio data is supplied to the audio signal processing unit ex354 while being supplied to the unit ex355. The video signal processing unit ex355 decodes the video signal by decoding using the video decoding method corresponding to the video encoding method described in each of the above embodiments (that is, an image according to an aspect of the present invention). For example, video and still images included in the moving image file linked to the home page are displayed from the display unit ex358 via the LCD control unit ex359. The audio signal processing unit ex354 decodes the audio signal, and the audio is output from the audio output unit ex357.

  In addition to the transmission / reception type terminal having both the encoder and the decoder, the terminal such as the mobile phone ex114 is referred to as a transmission terminal having only an encoder and a receiving terminal having only a decoder. There are three possible mounting formats. Furthermore, in the digital broadcasting system ex200, it has been described that multiplexed data in which music data or the like is multiplexed with video data is received and transmitted, but data in which character data or the like related to video is multiplexed in addition to audio data It may be video data itself instead of multiplexed data.

  As described above, the moving picture encoding method or the moving picture decoding method shown in each of the above embodiments can be used in any of the above-described devices / systems. The described effect can be obtained.

  Further, the present invention is not limited to the above-described embodiment, and various modifications or corrections can be made without departing from the scope of the present invention.

(Embodiment 7)
The moving picture coding method or apparatus shown in the above embodiments and the moving picture coding method or apparatus compliant with different standards such as MPEG-2, MPEG4-AVC, and VC-1 are appropriately switched as necessary. Thus, it is also possible to generate video data.

  Here, when a plurality of video data compliant with different standards are generated, it is necessary to select a decoding method corresponding to each standard when decoding. However, since it is impossible to identify which standard the video data to be decoded complies with, there arises a problem that an appropriate decoding method cannot be selected.

  In order to solve this problem, multiplexed data obtained by multiplexing audio data and the like on video data is configured to include identification information indicating which standard the video data conforms to. A specific configuration of multiplexed data including video data generated by the moving picture encoding method or apparatus shown in the above embodiments will be described below. The multiplexed data is a digital stream in the MPEG-2 transport stream format.

  FIG. 37 is a diagram showing a structure of multiplexed data. As shown in FIG. 37, multiplexed data is obtained by multiplexing one or more of a video stream, an audio stream, a presentation graphics stream (PG), and an interactive graphics stream. The video stream indicates the main video and sub-video of the movie, the audio stream (IG) indicates the main audio portion of the movie and the sub-audio mixed with the main audio, and the presentation graphics stream indicates the subtitles of the movie. Here, the main video indicates a normal video displayed on the screen, and the sub-video is a video displayed on a small screen in the main video. The interactive graphics stream indicates an interactive screen created by arranging GUI components on the screen. The video stream is encoded by the moving image encoding method or apparatus described in the above embodiments, or the moving image encoding method or apparatus conforming to the conventional standards such as MPEG-2, MPEG4-AVC, and VC-1. ing. The audio stream is encoded by a method such as Dolby AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, or linear PCM.

  Each stream included in the multiplexed data is identified by PID. For example, 0x1011 for video streams used for movie images, 0x1100 to 0x111F for audio streams, 0x1200 to 0x121F for presentation graphics, 0x1400 to 0x141F for interactive graphics streams, 0x1B00 to 0x1B1F are assigned to video streams used for sub-pictures, and 0x1A00 to 0x1A1F are assigned to audio streams used for sub-audio mixed with the main audio.

  FIG. 38 is a diagram schematically showing how multiplexed data is multiplexed. First, a video stream ex235 composed of a plurality of video frames and an audio stream ex238 composed of a plurality of audio frames are converted into PES packet sequences ex236 and ex239, respectively, and converted into TS packets ex237 and ex240. Similarly, the data of the presentation graphics stream ex241 and interactive graphics ex244 are converted into PES packet sequences ex242 and ex245, respectively, and further converted into TS packets ex243 and ex246. The multiplexed data ex247 is configured by multiplexing these TS packets into one stream.

  FIG. 39 shows in more detail how the video stream is stored in the PES packet sequence. The first row in FIG. 39 shows a video frame sequence of the video stream. The second level shows a PES packet sequence. As shown by arrows yy1, yy2, yy3, and yy4 in FIG. 39, a plurality of Video Presentation Units in the video stream are divided into pictures, B pictures, and P pictures, and are stored in the payload of the PES packet. . Each PES packet has a PES header, and a PTS (Presentation Time-Stamp) that is a picture display time and a DTS (Decoding Time-Stamp) that is a picture decoding time are stored in the PES header.

  FIG. 40 shows the format of a TS packet that is finally written into the multiplexed data. The TS packet is a 188-byte fixed-length packet composed of a 4-byte TS header having information such as a PID for identifying a stream and a 184-byte TS payload for storing data. The PES packet is divided and stored in the TS payload. The In the case of a BD-ROM, a 4-byte TP_Extra_Header is added to a TS packet, forms a 192-byte source packet, and is written in multiplexed data. In TP_Extra_Header, information such as ATS (Arrival_Time_Stamp) is described. ATS indicates the transfer start time of the TS packet to the PID filter of the decoder. As shown in the lower part of FIG. 40, source packets are arranged in the multiplexed data, and the number incremented from the head of the multiplexed data is called SPN (source packet number).

  In addition, TS packets included in multiplexed data include PAT (Program Association Table), PMT (Program Map Table), PCR (Program Clock Reference), and the like in addition to video, audio, and subtitle streams. PAT indicates what the PID of the PMT used in the multiplexed data is, and the PID of the PAT itself is registered as 0. The PMT has the PID of each stream such as video / audio / subtitles included in the multiplexed data and the attribute information of the stream corresponding to each PID, and has various descriptors related to the multiplexed data. The descriptor includes copy control information for instructing permission / non-permission of copying of multiplexed data. In order to synchronize ATC (Arrival Time Clock), which is the time axis of ATS, and STC (System Time Clock), which is the time axis of PTS / DTS, the PCR corresponds to the ATS in which the PCR packet is transferred to the decoder. Contains STC time information.

  FIG. 41 is a diagram for explaining the data structure of the PMT in detail. A PMT header describing the length of data included in the PMT is arranged at the head of the PMT. After that, a plurality of descriptors related to multiplexed data are arranged. The copy control information and the like are described as descriptors. After the descriptor, a plurality of pieces of stream information regarding each stream included in the multiplexed data are arranged. The stream information includes a stream descriptor in which a stream type, a stream PID, and stream attribute information (frame rate, aspect ratio, etc.) are described to identify a compression codec of the stream. There are as many stream descriptors as the number of streams existing in the multiplexed data.

  When recording on a recording medium or the like, the multiplexed data is recorded together with the multiplexed data information file.

  As shown in FIG. 42, the multiplexed data information file is management information of multiplexed data, has a one-to-one correspondence with the multiplexed data, and includes multiplexed data information, stream attribute information, and an entry map.

  As shown in FIG. 42, the multiplexed data information includes a system rate, a reproduction start time, and a reproduction end time. The system rate indicates a maximum transfer rate of multiplexed data to a PID filter of a system target decoder described later. The ATS interval included in the multiplexed data is set to be equal to or less than the system rate. The playback start time is the PTS of the first video frame of the multiplexed data, and the playback end time is set by adding the playback interval for one frame to the PTS of the video frame at the end of the multiplexed data.

  As shown in FIG. 43, in the stream attribute information, attribute information about each stream included in the multiplexed data is registered for each PID. The attribute information has different information for each video stream, audio stream, presentation graphics stream, and interactive graphics stream. The video stream attribute information includes the compression codec used to compress the video stream, the resolution of the individual picture data constituting the video stream, the aspect ratio, and the frame rate. It has information such as how much it is. The audio stream attribute information includes the compression codec used to compress the audio stream, the number of channels included in the audio stream, the language supported, and the sampling frequency. With information. These pieces of information are used for initialization of the decoder before the player reproduces it.

  In the present embodiment, among the multiplexed data, the stream type included in the PMT is used. Also, when multiplexed data is recorded on the recording medium, video stream attribute information included in the multiplexed data information is used. Specifically, in the video encoding method or apparatus shown in each of the above embodiments, the video encoding shown in each of the above embodiments for the stream type or video stream attribute information included in the PMT. There is provided a step or means for setting unique information indicating that the video data is generated by the method or apparatus. With this configuration, it is possible to discriminate between video data generated by the moving picture encoding method or apparatus described in the above embodiments and video data compliant with other standards.

  FIG. 44 shows steps of the moving picture decoding method according to the present embodiment. In step exS100, the stream type included in the PMT or the video stream attribute information included in the multiplexed data information is acquired from the multiplexed data. Next, in step exS101, it is determined whether or not the stream type or the video stream attribute information indicates multiplexed data generated by the moving picture encoding method or apparatus described in the above embodiments. To do. When it is determined that the stream type or the video stream attribute information is generated by the moving image encoding method or apparatus described in the above embodiments, in step exS102, the above embodiments are performed. Decoding is performed by the moving picture decoding method shown in the form. If the stream type or the video stream attribute information indicates that it conforms to a standard such as conventional MPEG-2, MPEG4-AVC, or VC-1, in step exS103, Decoding is performed by a moving image decoding method compliant with the standard.

  In this way, by setting a new unique value in the stream type or video stream attribute information, whether or not decoding is possible with the moving picture decoding method or apparatus described in each of the above embodiments is performed. Judgment can be made. Therefore, even when multiplexed data conforming to different standards is input, an appropriate decoding method or apparatus can be selected, and therefore decoding can be performed without causing an error. In addition, the moving picture encoding method or apparatus or the moving picture decoding method or apparatus described in this embodiment can be used in any of the above-described devices and systems.

(Embodiment 8)
The moving picture encoding method and apparatus and moving picture decoding method and apparatus described in the above embodiments are typically realized by an LSI that is an integrated circuit. As an example, FIG. 45 shows a configuration of LSI ex500 that is made into one chip. The LSI ex500 includes elements ex501, ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 described below, and each element is connected via a bus ex510. The power supply circuit unit ex505 is activated to an operable state by supplying power to each unit when the power supply is on.

  For example, when performing the encoding process, the LSI ex500 uses the AV I / O ex509 to perform the microphone ex117 and the camera ex113 based on the control of the control unit ex501 including the CPU ex502, the memory controller ex503, the stream controller ex504, the drive frequency control unit ex512, and the like. The AV signal is input from the above. The input AV signal is temporarily stored in an external memory ex511 such as SDRAM. Based on the control of the control unit ex501, the accumulated data is divided into a plurality of times as appropriate according to the processing amount and the processing speed and sent to the signal processing unit ex507. Signal encoding is performed. Here, the encoding process of the video signal is the encoding process described in the above embodiments. The signal processing unit ex507 further performs processing such as multiplexing the encoded audio data and the encoded video data according to circumstances, and outputs the result from the stream I / Oex 506 to the outside. The output multiplexed data is transmitted to the base station ex107 or written to the recording medium ex215. It should be noted that data should be temporarily stored in the buffer ex508 so as to be synchronized when multiplexing.

  In the above description, the memory ex511 is described as an external configuration of the LSI ex500. However, a configuration included in the LSI ex500 may be used. The number of buffers ex508 is not limited to one, and a plurality of buffers may be provided. The LSI ex500 may be made into one chip or a plurality of chips.

  In the above description, the control unit ex501 includes the CPU ex502, the memory controller ex503, the stream controller ex504, the drive frequency control unit ex512, and the like, but the configuration of the control unit ex501 is not limited to this configuration. For example, the signal processing unit ex507 may further include a CPU. By providing a CPU also in the signal processing unit ex507, the processing speed can be further improved. As another example, the CPU ex502 may be configured to include a signal processing unit ex507 or, for example, an audio signal processing unit that is a part of the signal processing unit ex507. In such a case, the control unit ex501 is configured to include a signal processing unit ex507 or a CPU ex502 having a part thereof.

  Here, although LSI is used, it may be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. Such a programmable logic device typically loads or reads a program constituting software or firmware from a memory or the like, so that the moving image encoding method or the moving image described in each of the above embodiments is used. An image decoding method can be performed.

  Furthermore, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

(Embodiment 9)
When decoding video data generated by the moving picture encoding method or apparatus described in the above embodiments, video data compliant with standards such as MPEG-2, MPEG4-AVC, and VC-1 is decoded. It is conceivable that the amount of processing increases compared to the case. Therefore, in LSI ex500, it is necessary to set a driving frequency higher than the driving frequency of CPU ex502 when decoding video data compliant with the conventional standard. However, when the drive frequency is increased, there is a problem that power consumption increases.

  In order to solve this problem, moving picture decoding apparatuses such as the television ex300 and the LSI ex500 are configured to identify which standard the video data conforms to and switch the driving frequency according to the standard. FIG. 46 shows a configuration ex800 in the present embodiment. The drive frequency switching unit ex803 sets the drive frequency high when the video data is generated by the moving image encoding method or apparatus described in the above embodiments. Then, the decoding processing unit ex801 that executes the moving picture decoding method described in each of the above embodiments is instructed to decode the video data. On the other hand, when the video data is video data compliant with the conventional standard, compared to the case where the video data is generated by the moving picture encoding method or apparatus shown in the above embodiments, Set the drive frequency low. Then, it instructs the decoding processing unit ex802 compliant with the conventional standard to decode the video data.

  More specifically, the drive frequency switching unit ex803 includes the CPU ex502 and the drive frequency control unit ex512 in FIG. Also, the decoding processing unit ex801 that executes the moving picture decoding method described in each of the above embodiments and the decoding processing unit ex802 that complies with the conventional standard correspond to the signal processing unit ex507 in FIG. The CPU ex502 identifies which standard the video data conforms to. Then, based on the signal from the CPU ex502, the drive frequency control unit ex512 sets the drive frequency. Further, based on the signal from the CPU ex502, the signal processing unit ex507 decodes the video data. Here, for example, the identification information described in the seventh embodiment may be used for identifying the video data. The identification information is not limited to that described in Embodiment 7, and any information that can identify which standard the video data conforms to may be used. For example, it is possible to identify which standard the video data conforms to based on an external signal that identifies whether the video data is used for a television or a disk. In some cases, identification may be performed based on such an external signal. In addition, the selection of the drive frequency in the CPU ex502 may be performed based on, for example, a lookup table in which video data standards and drive frequencies are associated with each other as shown in FIG. The look-up table is stored in the buffer ex508 or the internal memory of the LSI, and the CPU ex502 can select the drive frequency by referring to the look-up table.

  FIG. 47 shows steps for executing a method in the present embodiment. First, in step exS200, the signal processing unit ex507 acquires identification information from the multiplexed data. Next, in step exS201, the CPU ex502 identifies whether the video data is generated by the encoding method or apparatus described in each of the above embodiments based on the identification information. When the video data is generated by the encoding method or apparatus shown in the above embodiments, in step exS202, the CPU ex502 sends a signal for setting the drive frequency high to the drive frequency control unit ex512. Then, the drive frequency control unit ex512 sets a high drive frequency. On the other hand, if the video data conforms to the standards such as MPEG-2, MPEG4-AVC, and VC-1, the CPU ex502 drives the signal for setting the drive frequency low in step exS203. This is sent to the frequency control unit ex512. Then, in the drive frequency control unit ex512, the drive frequency is set to be lower than that in the case where the video data is generated by the encoding method or apparatus described in the above embodiments.

  Furthermore, the power saving effect can be further enhanced by changing the voltage applied to the LSI ex500 or the device including the LSI ex500 in conjunction with the switching of the driving frequency. For example, when the drive frequency is set low, it is conceivable that the voltage applied to the LSI ex500 or the device including the LSI ex500 is set low as compared with the case where the drive frequency is set high.

  In addition, the setting method of the driving frequency may be set to a high driving frequency when the processing amount at the time of decoding is large, and to a low driving frequency when the processing amount at the time of decoding is small. It is not limited to the method. For example, the amount of processing for decoding video data compliant with the MPEG4-AVC standard is larger than the amount of processing for decoding video data generated by the moving picture encoding method or apparatus described in the above embodiments. It is conceivable that the setting of the driving frequency is reversed to that in the case described above.

  Further, the method for setting the drive frequency is not limited to the configuration in which the drive frequency is lowered. For example, when the identification information indicates that the video data is generated by the moving image encoding method or apparatus described in the above embodiments, the voltage applied to the LSIex500 or the apparatus including the LSIex500 is set high. However, if the video data conforms to the standards such as MPEG-2, MPEG4-AVC, VC-1, etc., it may be considered to set the voltage applied to the device including LSIex500 or LSIex500 low. It is done. As another example, when the identification information indicates that the video data is generated by the moving image encoding method or apparatus described in the above embodiments, the driving of the CPU ex502 is stopped. If the video data conforms to the standards such as MPEG-2, MPEG4-AVC, VC-1, etc., the CPU ex502 is temporarily stopped because there is enough processing. Is also possible. Even when the identification information indicates that the video data is generated by the moving image encoding method or apparatus described in each of the above embodiments, if there is a margin for processing, the CPU ex502 is temporarily driven. It can also be stopped. In this case, it is conceivable to set the stop time shorter than in the case where the video data conforms to the standards such as MPEG-2, MPEG4-AVC, and VC-1.

  In this way, it is possible to save power by switching the drive frequency according to the standard to which the video data complies. In addition, when the battery is used to drive the LSI ex500 or the device including the LSI ex500, it is possible to extend the life of the battery with power saving.

(Embodiment 10)
A plurality of video data that conforms to different standards may be input to the above-described devices and systems such as a television and a mobile phone. As described above, the signal processing unit ex507 of the LSI ex500 needs to support a plurality of standards in order to be able to decode even when a plurality of video data complying with different standards is input. However, when the signal processing unit ex507 corresponding to each standard is used individually, there is a problem that the circuit scale of the LSI ex500 increases and the cost increases.

  In order to solve this problem, a decoding processing unit for executing the moving picture decoding method shown in each of the above embodiments and a decoding conforming to a standard such as conventional MPEG-2, MPEG4-AVC, or VC-1 The processing unit is partly shared. An example of this configuration is shown as ex900 in FIG. 49A. For example, the moving picture decoding method shown in the above embodiments and the moving picture decoding method compliant with the MPEG4-AVC standard are processed in processes such as entropy coding, inverse quantization, deblocking filter, and motion compensation. Some contents are common. For the common processing content, the decoding processing unit ex902 corresponding to the MPEG4-AVC standard is shared, and for other processing content specific to one aspect of the present invention that does not correspond to the MPEG4-AVC standard, a dedicated decoding processing unit A configuration using ex901 is conceivable. In particular, since one aspect of the present invention is characterized by motion compensation, for example, a dedicated decoding processing unit ex901 is used for motion compensation, and other entropy decoding, deblocking filter, and inverse quantization are performed. For any or all of these processes, it is conceivable to share the decoding processing unit. Regarding the sharing of the decoding processing unit, regarding the common processing content, the decoding processing unit for executing the moving picture decoding method described in each of the above embodiments is shared, and the processing content specific to the MPEG4-AVC standard As for, a configuration using a dedicated decoding processing unit may be used.

  Further, ex1000 in FIG. 49B shows another example in which processing is partially shared. In this example, a dedicated decoding processing unit ex1001 corresponding to the processing content specific to one aspect of the present invention, a dedicated decoding processing unit ex1002 corresponding to the processing content specific to another conventional standard, and one aspect of the present invention And a common decoding processing unit ex1003 corresponding to the processing contents common to the moving image decoding method according to the above and other conventional moving image decoding methods. Here, the dedicated decoding processing units ex1001 and ex1002 are not necessarily specialized in one aspect of the present invention or processing content specific to other conventional standards, and can execute other general-purpose processing. Also good. Also, the configuration of the present embodiment can be implemented by LSI ex500.

  As described above, the processing content common to the moving picture decoding method according to one aspect of the present invention and the moving picture decoding method of the conventional standard reduces the circuit scale of the LSI by sharing the decoding processing unit, In addition, the cost can be reduced.

  The present invention is applicable to, for example, a television receiver, a digital video recorder, a car navigation system, a mobile phone, a digital camera, a digital video camera, a content distribution server, or a content distribution system.

100, 1000 Image processing system 110 Broadcast system 200, 1100 Decoding device (image decoding device)
201 Entropy decoding unit 202 Inverse quantization unit 203 Inverse transform unit 204 Block memory 205 Frame memory 206, 307 Intra prediction unit 207 Inter prediction unit 210, 310 Processing unit 220, 320 Storage unit 230, 330 Communication unit 240, 340, 410 Control Part 250, 450, 650 Display part (monitor)
300, 1200 server (image transmission device)
301 Subtractor 302 Transform Quantizer 303 Entropy Coding Unit 304 Inverse Quantization Inverse Transformer 305 Adder 306 Deblocking Filter Unit 308 Motion Compensator 309 Motion Detector 311 Memory 400 Television (Receiver)
401 tuner 402 modulation / demodulation unit 403 multiplexing / demultiplexing unit 404 audio signal processing unit 405 video signal processing unit 409 output unit 411 power supply circuit unit 412 operation input unit 413 bridge 414 slot unit 415 driver 416 modem 417 interface unit 420 signal processing unit 430 , 431, 432 Recording media 433, 434, 435, 436 Buffer 460 Set top box (STB)
470 Reader / recorder 480 Speaker 490 Remote controller (remote control)
500 broadcast station 501 satellite 502 cable 503, 802 antenna 600 playback device 700 mobile phone (smart phone)
800 Car navigation (car navigation)
801 car 900 LSI
1101 Selection unit 1102, 1203 Transmission unit 1103 Request unit 1104, 1201 Reception unit 1105 Decoding unit 1106 Display control unit 1107 Display device 1110, 1210 Storage unit 1202 Request reception unit

Claims (18)

An image decoding device that receives an encoded image from an image transmission device and decodes the image received from the image transmission device,
A request unit that requests the image transmission apparatus to transmit the image;
A transmission unit that transmits, to the image transmission device, range information that indicates a range of a part of the image of the entire range of the image and is selected according to a user operation;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is received from the image transmission device; and (ii) the image is a second type. Of the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and reference to outside the range corresponding to each tile is prohibited in inter prediction. A receiving unit for receiving a part of the image from the image transmitting device by receiving only one or more tiles;
An image decoding device comprising: a decoding unit that decodes all or part of the image received from the image transmission device by the reception unit. The image decoding apparatus according to claim 1, further comprising: a display control unit that displays on the first display device all or part of the image decoded by the decoding unit. When receiving the part of the image, the receiving unit receives the part of the image according to the stored range information,
The decoding unit, when decoding a part of the image, decodes the part of the image received according to the stored range information,
The image decoding apparatus according to claim 2, wherein when displaying a part of the image, the display control unit displays the part of the image received and decoded according to the stored range information. The display control unit displays a part of the image received and decoded in accordance with the stored range information when displaying a part of the image after displaying the whole of another image. The image decoding device described. The display control unit may further display a part of the image received and decoded according to the stored range information when the display of the part of the image is resumed after the display of the part of the image is stopped. The image decoding device according to claim 3 or 4. The display control unit further displays a part of the image received and decoded according to the stored range information on a second display device different from the first display device. The image decoding device according to item 1. When the range information is not stored, the transmission unit transmits the range information to the image transmission device,
The range in which the range information is stored without being transmitted to the image transmission apparatus by the transmission unit when the range information is stored and a part of the image is received. The image decoding apparatus according to claim 3, wherein a part of the image is received according to information. The image decoding apparatus further includes a selection unit that selects a partial range of the image according to an operation of the user,
The image decoding device according to claim 1, wherein the transmission unit transmits the range information indicating the range selected by the selection unit to the image transmission device. In the state where the range of a part of the image is selected,
If the image belongs to the first type recommended for full display, the entire image is received;
The image decoding device according to any one of claims 1 to 8, wherein when the image belongs to the second type whose whole display is not recommended, a part of the image is received. In the state where the range of a part of the image is selected,
If the image belongs to the first type of commercial, the entire image is received;
The image decoding device according to any one of claims 1 to 9, wherein when the image belongs to the second type different from commercial, a part of the image is received. The image is provided with a flag indicating whether the image belongs to the first type or the second type,
The image decoding device according to claim 1, wherein the reception unit receives all or part of the image according to the flag. An image transmission device that transmits an encoded image to an image decoding device,
A request receiving unit that receives a request for transmission of the image from the image decoding device;
A receiving unit that receives, from the image decoding device, range information indicating a range of a part of the image out of the entire range of the image and selected in accordance with a user operation;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is transmitted to the image decoding device, and (ii) the image is a second type. Of the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and reference to outside the range corresponding to each tile is prohibited in inter prediction. An image transmission apparatus comprising: a transmission unit configured to transmit a part of the image to the image decoding apparatus by transmitting only one or more tiles. In the state where the range of a part of the image is selected,
If the image belongs to the first type that is recommended for full display, send the entire image,
The image transmission device according to claim 12, wherein when the image belongs to the second type whose whole display is not recommended, a part of the image is transmitted. In the state where the range of a part of the image is selected,
If the image belongs to the first type of commercial, send the entire image,
The image transmission device according to claim 12 or 13, wherein when the image belongs to the second type different from the commercial, a part of the image is transmitted. The image is provided with a flag indicating whether the image belongs to the first type or the second type,
The image transmission device according to claim 12, wherein the transmission unit transmits all or part of the image according to the flag. An image processing system for processing an encoded image,
An image transmitting device for transmitting the image to an image decoding device;
The image decoding device that receives the image from the image transmission device and decodes the image received from the image transmission device,
The image decoding device includes:
A request unit that requests the image transmission apparatus to transmit the image;
A first transmission unit that transmits, to the image transmission device, range information indicating a range that is a partial range of the image in the entire range of the image and that is selected according to a user operation;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is received from the image transmission device; and (ii) the image is a second type. Of the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and reference to outside the range corresponding to each tile is prohibited in inter prediction. A first receiving unit for receiving a part of the image from the image transmitting device by receiving only one or more tiles;
A decoding unit that decodes all or part of the image received from the image transmission device by the first receiving unit;
The image transmission device includes:
A request receiving unit that receives a request for transmission of the image from the image decoding device;
A second receiving unit for receiving the range information from the image decoding device;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is transmitted to the image decoding device, and (ii) the image is the first image. When belonging to two types, the image processing apparatus includes: a second transmission unit configured to transmit only one or more tiles of the plurality of tiles included in the image to transmit a part of the image to the image decoding device. Processing system. An image decoding method for receiving an encoded image from an image transmission device and decoding the image received from the image transmission device,
A requesting step for requesting the image transmitting apparatus to transmit the image;
A transmission step of transmitting, to the image transmission device, range information indicating a range of a part of the image of the entire range of the image and selected according to a user operation;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is received from the image transmission device; and (ii) the image is a second type. Of the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and reference to outside the range corresponding to each tile is prohibited in inter prediction. Receiving a part of the image from the image transmitting device by receiving only one or more tiles;
And a decoding step of decoding all or part of the image received from the image transmission device in the reception step. An image transmission method for transmitting an encoded image to an image decoding device,
A request receiving step for receiving a request for transmission of the image from the image decoding device;
A receiving step of receiving, from the image decoding device, range information indicating a range of a part of the image out of the entire range of the image and indicating a range selected according to a user operation;
In a state where a partial range of the image is selected, (i) when the image belongs to the first type, the entire image is transmitted to the image decoding device, and (ii) the image is a second type. Of the plurality of tiles included in the image, one or more tiles corresponding to a partial range of the image, and reference to outside the range corresponding to each tile is prohibited in inter prediction. A transmission step of transmitting a part of the image to the image decoding apparatus by transmitting only one or more tiles.

Images