CN105659610A

CN105659610A - Image processing device and method

Info

Publication number: CN105659610A
Application number: CN201480058310.7A
Authority: CN
Inventors: 中神央二; 铃木辉彦
Original assignee: Sony Corp
Current assignee: Sony Corp
Priority date: 2013-11-01
Filing date: 2014-10-20
Publication date: 2016-06-08
Also published as: JPWO2015064402A1; US20160277771A1; US10356442B2; JP6654434B2; WO2015064402A1

Abstract

The present disclosure relates to an image processing device and method whereby it is possible to facilitate improvement of encoding efficiency. The present disclosure is provided with: a packing processing unit that performs a packing process in which pieces of pixel data of RAW data, which is image data to be subjected to a demosaicing process, are sorted according to the degrees of relevance between the pieces of pixel data; and an encoding unit that encodes the RAW data after the packing processing unit has performed the packing process on the RAW data. The present disclosure can be applied to image processing devices such as, e.g., image encoding devices which encode RAW data and image decoding devices which decode encoded RAW data.

Description

Image processing equipment and method

Technical field

It relates to a kind of image processing equipment and method, and specifically, it relates to a kind of image processing equipment and method being easy to improve coding efficiency.

Background technology

In recent years, for compression as in image sensor etc. and constantly increase in the demand performing the RAW data of the view data of generation before demosaicing processes.

As system view data encoded, there is MPEG-4Part10 (advanced video coding hereinafter, writes AVC). In recent years, in order to improve coding efficiency, such as, it is called that the stdn of coding scheme of efficient video coding (HEVC) is always by carrying out as integration and cooperation team-video coding (JCTVC) of Union Internationale des Telecommunications's telecommunication standardization group (ITU-T) and the combination with standardization tissue of International Standards Organization/International Power Technical Committee (ISO/IEC) (with reference to non-patent document 1).

Quote list

Non-patent document

Non-patent document 1:BenjaminBross, Woo-JinHan, Jens-RainerOhm, GaryJ.Sullivan, Ye-KuiWang, ThomasWiegand, " HighEfficiencyVideoCoding (HEVC) textspecificationdraft10 (forFDIS&LastCall) ", JCTVC-L1003_version34,2013-03-19

Summary of the invention

The problem to be solved in the present invention

But, in RAW data, the pixel of low correlation is adjacent. Therefore, image encoding system (such as HEVC) is difficult to perform efficiently coding.

In view of above-mentioned situation has proposed the disclosure, and the disclosure makes it possible to be easy to when RAW data being encoded improve coding efficiency.

The solution of problem

The image processing equipment of an aspect of this technology is image processing equipment, comprising: packing processing unit, this packing processing unit is configured to perform packing process: according to the degree of correlation, rearranges as the pixel data in the RAW data performing the view data before demosaicing processes; And coding unit, this coding unit is configured to the RAW data to the packing process through performing by packing processing unit and encodes.

Packing processing unit can comprise: separating unit, and this separating unit is separated the pixel data of RAW data to have one by one based on the data of high correlation; Rearranging unit, this rearranges unit and rearranges by separating unit to have the pixel data group of separation based on the data of high correlation one by one; And generation unit, this generation unit generates the package information of relevant packing process.

Separating unit can carry out separate pixel data based on individual element, and this pixel is assigned the wave filter of identical type.

Separating unit can carry out separate pixel data based on individual element, and this pixel is assigned the color filter of same color.

Rearrange unit and can rearrange the component of pixel data group as predetermined color space, and component can be encoded by coding unit together or independently of one another.

Component can be encoded by coding unit based on the region of part one by one of the image of RAW data.

Rearrange unit and can rearrange the different from each other part region of pixel data group as one or more image.

Rearrange unit and can rearrange the data of pixel data group as the layering different from each other of stratification view data.

Rearrange unit and the view data processed through demosaicing can be re-arranged to part layering.

The image processing method of an aspect of this technology is image processing method, comprising: perform packing process: according to the degree of correlation, rearranges as the pixel data in the RAW data performing the view data before demosaicing processes; And the RAW data processed through packing are encoded.

The image processing equipment of the another aspect of this technology is image processing equipment, comprising: decoding unit, this decoding cell location is decoded by the coded data being the encoded RAW data as the view data before performing demosaicing process; And separate bag processing unit, this solution bag processing unit is configured to perform Xie Bao process: for the RAW data of the packing process through rearranging pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, the decoding that these RAW data are undertaken by decoding unit obtains.

Separating bag processing unit can comprise: resolution unit, this resolution unit resolves the package information of relevant packing process; Separating unit, this separating unit resolves the pixel data of the RAW data that the package information separation obtained processes through packing based on resolution unit; And rearranging unit, this rearranges unit and resolves, based on resolution unit, the package information that obtains and return to have the pixel data of separation based on the data of high correlation one by one by separating unit and to perform the layout before packing processes.

Separating unit can be separated the pixel data rearranged based on individual element, and this pixel is assigned the wave filter of identical type by packing process, and this rearranges unit and can rearrange pixel data according to the array of wave filter.

Separating unit can be separated the pixel data rearranged based on individual element, and this pixel is assigned the color filter of same color by packing process, and rearranges unit and can rearrange pixel data according to the array of color filter.

Separating unit can to have the pixel data being separated based on the data of high correlation and rearranging the component into predetermined color space one by one.

Coded data can be decoded by decoding unit based on part region one by one, and this coded data is the encoded component based on the region of part one by one of the image of RAW data.

Separating unit can to have the pixel data being separated based on the data of high correlation and rearranging in the part region different from each other of one or more image one by one.

Separating unit can to have the pixel data being separated based on the data of high correlation and rearranging in the layering different from each other of stratification view data one by one.

Separating unit can be separated the pixel data rearranged in other layering except part layering, and the view data through demosaicing process is arranged on the portion.

The image processing method of the another aspect of this technology is image processing method, comprising: decoded by the coded data being the encoded RAW data as the view data before performing demosaicing process; And perform Xie Bao process: for the RAW data of the packing process through rearranging pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, these RAW data obtain by decoding.

In in one of this technology, perform packing process: according to the degree of correlation, rearrange as the pixel data in the RAW data performing the view data before demosaicing processes; And the RAW data processed through packing are encoded.

In the another aspect of this technology, being decoded by the coded data as encoded RAW data, these encoded RAW data are performing the view data before demosaicing processes; And perform Xie Bao process: for the RAW data of the packing process through rearranging pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, these RAW data obtain by decoding.

The useful effect of the present invention is as follows:

According to the disclosure, it is possible to image is carried out coding/decoding. Specifically, it is possible to improve coding efficiency more easily.

Accompanying drawing explanation

Fig. 1 is the schematic diagram of the configuration example for describing coding unit.

Fig. 2 be a diagram that the schematic diagram of the example of apparatus of layered picture coding apparatus of picture system.

Fig. 3 is the schematic diagram of the example for describing space scalable coding.

Fig. 4 is the schematic diagram of the example for describing time scalable coding.

Fig. 5 is the schematic diagram of the example of the scalable coding for describing signal/noise ratio.

Fig. 6 is the schematic diagram for describing packing process example.

Fig. 7 is the schematic diagram of the example for describing packing process.

Fig. 8 is the schematic diagram of the example for describing packing process.

Fig. 9 is the schematic diagram of the example for describing packing process.

Figure 10 is the schematic diagram of the example for describing packing process.

Figure 11 is the schematic diagram of the example for describing packing process.

Figure 12 is the schematic diagram of the example for describing packing process.

Figure 13 is the schematic diagram of the example for describing packing process.

Figure 14 is the schematic diagram of the example of the characteristic for describing packing process.

Figure 15 A and Figure 15 B be a diagram that the schematic diagram of the example of syntax and semantics.

Figure 16 be a diagram that the schematic diagram of the example of the grammer of CFAPSEI.

Figure 17 be a diagram that the schematic diagram of the example of the semanteme of CFAPSEI.

Figure 18 be a diagram that the schematic diagram of the example of CFAP_type.

Figure 19 A and Figure 19 B be a diagram that the schematic diagram arranging example of CFAPSEI.

Figure 20 A and Figure 20 B be a diagram that the schematic diagram arranging example of CFAPSEI.

Figure 21 A and Figure 21 B be a diagram that the schematic diagram arranging example of CFAPSEI.

Figure 22 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 23 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 24 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 25 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 26 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 27 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 28 be a diagram that the schematic diagram of the example arranging various grammer element.

Figure 29 be a diagram that the block diagram of the main configuration example of picture coding device.

Figure 30 be a diagram that the block diagram of the main configuration example of packaged unit.

Figure 31 be a diagram that the block diagram of the main configuration example of image coding unit.

Figure 32 be a diagram that the block diagram of the main configuration example of coding unit.

Figure 33 be a diagram that the block diagram of the main configuration example of base layer image coding unit.

Figure 34 be a diagram that the block diagram of the main configuration example of enhancement layer image coding unit.

Figure 35 is the schema of the example of the flow process for describing RAW image coded treatment.

Figure 36 is the schema of the example of the flow process for describing packing process.

Figure 37 is the schema of the example of the flow process for describing picture coding process.

Figure 38 is the schema of the example of the flow process for describing coded treatment.

Figure 39 is the schema of the example of the flow process for describing basal layer coded treatment.

Figure 40 is the schema of the example of the flow process for describing enhancement layer coding process.

Figure 41 be a diagram that the block diagram of the main configuration example of picture decoding apparatus.

Figure 42 be a diagram that the block diagram of the main configuration example of image decoding unit.

Figure 43 be a diagram that the block diagram of the main configuration example of decoding unit.

Figure 44 be a diagram that the block diagram of the main configuration example of base layer image decoding unit.

Figure 45 be a diagram that the block diagram of the main configuration example of enhancement layer image decoding unit.

Figure 46 be a diagram that the block diagram of the configuration example of unwrapper unit.

Figure 47 is the schema of the example of the flow process for describing RAW image decoding process.

Figure 48 is the schema of the example of the flow process for describing image decoding process.

Figure 49 is the schema of the example of the flow process for describing decoding process.

Figure 50 is the schema of the example of the flow process for describing basal layer decoding process.

Figure 51 is the schema of the example of the flow process for describing enhancement layer decoder process.

Figure 52 is the schema of the example of the flow process for describing Xie Bao process.

Figure 53 be a diagram that the schematic diagram of the example of multi-view image coding scheme.

Figure 54 be a diagram that the schematic diagram of the main configuration example of the multi-view image coding device applying this technology.

Figure 55 be a diagram that the schematic diagram of the main configuration example of the multi-view image decoding device applying this technology.

Figure 56 be a diagram that the block diagram of the main configuration example of computer.

Figure 57 be a diagram that the block diagram of the example of the illustrative configurations of TV device.

Figure 58 be a diagram that the block diagram of the example of the illustrative configurations of mobile phone device.

Figure 59 be a diagram that the block diagram of the example of the illustrative configurations of data recording/reproducing device.

Figure 60 be a diagram that the block diagram of the example of the illustrative configurations of imaging device.

Figure 61 be a diagram that the block diagram of the example of the illustrative configurations of video machine.

Figure 62 be a diagram that the block diagram of the example of the illustrative configurations of video processing device.

Figure 63 be a diagram that the block diagram of another example of the illustrative configurations of video processing device.

Embodiment

Hereafter embodiment of the present disclosure (being hereafter called embodiment) is implemented in description. Note, will be described in the following order.

1, the first embodiment (transmission of packing and package information)

2, the 2nd embodiment (picture coding device)

3, the 2nd embodiment (picture decoding apparatus)

4, the 4th embodiment (multi-view image coding/multi-view image decoding device)

5, the 5th embodiment (computer)

6, the 6th embodiment (application example)

7, the 7th embodiment (setting unit module handler)

In recent years, such as, promoting digitally processing image information to transmit efficiently and cumulative information and the device that performed the coding scheme compressed with the use of the intrinsic redundancy of graphic information by orthogonal transformation (discrete cosine transform) and motion compensation and image compressed by adopting and encode always. The example of this coding scheme comprises Motion Picture Experts Group (MPEG).

Specifically, MPEG2 (ISO/IEC13818-2) is defined as general image coding scheme, and this MPEG2 (ISO/IEC13818-2) is applicable to staggered scanning image and image two kinds of lining by line scan, and is applicable to standard-resolution image and high-definition image. Such as, at present, MPEG2 be used in for specialty human consumer and ordinary consumer diversified application in. With the use of MPEG2 compression system, the size of code (bit rate) of 4Mbps to 8Mbps is distributed to the interlaced image of the standard resolution with 720 �� 480 pixels, and the size of code (bit rate) of 18Mbps to 22Mbps is distributed to the high-resolution interlaced image with 1920 �� 1088 pixels. Therefore, it is possible to realize high compression rate and good picture quality.

MPEG2 is mainly the high quality graphic code Design of applicable broadcast, but cannot be compatible with the size of code (bit rate) lower than the MPEG1 or coding scheme with more high compression rate. Universal along with mobile terminal, be expected to increase in the future and meet the demand to this kind of coding scheme, stdn MPEG4 coding scheme. As for image encoding system, in December, 1998, ISO/IEC14496-2 standard is approved as international standard.

Further, arranging the standard (Union Internationale des Telecommunications telecommunication standardization group (ITU-T) Q6/16 Video Coding Experts group (VCEG)) being called H.26L at present, this standard is intended to be encoded by the image for video conference at first. As compared to conventional coding scheme (such as, MPEG2 with MPEG4), H.26L need bigger calculated amount when Code And Decode, but it is well known that higher coding efficiency can be realized. Further, as a part for MPEG4 activity, at present by the function H.26L do not supported is incorporated to based on function H.26L set up enhancement type compression video coding conjunctive model as the standard realizing more high coding efficiency.

According to stdn schedule, in March, 2003 this standard is approved as called after H.264 with the international standard (advanced video coding is hereafter called AVC) of MPEG-4Part10.

Further, as a kind of expansion H.264/AVC, the FRExt (fidelity range extension) of the coding instrument needed for professional purpose will be related in February, 2005, such as, RGB (4:2:2 and 4:4:4) and MPEG-2 defines 8 �� 8DCT and quantization matrix are set to standard. This is a kind of coding method showed better with the use of the film noise H.264/AVC enabling or even being contained in cinefilm, and uses in a wide variety of applications at present, such as, and Blu-ray Disc (trade mark).

But, compress with the image to about 4000 �� 2000 pixels with higher than high-definition image resolving power four times or (such as internet) distributes high-definition image when limited transmittability demand increases day by day encoding with more high compression rate. Therefore, under ITU-T, still continue the research about improving coding efficiency by VCEG.

Therefore, at present, in order to improve coding efficiency by AVC further, it is called that the stdn of the coding scheme of efficient video coding (HEVC) is undertaken by as integration and cooperation team-video coding (JCTVC) of ITU-T and the combination with standard group of ISO/International Power Technical Committee (ISO/IEC) always. Such as, as HEVC standard, issue the committee draft (see non-patent document 1) drawing up Version Spec in January, 2013.

Hereinafter, by be applied as example to describe this technology for the encoding/decoding image of efficient video coding (HEVC) system.

In advanced video coding (AVC) system, define by grand piece and son grand piece of layered structure formed. But, grand piece of 16 �� 16 pixels is not best for bigger frame (such as, ultra high-definition (UHD:4000 �� 2000 pixel) frame), and this kind of bigger frame needs to be encoded by coding method of future generation.

On the contrary, in HEVC system, as shown in Figure 1, coding unit (CU) is defined.

CU is also referred to as coding tree block (CTB), and is the part region of the image in units of picture, and this part region plays a part with grand piece in AVC system similar. Although the size of the latter is fixed as 16 �� 16 pixels, but the former size is not fixed as specific size, and specifies the former size in the compressed image information of each sequence.

Such as, in the sequential parameter collection (SPS) being contained in coded data to be output, define largest amount (maximum coding unit (LCU)) and the minimal size (minimum coding unit (SCU)) of CU.

In each LCU, split_flag=1 is arranged in the scope being not less than SCU size, such that it is able to each LCU to be divided into the CU of less size. In the example of fig. 1, the size of LCU is 128, and maximum point of layer depth is 5. When the value of split_flag is 1, the CU of 2N �� 2N size being divided into the CU of N �� N size, this size is a lower layering.

Further, further CU is divided into predicting unit (PU), this PU is the processing unit region (the part region of the image in units of picture) for infra-frame prediction or inter prediction, or CU is divided into conversion unit (TU), and this TU is the processing unit region for orthogonal transformation (the part region of the image in units of picture). At present, it is possible in HEVC system, use 16 �� 16 and 32 �� 32 orthogonal transformations and 4 �� 4 and 8 �� 8 orthogonal transformations.

When defining CU and perform the coding scheme of various types of process according to the mode of such as HEVC system in units of CU, it is possible to think grand piece in AVC system corresponding with LCU, and can think that block (sub-block) is corresponding with CU. Further, it is possible to think that the motion compensation block in AVC system is corresponding with PU. But, CU has layered structure. Such as, therefore, the size of the LCU on best result layer is usually equally big with 128 �� 128 pixels, and this size is bigger than the size of grand piece in AVC system.

Therefore, hereinafter, LCU comprises grand piece in AVC system, and CU comprises the block in AVC system (sub-block). That is, the arbitrary portion region that " block " used in the following description refers in picture, and do not limit its size, shape, characteristic etc. That is, " block " comprises arbitrary region (processing unit), such as, and TU, PU, SCU, CU, LCU, sub-block, grand piece and section. Obviously, also comprise the part region (processing unit) except above region. When needs define size, processing unit etc., will suitably be described.

Further, in this manual, coding tree unit (CTU) is coding tree block (CTB) comprising LCU (CU of maximum quantity) and the unit of the parameter when processed in the execution of LCU basic unit (level) place. Further, the coding unit (CU) being configured to CTU is the unit of parameter comprising coding block (CB) and when processing in the execution of CU basic unit (level) place.

By the way, in order to realize higher coding efficiency in AVC coding scheme and HEVC coding scheme, suitable predictive mode is selected to be very important.

The example of this kind of selective system comprises the method being arranged in H.264/MPEG-4AVC reference software, is called that united mode (JM) is (visiblehttp://iphome.hhi.de/suehring/tml/index.htm)��

In JM, it is possible to select the two kinds of mode determining method comprising high complicacy pattern and low-complexity pattern described below. By any one method in above method, calculate the cost function value for often kind of predictive mode Mode, and select to make the minimized predictive mode of cost function value as the optimal mode of block or grand piece.

The cost function under high complicacy pattern is expressed by following equation (1).

[mathematical formula 1]

Cost (Mode �� )=D+ �� * R (1)

Herein, �� represents for the generic set of block or the grand piece of candidate pattern encoded, and D represents when performing coding under predictive mode at differential power between decoded picture and input picture. �� represents as Lagrange (Lafrange) multiplier undetermined that quantization parameter function provides. R represents the total bit rate performed in this mode when encoding, and comprises orthogonal transform coefficient.

That is, in order to perform coding under high complicacy pattern, it is necessary to perform interim coded treatment under all candidate pattern once to calculate above parameter D and R. Consequently, it is desirable to bigger calculated amount.

The cost function under low-complexity pattern is expressed by following equation (2).

[mathematical formula 2]

Cost (Mode �� )=D+QP2Quant (QP) * HeaderBit (2)

Herein, D is different from the D under high complicacy pattern, and represents the differential power between predicted picture and input picture. QP2Quant (QP) represents the function of quantization parameter QP, and HeaderBit represents with the information-related size of code not comprising orthogonal transform coefficient and belongs to header, such as, and motion vector sum pattern.

That is, under low-complexity pattern, it is necessary to for each candidate pattern performs prediction processing, but do not need decoded picture. Therefore, it is not necessary to perform coded treatment. Thus, calculated amount can be less than the calculated amount under high complicacy pattern.

By the way, normal image coding scheme (such as MPEG2 and AVC) has extensibility function. Scalable coding (layering coding) makes image become multiple layer (layering) and perform the system of coding based on layer one by one. Fig. 2 be a diagram that the schematic diagram of the example of apparatus of layered picture coding apparatus of picture system.

As shown in Figure 2, in the stratification of image, based on the preset parameter with extensibility function, an image is divided into multiple layering (layer). That is, the image that stratification image (layered image) is included in multiple layerings (layer) with preset parameter, this preset parameter has value different from each other. Multiple layers of layered image are made up of basal layer and non-basic layer (also referred to as enhancement layer), coming for this basal layer performs coding/decoding when not using the image of other layer only by the image of the layer using oneself, the image with the use of other layer comes for this non-basic layer performs coding/decoding. Non-basic layer can use the image of basal layer, or can use the image of another non-basic layer.

Usually, configure non-basic layer by the data (difference data) of the difference image between the image of oneself and the image of another layer, to reduce redundancy. Such as, when an image being divided into basal layer and non-basic layer (also referred to as enhancement layer) two layerings, can only utilize the data of basal layer to obtain the quality image lower than original image, and original image (that is, high quality graphic) can be obtained by the data of the data and non-basic layer of synthesizing basal layer.

By image as described above is carried out stratification, it is possible to according to circumstances obtain the image with multiple quality easily. When not performing to turn code process, it is possible to from server transport according to the compressed image information of the ability of terminal and network. Such as, for the terminal (such as, mobile phone) with reduction process ability, only transmit the compressed image information of basal layer, and make the moving-image reproducing that there is low spatial and temporal resolution or there is low image quality. For the terminal with high throughput (such as, televisor or Personal Computer), except the compressed image information of basal layer, also transmit the compressed image information of enhancement layer, and make the moving-image reproducing that there is high-spatial and temporal resolution or there is high image quality.

In this kind of apparatus of layered picture coding apparatus of picture/layered image decoding (scalable coding/scalable decoding), the parameter with extensibility function is arbitrary. Such as, it is possible to the spatial resolution shown in Fig. 3 is used as parameter (spatial scalability). When spatial scalability, the resolving power of image is different in each layer. Namely, as shown in Figure 3, each picture layering is turned to two layerings: basal layer and the enhancement layer with the spatial resolution lower than original image, utilizes this enhancement layer, it is possible to by obtaining original image (original spatial resolution) with the Images uniting of basal layer. Obviously, the quantity of layering is an example, and can by the image layered layering turning to any amount.

Further, such as, as the parameter with extensibility, it is possible to the temporal resolution rate (time extensibility) shown in application drawing 4. When time extensibility, frame rate is different in each layer. As shown in Figure 4, that is, the layer with frame rate different from each other is turned to by image layered. The layer with high frame rate is added to the layer with lower frame rate, such that it is able to obtain the moving image with higher frame rate, and adds all layers, such that it is able to obtain original moving image (original frame rate). The quantity of layering is an example, and can by the image layered layering turning to any amount.

Further, such as, as the parameter with extensibility, it is possible to the signal to noise ratio (SNR) (SNR extensibility) shown in application drawing 5. When SNR extensibility, SN is different than in each layer. Namely, in this case, as shown in Figure 5, each picture layering is turned to two layerings: basal layer and the enhancement layer with the SNR lower than original image, utilize this enhancement layer, it is possible to by obtaining original image (original SNR) with the Images uniting of basal layer. That is, in base layer image compressed information, the image-related information transmitted with there is low PSNR, and add enhancement layer image compressed information to this information, such that it is able to reconfigure high PSNR image. Obviously, the quantity of layering is an example, and can by the image layered layering turning to any amount.

The parameter with extensibility can be another parameter. Such as, there is bit-depth extensibility, in this bit-depth extensibility, basal layer is made up of 8 bitmap pictures, and enhancement layer adds to this 8 bitmap picture, such that it is able to obtain 10 bitmap pictures.

Further, there is colourity extensibility, in this colourity extensibility, basal layer is made up of the component image of 4:2:0 form, and enhancement layer adds to this component image, such that it is able to obtain the component image of 4:2:2 form.

By the way, in recent years, used as in the RAW data performing the view data before demosaicing processes.

In many digital cameras adopt monolithic formula charge coupled device (CCD) image sensor or complementary metal oxide semiconductor (CMOS) image sensor in, each pixel has monochromatic information. Therefore, usually, when catching, perform " demosaicing " (de-mosaic) process: the undercolour information assembling the peripheral pixels from each pixel is to compensate colouring information, and creates full-colour image. Further, in many digital cameras, perform image procossing parallel with demosaicing: the hue and luminance of automatic dressing color; And such as, store with general picture format (JPEG (JPEG) or tagged image file format (TIFF)) and completed image.

But, the picture quality completing image can be produced materially affect by the tolerance range of demosaicing and automatic dressing. Further, after process (also referred to as development treatment), fixing white balance (colour temperature) etc. Therefore, cannot easily perform amendment. Further, finally store the jpeg format normally lossy compression method used, and perform to subtract thin colouring information in the horizontal direction. Therefore, in principle, compared with raw data, picture quality may worsen. Further, the color depth of form mostly just has 8 (24) for often kind of color totally. Therefore, have no option except significantly abandoning the information received from the image sensor with 12 tolerance range, and exposure (bright and the dark and brightness of image) adjustment after such capture may become difficulty.

As described above, in order to suppress deterioration of image quality, maintain degree of freedom etc. when adjusting the various parameter relevant with picture quality, the former data before demosaicing have been employed, that is, RAW data. Such as, installed in digital camera etc. and store the RAW data obtained by imaging and allow user etc. that RAW data freely perform the function of image procossing after a while.

But, compared with jpeg data etc., the size of data of RAW data is relatively big, and is storing and having shortcoming in transmission. Specifically, in recent years, size of data continues to increase due to the increase of pixel quantity, and processes (storage, transmission etc.) RAW data and become more difficult.

Therefore, the demand of compression RAW data is constantly increased, and constantly needs higher compression method. View data can be encoded by above-mentioned HEVC efficiently. But, owing to RAW data obtain from image sensor etc., so the color (band) of the information comprised at each pixel data depends on the array of the wave filter arranging image sensor etc. in each pixel.

Usually, wave filter is arranged, thus make whole pixel region can become even. Therefore, when being furnished with the wave filter of multiple color (multiple band), the wave filter making color (band) different from each other arranges adjacent to each other. As a result, in the RAW data obtained from this kind of image sensor etc., the pixel with low correlation is adjacent.

Image encoding system (such as HEVC), on the direction, space or time orientation of image, with the use of dependency, carrys out compressed information amount. Therefore, even if image encoding system simply is applied to the coding of RAW data, in these RAW data, the pixel with low correlation is adjacent, also cannot perform coding easily, efficiently.

Therefore, perform packing process according to the degree of correlation in coding side: rearrange as each pixel data in the RAW data performing the view data before demosaicing processes; And the RAW data processed through packing are encoded. That is, when RAW image data being encoded, as process (pre-treatment) before the coding, packing process is performed according to the degree of correlation: the pixel data rearranging RAW data.

By performing packing process before the coding, it is possible to the image that highly relevant pixel is assembled is encoded. Therefore, can by the method (do not need RAW data are carried out special processing) similar with the normal method after demosaicing process, view data encoded, being encoded by RAW data efficient, this is similar with the situation of the normal image data after demosaicing process. Namely, it is possible to improve coding efficiency more easily.

Such as, packing process can to have the pixel data being separated RAW data based on the data of high correlation one by one, rearrange the pixel data group to have separation based on the data of high correlation one by one, and generate with packing process relevant package information for Xie Bao process, this Xie Bao process processes contrary process.

Like this, it is possible to rearrange the pixel data of RAW data according to the degree of correlation, and can come with the use of package information easily to recover original RAW data.

It is arbitrary to have the method for the pixel data being separated RAW data based on the data of high correlation one by one. Such as, it is possible to check the dependency of pixel data, and separate pixel data can be carried out based on check result. Further, such as, it is possible to use the information provided in advance. Such as, it is possible to carry out separate pixel data based on individual element, this pixel is assigned the wave filter of identical type. Such as, when being distributed by color filter, it is possible to carry out separate pixel data based on individual element, this pixel is assigned the wave filter of same color. Such as, when RGB color wave filter is distributed, it is possible to pixel data is separated into three groups (pixel data group): be assigned the pixel data of the pixel of redness (R) wave filter, the pixel data of the pixel of (G) wave filter that is assigned green and be assigned the pixel data of the pixel of blueness (B) wave filter.

Obviously, the color of color filter and the quantity of color are arbitrary. Further, wave filter can comprise the wave filter of light in addition to visible light, such as, and infrared filter. Such as, it is possible to comprise white pixel (W), this white pixel (W) is the pixel (or unallocated pixel having wave filter) being assigned the wave filter allowing all bands of transmission. Further, such as, it is possible to comprise the pixel for detecting depth data (Depth). Further, such as, it is possible to wave filter and infrared filter to being made up of RGB distribute. In this case, such as, it is possible to pixel data is separated into four groups (pixel data group): be assigned the pixel data of the pixel of redness (R) wave filter, the pixel data of the pixel of (G) wave filter that is assigned green, the pixel data of the pixel of (B) wave filter that is assigned blueness and be assigned the pixel data of the pixel of infrared filter.

Further, when using the information provided in advance, it is possible to perform separation based on information in addition to the filter. Such as, it is possible to be separated a part or all pixels based on information in addition to the filter.

As described above, with the use of the information of the wave filter distributed, the degree of correlation checked in pixel data becomes unnecessary. Therefore, it is possible to perform the separation of pixel data more easily.

It is arbitrary for rearranging the method to have the pixel data group of separation based on the data of high correlation one by one.

Such as, as shown in Figure 6, it is possible to rearrange the component of pixel data group as predetermined color space. In the case of fig. 6, as shown in the left diagram, the RGB color wave filter in Bayer (Bayer) array is distributed to RAW data. RAW data are performed the pre-treatment of packing process as coding, and is the pixel data group being made up of corresponding colored pixels by RAW data separating, as shown at right. Then, it is possible to rearrange the respective component (planes of color) of pixel data group as YCbCr color space. Specifically, it is associated with the brightness (Y) of YCbCr component signal by green (G) the pixel data group shown in white square, it is associated with the colourity (Cb) of YCbCr component signal by having blueness (B) the pixel data group shown in the square of the oblique line from upper right side to lower left, and it is associated with the colourity (Cr) of YCbCr component signal from lower right to redness (R) the pixel data group shown in upper left cornerwise square by having. Noting, the color space being furnished with pixel data group is arbitrary, and can adopt another color space except YCbCr, such as, and YUV.

Such as, as described above, with predetermined color form (4:2:0,4:2:2 or 4:4:4), the RAW data being separated into planes of color are encoded. Under this kind of color format, according to the ratio of pixel quantity with planes of color, RAW data are encoded. Such as, in the case of fig. 6, owing to color filter is Bayer (Bayer) array, so the pixel quantity of green (G) pixel data group is red (R) pixel data group and the twice of blue (B) pixel data group. Therefore, color format is 4:2:0.

Note, it is possible to planes of color (component) is encoded independently of one another, or planes of color can be encoded jointly (encoding together).

Further, it is possible to each part region of the image of RAW data is performed above coding, such as the example of Fig. 7. In the case of fig. 7, for each predetermined portion region, such as, tile or section, pixel data group is encoded, in this pixel data group, is processed by packing, being separated based on color one by one and rearrange the RAW data being assigned the RGB color wave filter of (Bayer) array in Bayer, the situation of this and Fig. 6 is similar. Like this, it is possible to each part region (tile or section) is performed decoding.

Further, such as, as shown in Figure 8, it is possible to rearrange the different from each other part region of pixel data group as an image. In the case of fig. 8, processing spatially separation by packing and rearrange the RAW data being assigned the RGB color wave filter of (Bayer) array in Bayer, the situation of this and Fig. 6 is similar. Such as, such as, the pixel data group of separation is rearranged in the part region different from each other (tile or section) of an image.

Such as, in this case, RAW data are encoded, as an image (only brightness (Y)) of a component. That is, all pixel data groups are considered as brightness component, and with 4:0:0 form, RAW data are encoded. Further, now, for each the part region (tile or section) for separating of pixel data group described above, RAW data are encoded.

Further, such as, as shown in Figure 9, it is possible to rearrange the data of pixel data group as the layering different from each other (layer) of stratification view data. When Fig. 9, processed by packing, to have based on the data of high correlation spatially separation one by one and rearrange RAW data, the situation of this and Fig. 6 is similar. Such as, the pixel data group of separation is rearranged in the layering different from each other (layer 0 to layer 3) of an image.

The data of layering (layer) can be encoded independently of one another, or the data of layering (layer) can be encoded (encoding together) jointly. Such as, when there is dependency between layering (pixel data group), perform coding with the use of inter-layer prediction, such that it is able to improve coding efficiency. Further, such as, when there is not dependency between layering (pixel data group), such as degree of depth information, the data of layering (layer) are encoded independently of one another, such that it is able to perform decoding based on layering (layer).

Further, such as, as shown in Figure 10, it is possible to the view data processed through demosaicing is rearranged in part layering. In other words, it is possible to the view data processed through demosaicing is arranged in part layering. In case of fig. 10, being processed by packing, be separated the RAW data being assigned the RGB color wave filter of (Bayer) array in Bayer based on color one by one, the situation of this and Fig. 6 is similar; And spatially separation and rearrange the pixel data group of respective color. Such as, the pixel data group of separation is rearranged in the layering different from each other (layer 1 to layer 3) of an image. Then, in predetermined layering (layer 0), placement of images data, this view data is that the RAW data after development are (through demosaicing process and image procossing, such as, the finishing of the hue and luminance of color) (or the data (thumbnail image data) of thumbnail image of downscaled images as view data).

Similar with the situation of Fig. 9, it is possible to the data of layering (layer) are encoded independently of one another, or the data of layering (layer) can be encoded (encoding together) jointly.

Further, such as, as shown in figure 11, it is possible to rearrange the different from each other part region of pixel data group as multiple image. In case of fig. 11, carrying out packing process to by the RAW data being assigned the RGB color wave filter of (Bayer) array in Bayer, the situation of this and Fig. 6 is similar. Such as, now, as shown in figure 11, or even be assigned the pixel (green (G of identical green (G) wave filter₀) pixel and green (G₁) pixel) it is in phase place (spatially deviate from a pixel in the lateral direction) different from each other. Thus it is possible to infra-frame prediction etc. can not correctly be performed. In case of fig. 11, therefore, by green (G₀) pixel and green (G₁) pixel separation is pixel data group different from each other, and rearranges.

In the example of fig. 11, by green (G₀) pixel data group and green (G₁) pixel data group rearranges in image (frame) different from each other. Such as, by green (G₀) pixel data group is re-arranged to frame 0, and by green (G₁) pixel data group is re-arranged to frame 1.

Note, it is possible to redness (R) pixel data group and blue (B) pixel data group are re-arranged to any frame (frame 0 or frame 1) respectively. Now, it is possible to redness (R) pixel data group and blue (B) pixel data group are rearranged into green (G₀) pixel data group and green (G₁) pixel data group distinct portions region (tile or section), or red (R) pixel data group and blue (B) pixel data group can be rearranged into green (G₀) pixel data group and green (G₁) layering (layer) that pixel data group is different.

As described above, based on image one by one (frame), the pixel data group being again arranged in multiple image (frame) is encoded.

Note, it is possible to by green (G₀) pixel data group and green (G₁) pixel data group rearranges in the part region different from each other (tile or section) of identical image, as described by with reference to figure 8. In this case, by four pixel data groups: green (G₀) pixel data group, green (G₁) pixel data group, redness (R) pixel data group and blueness (B) pixel data group rearrange in part region (tile or section) different from each other. With the coding performing this kind of situation with reference to the situation described by figure 8 similarly.

Further, it is possible to by green (G₀) pixel data group and green (G₁) pixel data group rearranges in the layering different from each other (layer) of identical image, as described by with reference to figure 9 and Figure 10. In this case, by four pixel data groups: green (G₀) pixel data group, green (G₁) pixel data group, redness (R) pixel data group and blueness (B) pixel data group rearrange in layering (layer) different from each other. With the coding performing this kind of situation with reference to the situation described by figure 9 and Figure 10 similarly.

Note, green (G₀) pixel and green (G₁) pixel has high correlation. Therefore, when by green (G₀) pixel data group and green (G₁) pixel data group is when rearranging in image (frame or layer) different from each other, as shown in the example of fig. 11, these pixel datas (frame or layer) are applied inter prediction, as shown in the illustration in fig 12, and coding efficiency can be improved. Note, as described above, green (G₀) pixel and green (G₁) pixel is in locus different from each other (position be deviation). This kind of deviation can be adjusted as pre-treatment, or, in inter prediction, it is possible to adjust this kind of deviation with the use of motion vector.

Further, (random) that the position of the pixel of the pixel data group of separation as described above is normally discrete. Thus it is possible to the process of block elimination filtering when can not correctly operate in coding, sample self-adaptation skew (SAO) etc. Therefore, when encoding, it is possible to forbid that (not performing) carries out filtering process in the ring of process described above by control. Such as, when performing packing described above in pre-treatment and process, it is possible to the value of the grammer element relevant with filtering process in ring is set to the value not performing filtering process in ring concentrated in parameter.

Illustrate the characteristic of packaging method in fig. 14. Such as, when rearranging component (pixel data group) to be separated into planes of color, and when planes of color being encoded together, parameter becomes general, and thus, cannot perform decoding independent of component. Further, in this case, unless rearranged multiple picture or section, otherwise the quantity limitation of the component that can process is three or less. Under this approach, the demoder in the specification sheets of the version 1 of HEVC standard can process decoding. Further, when planes of color being encoded independently of one another, the quantity limitation of the component that can process is three or less.

When performing the method for coding with 4:0:0 form, with the use of the part region of tile or section, it is possible to the component of process any amount. Such as, when rearranging the method for multiple layering (layer), it is possible to the component of process any amount, as long as this quantity is the layering of limited quantity or less (64 layerings or less). Further, by performing inter prediction when encoding, it is possible to improve coding efficiency with the use of the dependency between component. Whether improve coding efficiency by execution inter prediction and depend on multiple condition, such as, the degree of correlation etc. between component. Such as, when with the degree of correlation of another component lower time, such as depth data, it may be necessary to carry out independent coding.

Further, such as, by being arranged on basal layer by the image (or thumbnail image of image) after development treatment, the image (or thumbnail image of image) after development treatment can be decoded by the demoder in the specification sheets of the version 1 of HEVC standard.

As the pre-treatment of decoding, perform above packing process, therefore, when RAW data being encoded, it is possible to improve coding efficiency more easily.

Noting, the content of packing process can be just any content, and is not limited to content described above, as long as packing process rearranges as in the RAW data performing the view data before demosaicing processes according to the degree of correlation.

If decoded by the coded data that the RAW data after performing packing and processing encode and obtain, the RAW data after packing processes so can be obtained. That is, in order to recover original RAW data, after the decoding process, it is necessary to performing Xie Bao process further, this Xie Bao process processes contrary process.

In order to carry out the packing processing execution Xie Bao process to executed by correct method, it is necessary to provide package information to decoding side, this package information processes relevant information with packing and the generation when packing processes. Certainly, it is possible to omit package information offer, and can by preordering method perform packing process conciliate bag process. But, treatment process is limited. Therefore, the specification of coding side and decoding side is limited, and the versatility of system reduces. By transmitting package information described above, it is possible to suppress the reduction of the degree of freedom in the content of packing process (Xie Bao process), and the reduction of the versatility of system can be suppressed. , it is possible to by the demoder of more kinds of specification, namely the coded data of the encoder encodes by more kinds of specification correctly is decoded.

The method providing package information is arbitrary. Such as, it is provided that package information flows different data as from the position of coded data.

Further, it is provided that the package information being included in the position stream of coded data. Such as, in coding side, it is possible to arrange Additional Information, this Additional Information comprises and processes relevant package information with packing, and this packing process is: according to the degree of correlation, rearranges the pixel data of the RAW data as the view data before processing at demosaicing; The RAW data processed through packing can be encoded; And the position stream comprising the coded data of acquisition and the Additional Information of setting can be generated.

Like this, it is not necessary to be provided for providing to come respectively the method for the package information of self-alignment stream. Therefore, the transmission of package information becomes to be more prone to. Further, it is possible to information is put into together a position stream, and thus, the management of the data transmitted becomes easy. That is, decode side and can obtain package information (content of packing process can be grasped) more easily, and coded data correctly can be decoded (comprising Xie Bao process). That is, when RAW data are encoded, it is possible to improve coding efficiency more easily.

Method package information comprised in stream in place needs to meet coding method. Like this, it is possible to decode by meeting the demoder contraposition stream of the specification of coding method, and Xie Bao process can be performed. Namely, it is possible to contraposition stream carries out decoding (comprising Xie Bao process) more easily and correctly. That is, when RAW data are encoded, it is possible to improve coding efficiency more easily.

Such as, it is possible to carry out parameters collection according to the packing of RAW data process. Such as, it is possible to whether the content arranging instruction position stream in video parameter collection (VPS), sequential parameter collection (SPS) etc. is the mark (general_camera_sensor_image_packed_flag) of RAW data.

Figure 15 A illustrates the example of its grammer. Further, Figure 15 B illustrates the example of its semanteme. Whether general_camera_sensor_image_packed_flag is the content of instruction position stream is the label information of the coded data (that is, RAW data) of the data of camera sensor image (RAW image).

Then, as the Additional Information comprising package information, it is provided that supplemental enhancement information (SEI). Such as, as the Additional Information comprising package information, it is possible to arrange color filter array package information SEI (CFAPSEI).

In this case, such as, when the value of general_camera_sensor_image_packed_flag is " 1 ", it is possible to instruction exists CFAPSEI, and when the value of general_camera_sensor_image_packed_flag is " 0 ", it is possible to instruction does not exist CFAPSEI. Noting, when demoder cannot process CFAPSEI, demoder can only ignore the value of general_camera_sensor_image_packed_flag.

Figure 16 illustrates the example of the grammer of CFAPSEI. Figure 17 illustrates semantic example. As shown in figure 16, in CFAPSEI, it is provided with grammer element, such as, CFAP_id, CFAP_type, CFAP_component_num, CFAP_component_id [i], CFAP_mapping_info [i], CFAP_bitstream_constraint_indication_flag and CFAP_no_inloop_filter_used_flag.

As shown in figure 17, CFAP_id is the ID for distinguishing CFAP_SEI. Arbitrary value can be set. When multiple CFAP_SEI being set to a position and flows, CFAP_id is important. CFAP_type is the information of the type of instruction packing process (rearranging the method for pixel data when packing processes). Figure 18 illustrates the example of the value of CFAP_type. When Figure 18, component is separated into the method for planes of color by CFAP_type=0 instruction, such as the example of Fig. 6. Further, component is re-arranged in section and performs the method for coding with YUV400 form by CFAP_type=1 instruction, such as the example of Fig. 8. Further, component is re-arranged in tile and performs the method for coding with YUV400 form by CFAP_type=2 instruction, such as the example of Fig. 8. Further, CFAP_type=3 indicates the method being re-arranged in layer by component, such as the example of Fig. 9.

Noting, the value of CFAP_type is arbitrary. For often kind of packaging method, it is possible to the value except the example shown in Figure 18 is distributed. Such as, it is possible to the value except numeral is distributed. Further, it is possible to value is distributed to the CFAP_type about the packaging method except the packaging method shown in Figure 18. Note, can pass through table information in CFAPSEI, parameter collection etc., this table information being defined previously as coding standard, or this table information can transfer to decoding side, this table information is used for distributing to packaging method the value of the CFAP_type shown in Figure 18. In this case, it is possible to any packetization types is added in table.

CFAP_component_num is the information of the quantity of the component indicating the RAW data through packing process. Value by 1 or bigger is set to CFAP_component_num. Such as, when color filter (Bayer (Bayer) wave filter) in Bayer (Bayer) array, there are RGB tri-kinds of colors, and thus the value of CFAP_component_num is set to " 3 ". Further, such as, when color filter (Bayer (Bayer) wave filter) in Bayer (Bayer) array, and according to space phase, green (G) is divided into green (G₀) and green (G₁) when, component is treated to RG₀G₁B tetra-components, and thus the value of CFAP_component_num is set to " 4 ".

CFAP_component_id [i] is the identifying information of the component indicating the RAW data through packing process. This value arbitrary value can be set to its value, as long as can identify component.

CFAP_mapping_info [i] is the information of the configuration indicating the RAW data through packing process. This value uses in conjunction with CFAP_type. Details will be provided below.

CFAP_bitstream_constraint_indication_flag is the information whether instruction encoder performs the position stream constraint of any type when encoding. On duty when being " 1 ", this value instruction encoder has performed the position stream constraint of any type, and has transmitted specific Additional Information. Further, on duty when being " 0 ", there is not position stream constraint in the instruction of this value.

CFAP_no_inloop_filter_used_flag is the information whether instruction encoder employs wave filter in ring when encoding. Such as, on duty when being " 1 ", the instruction of this value does not have to use wave filter in ring (block elimination filtering or SAO) in stream in place. Further, on duty when being " 0 ", there is not this kind of constraint in the instruction of this value. When not transmitting this CFAP_no_inloop_filter_used_flag, decoding side thinks that its value is for " 0 ".

Noting, in CFAPSEI, the package information of definition can be any information, as long as this information is relevant with packing process, and is not limited to the example of grammer element described above.

Figure 19 A and Figure 19 B be a diagram that the schematic diagram arranging example of CFAPSEI. In the example of Figure 19 A and Figure 19 B, as shown in Figure 19 A, the value of CFAP_type is set to " 1 ", the value of CFAP_component_num is set to " 3 ", the value of CFAP_mapping_info [0] is set to " 0 ", the value of CFAP_mapping_info [1] is set to " 1 ", and the value of CFAP_mapping_info [2] is set to " 2 ". In this case, as shown in Figure 19 B, component is split by section segmentation. Then, the component that component is numbered 0 is (such as, G) it is stored in the section that slice number is 0, the component that component is numbered 1 is (such as, R) it is stored in the section that slice number is 1, such as, and the component (B) that component is numbered 2 is stored in the section that slice number is 2. Then, perform coding with 4:0:0 form. This can also grasp by the value of the chroma_format_idc in stream in place is set to " 0 ". But, with reference to the value (" 1 ") of the CFAP_type in metadata (CFAPSEI), it is possible to grasp this kind of situation more easily.

Figure 20 A and Figure 20 B be a diagram that another schematic diagram arranging example of CFAPSEI. In the example of Figure 20 A and Figure 20 B, as shown in FIG. 20 A, the value of CFAP_type is set to " 3 ", the value of CFAP_component_num is set to " 3 ", the value of CFAP_mapping_info [0] is set to " 1 ", the value of CFAP_mapping_info [1] is set to " 2 ", and the value of CFAP_mapping_info [2] is set to " 3 ". In this case, as shown in fig. 20b, component is divided into layer. Then, the component that component is numbered 0 is (such as, G) being stored in layer is numbered in the layer of 1, the component that component is numbered 1 is (such as, R) being stored in layer is numbered in the layer of 2, such as, and the component (B) that component is numbered 2 is stored in layer and is numbered in the layer of 3. Note, it is not numbered the Layer assignment component numbering of 0 to layer. This instruction is numbered in the layer of 0 at layer to store normal YUV image (after development treatment).

Figure 21 A and Figure 21 B be a diagram that CFAPSEI's one arranges the schematic diagram of example again. In the example of Figure 21 A and Figure 21 B, component is re-arranged in two frames. Therefore, it is provided with two CFAP_SEI. As illustrated in fig. 21, in the CFAP_SEI relevant with the first frame, the value of CFAP_id is set to " 0 ", the value of CFAP_type is set to " 2 ", the value of CFAP_component_num is set to " 2 ", the value of CFAP_component_id [0] is set to " 0 ", the value of CFAP_mapping_info [0] is set to " 0 ", the value of CFAP_component_id [1] is set to " 2 ", and the value of CFAP_mapping_info [1] is set to " 1 ".

Further, in the CFAP_SEI relevant with the 2nd frame, the value of CFAP_id is set to " 1 ", the value of CFAP_type is set to " 2 ", the value of CFAP_component_num is set to " 2 ", the value of CFAP_component_id [0] is set to " 1 ", the value of CFAP_mapping_info [0] is set to " 0 ", the value of CFAP_component_id [1] is set to " 3 ", and the value of CFAP_mapping_info [1] is set to " 1 ".

Such as, in this case, as illustrated in fig. 21b, component is numbered the component (G of 0₀) be stored in the tile that the tile number of frame that frame is numbered 0 is 0, and such as, the component (R) that component is numbered 2 is stored in the tile that tile number is 1. Such as, further, component is numbered the component (G of 1₁) be stored in the tile that the tile number of frame that frame is numbered 1 is 0, and such as, the component (B) that component is numbered 3 is stored in the tile that tile number is 1.

By arranging the value of the grammer element of CFAP_SEI, it is possible to express various packaging method. That is, with reference to CFAP_SEI, decoding side can perform Xie Bao process more easily and correctly.

Further, it is possible to the various grammer elements of various parameter collection are suitably set according to the content of various types of packing described above process.

What Figure 22 illustrated the various grammer elements of when component being separated into planes of color and jointly is encoded by YC (example of Fig. 6) arranges example. Further, what Figure 23 illustrated the various grammer elements of when component being separated into planes of color and independently is encoded by YC (example of Fig. 6) arranges example. Noting, Figure 23 illustrate only the part with the setting different from the example shown in Figure 22. Further, Figure 24 illustrate component is separated into planes of color and for each part region, such as, section or tile perform coding when (example of Fig. 7) various grammer elements example is set. Noting, Figure 24 illustrate only the part with the setting different from the example shown in Figure 22. As of fig. 24, arranging part grammer element and concentrate at image parameters and there occurs change, and the quantity of passage subregion provides slice header, therefore, it is possible to easily process the coding/decoding in this kind of part region to each.

Figure 25 illustrate such as, component is separated into part region (section of brightness (Y) or tile) and with 4:0:0 form perform coding when (example of Fig. 8) various grammer elements example is set. Noting, Figure 25 illustrate only the part with the setting different from the example shown in Figure 22. Further, what Figure 26 illustrated the various grammer elements of when component is separated into layering (layer) (example of Fig. 9) arranges example. Noting, Figure 26 illustrate only the part with the setting different from the example shown in Figure 22. Further, Figure 27 illustrates and component is being separated into layering (layer) and the various grammer elements of (example of Figure 10) when being stored in basal layer through the thumbnail image of development treatment are arranged example. Noting, Figure 27 illustrate only the part with the setting different from the example shown in Figure 22 and 26.

What Figure 28 illustrated the various grammer elements of when forbidding wave filter in ring (example of Figure 13) arranges example.

Content according to packing process suitably arranges the value of the various grammer elements of various parameter collection, as described above, thus the RAW data processed through packing can suitably be encoded by encoder. Further, reference parameter sets, coded data can be decoded by demoder more easily and correctly. That is, when RAW data are encoded, it is possible to improve coding efficiency more easily.

Further, in order to corresponding with the process of coding side, to being, the coded data as in the encoded RAW data performing the view data before demosaicing processes decodes in decoding side, and performs Xie Bao process: is returned to by the pixel data of RAW data that are that obtain by decoding and that process through rearranging the packing of pixel data according to the degree of correlation and is performing the layout before packing processes.

Further, in Xie Bao processes, can resolve processing relevant package information with packing, the pixel data that can be separated the RAW data through packing process based on the package information that parsing obtains, and the pixel data to have separation based on the data of high correlation one by one can be rearranged to return to the layout before execution packing process based on resolving the package information obtained.

Now, it is possible to be separated to be assigned, by packing process, the pixel data rearranged based on the pixel of the wave filter of identical type one by one, and pixel data can be rearranged according to the array of wave filter. Further, it is possible to be separated to be assigned the pixel data arranged based on the pixel of the color filter of same color one by one by packing process, and pixel data can be rearranged according to the array of color filter. Further, it is possible to the pixel data of separation to have the component being re-arranged to predetermined color space based on the data of high correlation one by one. Further, it is possible to separation is to have the pixel data rearranged based on the data of high correlation in the part region different from each other of one or more image one by one. Further, it is possible to separation is to have the pixel data rearranged based on the data of high correlation in the layering different from each other of stratification view data one by one. Further, it is possible to separation rearranges at the pixel data except being furnished with in other layering except a part of layering of the view data of demosaicing process.

Further, when decoding, it is possible to for each part region, the coded data of the encoded component in each the part region as the image for RAW data is decoded.

When doing as described above, can by the method similar with the normal circumstances that the coded data as the coded picture data of (not needing RAW data are carried out special processing) after demosaicing process is decoded, correctly decode with the coded data of the RAW data assembling the pixel with high correlation through packing process, and it is carried out Xie Bao process. Namely, it is possible to by easily and correctly to perform decoding (comprising Xie Bao process) corresponding to the method for the coding method being associated with packing process. Namely, it is possible to improve coding efficiency more easily.

Further, in order to corresponding with the process of coding side, decoding side to comprising in stream in place and is decode as the coded data in the encoded RAW data performing the view data before demosaicing processes, and package information can be extracted from the Additional Information that comprises stream in place, this package information be process relevant information with packing and for Xie Bao process: by through packing process and performing the layout before packing process by being returned to by the pixel data of the RAW data that decoded data decodes and obtain.

As package information, it is possible to extract the information that instruction rearranges the method for pixel data when packing processes. Further, it is possible to extract the information of instruction through the quantity of the component of the RAW data of packing process. Further, it is possible to extract the identifying information of the component of the RAW data through packing process. Further, it is possible to extract the information of instruction through the configuration of the RAW data of packing process. Further, it is possible to extract the information indicating whether to perform position stream constraint when generating coded data. Further, it is possible to extract the information indicating whether to employ wave filter in ring when generating coded data.

Note, it is possible to extract package information from the supplemental enhancement information (SEI) comprised stream in place. Further, it is possible to the use of the package information extracted, to the RAW data execution Xie Bao process obtained by coded data is decoded, process through packing.

Package information is extracted flowing from position, and for Xie Bao process, as described above, therefore, demoder can decode more easily and suitably to the coded data as the encoded RAW data processed through packing, and can perform Xie Bao process. That is, when RAW data are encoded, it is possible to improve coding efficiency more easily.

Next, the device and the method thereof that achieve described above technology will be described. Picture coding device 100 shown in Figure 29 is the device RAW data as the view data before performing demosaicing process encoded more easily and efficiently. As shown in figure 29, picture coding device 100 comprises camera sensor 101, packaged unit 102, image coding unit 103 and transmission unit 104.

Camera sensor 101 makes this image objects to generate RAW data by the light from the object being incident on pixel region carries out opto-electronic conversion, and these RAW data are supplied to packaged unit 102. Note, it is possible to adopt any processing unit except camera sensor 101, as long as this processing unit can obtain RAW data and these RAW data are supplied to packaged unit 102. Such as, it is possible to the input unit etc. obtaining RAW data from the external world is provided in the position of camera sensor 101.

The RAW data provided are performed the packing process rearranging pixel data according to the degree of correlation by packaged unit 102, as described in the first embodiment. Further, packaged unit 102 generates and processes relevant package information with packing. The RAW data processed through packing and package information are supplied to image coding unit 103 by packaged unit 102.

The RAW data after packing processes provided, by the method substantially similar with situation about the view data after demosaicing process being encoded, are encoded by image coding unit 103. Further, image coding unit 103 based on the package information parameters collection obtained and generates Additional Information. Image coding unit 103 generates position stream, and this stream comprises the coded data of acquisition and comprises the Additional Information of package information. The position stream of acquisition is supplied to transmission unit 104 by image coding unit 103.

Transmission unit 104 by recording medium, transmission line etc. by the bit stream of offer to decoding side. That is, unit 104 is transmitted by bitstream recording in the recording medium and by bit stream to transmission line, such as, network.

Noting, packaged unit 102 and image coding unit 103 can be configured to a processing unit (RAW data encoding unit 105). Such as, that is, this RAW data encoding unit 105 (packaged unit 102 and image coding unit 103) can be configured to an image processing equipment.

Figure 30 be a diagram that the block diagram of the main configuration example of the packaged unit 102 of Figure 29. As shown in figure 30, packaged unit 102 comprises separating unit 111, rearranges unit 112 and package information generation unit 113.

Such as, separating unit 111 passes through such as method described in the first embodiment, to have the pixel data of the RAW data being separated offer based on the data of high correlation one by one. Separating unit 111 rearranges unit 112 by have based on the data of high correlation the pixel data group of separation one by one and be supplied to.

Such as, the pixel data group that unit 112 rearranges offer by such as described in the first embodiment method is rearranged. Rearrange unit 112 and the RAW data with the pixel data group rearranged are supplied to package information generation unit 113.

Such as, package information generates unit 113 and generates package information, and this package information is relevant with the packing process RAW data provided performed, as described in the first embodiment. Package information generates unit 113 and the package information of the RAW data after packing processes provided and generation is supplied to image coding unit 103. (Figure 29)

Figure 31 be a diagram that the block diagram of the main configuration example of the image coding unit 103 of Figure 29. As shown in figure 31, image coding unit 102 comprises setting unit 121 and coding unit 122.

Setting unit 121 comes parameters collection and Additional Information (such as based on the package information provided, CFAP_SEI), such as, this parameter collection and Additional Information (CFAP_SEI) are to be added to the coded data of the RAW data after packing processes provided. Setting unit 121 by arrange parameter collection and Additional Information (such as, CFAP_SEI), together with provide package information and packing process after RAW data, it is provided that to coding unit 122.

Coding unit 122 based on provide parameter collection and Additional Information (such as, CFAP_SEI), by the method substantially similar with the situation that the view data after development treatment is encoded, the RAW data after packing processes provided are encoded. Coding unit 122 generates position stream (also referred to as encoded stream) comprising coded data, the parameter collection of offer and the Additional Information etc. that obtain by encoding, and position stream is supplied to transmission unit 104 (Figure 29).

Figure 32 be a diagram that the block diagram of the main configuration example of the coding unit 122 of Figure 31. The coding method of coding unit 122 is arbitrary. Such as, the RAW data after processing in packing are performed apparatus of layered picture coding apparatus of picture (scalable coding) by coding unit 122. In Figure 32, situation will be described below: the RAW data after packing processes are made up of basal layer and enhancement layer two layerings, and RAW data are performed apparatus of layered picture coding apparatus of picture (scalable coding) by coding unit 122. As shown in figure 32, coding unit 122 comprises base layer image coding unit 131, enhancement layer image coding unit 132, multiplexed device 133 and control unit 134.

The data (also referred to as base layer image) of the basal layer of the RAW data after packing processes it are provided in base layer image coding unit 131. The data of basal layer are encoded to generate the encoded stream of base layer image as the position stream comprising coded data by base layer image coding unit 131.

The data (also referred to as enhancement layer image) of the enhancement layer of the RAW data after packing processes it are provided in enhancement layer image coding unit 132. The data of enhancement layer are encoded to generate the encoded stream of enhancement layer image as the position stream comprising coded data by enhancement layer image coding unit 132.

The encoded stream of enhancement layer image of the encoded stream of base layer image generated in base layer image coding unit 131 and generation in enhancement layer image coding unit 132 is carried out multiplexed to generate the encoded stream of layered image by multiplexed device 133, and the encoded stream of this layered image is the position stream of the coded data of the corresponding layering comprising RAW data. Such as, further, multiplexed device 133 stores the parameter collection from control unit 134 to the encoded stream of layered image and Additional Information (CFAP_SEI) that provide from. The encoded flow transmission of layered image extremely transmission unit 104 (Figure 29) that multiplexed device 133 will generate.

Control unit 134 is based on the parameter collection provided from setting unit 121 and Additional Information (such as, CFAP_SEI), by control base layer image coding unit 131 and enhancement layer image coding unit 132, control the coding to corresponding layer (Figure 31). Such as, further, parameter collection and Additional Information (CFAP_SEI) are supplied to multiplexed device 133 by control unit 134.

Figure 33 be a diagram that the block diagram of the main configuration example of the base layer image coding unit 131 of Figure 32. As shown in figure 33, base layer image coding unit 131 comprise screen rearrange buffer memory 142, calculate unit 143, orthogonal transform unit 144, quantifying unit 145, lossless encoding unit 146, accumulation buffer memory 147, inverse quantization unit 148 and anti-quadrature conversion unit 149. Further, base layer image coding unit 131 comprises calculating unit 150, loop filter 151, frame memory 152, selection unit 153, intraprediction unit 154, inter prediction unit 155, predicted picture selection unit 156 and Rate control unit 157. The processing unit (screen rearranges buffer memory 142 to Rate control unit 157) of base layer image coding unit 131 controls (Figure 32) by control unit 134 and is operated.

Screen rearranges the frame of the view data (base layer image information) of sequence store input digital data of buffer memory 142 to show. Further, screen rearranges buffer memory 142 and is re-arranged in frame with the order encoded by the image of the frame of the sequence store having to show according to picture group (GOP), and is supplied to by the image of the frame of the order having to rearrange and calculates unit 143. Further, screen rearranges buffer memory 142 and the image of the frame of the order having to rearrange also is supplied to intraprediction unit 154 and inter prediction unit 155.

Calculate unit 143 to rearrange the image that buffer memory 142 reads from screen the predicted picture provided from intraprediction unit 154 and inter prediction unit 155 by predicted picture selection unit 156 is provided. Such as, when performing in frame the image of coding, calculate unit 143 and rearrange the image that buffer memory 142 reads from screen the predicted picture provided from intraprediction unit 154 is provided. Further, such as, when performing the image of interframe encode, calculate unit 143 from screen rearrange buffer memory 142 read image subtract from inter prediction unit 155 provide predicted picture. Calculate unit 143 and subtract each other result (difference information) to orthogonal transform unit 144 output.

Such as, orthogonal transformation (discrete cosine transform and Karhunen-Loeve conversion) is applied to from the difference information calculating unit 143 offer by orthogonal transform unit 144. Orthogonal transform unit 144 vectorization unit 145 provides its transformation coeffcient.

The quantization of transform coefficients that quantifying unit 145 will provide from orthogonal transform unit 144. Quantifying unit 145 arranges quantization parameter based on the information relevant with the target value of the size of code provided from Rate control unit 157, and performs it and quantize. The transformation coeffcient of quantification is supplied to lossless encoding unit 146 by quantifying unit 145.

The transformation coeffcient quantized in quantifying unit 145 is encoded by lossless encoding unit 146 in any coding scheme. Under the control of Rate control unit 157, coefficient data is quantized. Therefore, this size of code turns into the target value (or approximate target value) that arranges by Rate control unit 157.

Further, lossless encoding unit 146 obtains the information of the instruction intra prediction mode etc. from intraprediction unit 154, and obtains the information of the instruction inter-frame forecast mode from inter prediction unit 155, differential motion vector information etc. Further, network abstract layer (NAL) unit of lossless encoding unit 146 suitably formation base layer, this network abstract layer (NAL) unit comprises sequential parameter collection (SPS), image parameters collection (PPS) etc., and NAL unit is encoded by any means.

Lossless encoding unit 146 by these various types of information, together with the coded data as the encoded quantization transform coefficient provided from quantifying unit 145, it is provided that to accumulation buffer memory 147, and information is accumulated in wherein.

The example of the coding scheme of lossless encoding unit 146 comprises Variable Length Code and arithmetic coding. The example of Variable Length Code is included in H.264/AVC system the CAVLC (CAVLC) determined. The example of arithmetic coding comprises context-adaptive binary arithmetic coding (CABAC).

Accumulation buffer memory 147 temporarily stores the coded data etc. provided from lossless encoding unit 146. Accumulation buffer memory 147 exports the data of storage to multiplexed device 133 (Figure 32) in predetermined timing place and flows (also referred to as the encoded stream of base layer image) as position, and this stream comprises the coded data etc. of basal layer. That is, accumulation buffer memory 147 is also the transmission unit of the transmission encoded stream of base layer image.

Further, also to inverse quantization unit 148 be provided in quantifying unit 145 quantize transformation coeffcient. Inverse quantization unit 148 makes the transformation coeffcient inverse quantization of quantification by the method corresponding with the quantification performed by quantifying unit 145. The transformation coeffcient of acquisition is supplied to anti-quadrature conversion unit 149 by inverse quantization unit 148.

The transformation coeffcient provided from inverse quantization unit 148 is carried out reverse orthogonal transformation by the method corresponding with the orthogonal transformation process performed by orthogonal transform unit 144 by anti-quadrature conversion unit 149. The output (difference information of recovery) of reverse orthogonal transformation is provided to calculating unit 150.

Calculate unit 150 to add to by predicted picture selection unit 156 as the difference information converting the recovery of anti-quadrature transformation results that unit 149 provides from anti-quadrature to obtain the image (decoded picture) of local decode from the predicted picture that intraprediction unit 154 or inter prediction unit 155 provide. Decoded picture is supplied to loop filter 151 or frame memory 152.

Loop filter 151 comprises de-blocking filter, auto-adaptive loop filter etc., and suitably performs filtering process to from the image reconfigured calculating unit 150 offer. Such as, loop filter 151 by performing the block distortion that block elimination filtering process removes the image reconfigured to the image reconfigured. Further, such as, loop filter 151 processes with the use of Wiener wave filter execution loop filtering and block elimination filtering result (having eliminated the image reconfigured of block distortion) performs picture quality raising. Loop filter 151 provides filtering process result (being hereafter called decoded picture) to frame memory 152.

Noting, the image reconfigured can be performed another any filtering process by loop filter 151 further. Further, loop filter 151 can provide information (such as, the filter factor for filtering process) to lossless encoding unit 146, and, if desired, this information can be encoded by lossless encoding unit 146.

Frame memory 152 stores the decoded picture provided, and in predetermined timing place, as with reference to image, the decoded picture stored is provided to selection unit 153.

Specifically, frame memory 152 stores from calculating the image reconfigured that unit 150 provides and the decoded picture provided from loop filter 151. The image reconfigured stored, in predetermined timing place or based on the request from extraneous (such as, intraprediction unit 154 etc.), is supplied to intraprediction unit 154 by selection unit 153 by frame memory 152. Further, the decoded picture stored, in predetermined timing place or based on the request from extraneous (inter prediction unit 155 etc.), is supplied to inter prediction unit 155 by selection unit 153 by frame memory 152.

Selection unit 153 selects the offer point of destination of the reference picture provided from frame memory 152. Such as, when infra-frame prediction, the reference picture provided from frame memory 152 is supplied to intraprediction unit 154 by selection unit 153. Further, such as, when inter prediction, the reference picture provided from frame memory 152 is supplied to inter prediction unit 155 by selection unit 153.

Intraprediction unit 154 performs the prediction processing of the photo current about the image as present frame with generation forecast image. Intraprediction unit 154 performs prediction processing based on predetermined block one by one (in units of block). That is, intraprediction unit 154 generates the predicted picture of the current block as pending object of photo current. Now, intraprediction unit 154 is by performing prediction processing (in screen prediction (also referred to as infra-frame prediction)) by being used as reference picture by selection unit 153 from the image reconfigured that frame memory 152 provides. That is, intraprediction unit 154 with the use of the peripheral pixels value of the current block being included in the image reconfigured to generate predicted picture. Being the pixel value of the pixel of photo current for the peripheral pixels value of infra-frame prediction, this pixel has been processed by the past. As infra-frame prediction (that is, the mode of generation forecast image), prepare multiple method (also referred to as intra prediction mode) in advance alternatively. Intraprediction unit 154 performs infra-frame prediction under multiple pre-prepd intra prediction mode.

Intraprediction unit 154 is generation forecast image under all intra prediction modes alternatively, rearranges input picture that buffer memory 142 provides to assess the cost function value of corresponding predicted picture with the use of from screen, and selects optimal mode. When have selected optimum frame inner estimation mode, intraprediction unit 154 is encouraged the predicted picture generated in a best mode and is supplied to predicted picture selection unit 156.

Further, as described above, intraprediction unit 154 suitably provides the intraprediction mode information etc. of the intra prediction mode of instruction employing to lossless encoding unit 146, and this information can be encoded by lossless encoding unit 146.

Inter prediction unit 155 performs the prediction processing about photo current with generation forecast image. Inter prediction unit 155 performs prediction processing based on predetermined block one by one (in units of block). That is, inter prediction unit 155 generates the predicted picture of the current block of the pending object as photo current. Now, inter prediction unit 155 with the use of from screen rearrange buffer memory 142 provide input picture view data and from frame memory 152 provide decoded picture view data as with reference to image to perform prediction processing. This decoded picture be the image of the frame (not being another picture of photo current) of the pre-treatment at photo current. That is, inter prediction unit 155 performs the image with the use of another picture to generate the prediction processing (predicting (also referred to as inter prediction) between screen) of predicted picture.

By moving, prediction forms this inter prediction with motion compensation. Specifically, inter prediction unit 155 performs the motion about current block with the use of input picture and reference picture and predicts, and detects motion vector. Then inter prediction unit 155 performs motion compensation process with the use of reference picture according to the motion vector detected, and generates the predicted picture (inter prediction graphic information) of current block. As this inter prediction (that is, the mode of generation forecast image), prepare multiple method (also referred to as inter-frame forecast mode) in advance alternatively. Inter prediction unit 155 performs inter prediction under multiple pre-prepd inter-frame forecast mode.

Inter prediction unit 155 is generation forecast image under all inter-frame forecast modes alternatively. Inter prediction unit 155 with the use of rearranging, from screen, input picture that buffer memory 142 provides, the information etc. of differential motion vector of generation assesses the cost function value of corresponding predicted picture, and selects optimal mode. When have selected best inter-frame forecast mode, the predicted picture generated in a best mode is supplied to predicted picture selection unit 156 by inter prediction unit 155.

Inter prediction unit 155 provides the information of the inter-frame forecast mode of instruction employing, when being decoded by coded data, execution under inter-frame forecast mode is processed necessary information etc. to lossless encoding unit 146, and makes lossless encoding unit 146 perform coding. As essential information, such as, the information of the differential motion vector of generation and instruction are as the mark of index of the predicted motion vector of predicted motion vector information.

Predicted picture selection unit 156 is selected to be supplied to calculating unit 143 and calculate the offer source of the predicted picture of unit 150. Such as, in the case of intra-coding, predicted picture selection unit 156 selects intraprediction unit 154 as the offer source of predicted picture, and provides, to calculating unit 143 and calculate unit 150, the predicted picture provided from intraprediction unit 154. Further, such as, in the case of inter coding, predicted picture selection unit 156 selects inter prediction unit 155 as the offer source of predicted picture, and provides, to calculating unit 143 and calculate unit 150, the predicted picture provided from inter prediction unit 155.

Rate control unit 157 controls the speed of the quantization operation of quantifying unit 145, not generate overflow or underflow based on the size of code of the coded data being accumulated in accumulation buffer memory 147.

Further, if desired, the information etc. of the basal layer of storage decoded picture is supplied to enhancement layer image coding unit 132 (Figure 32) by frame memory 152. Noting, frame memory 152 can obtain and storage information, such as, from the enhancement layer of enhancement layer image coding unit 132 offer decoded picture etc. If desired, by selection unit 153, this information is supplied to intraprediction unit 154 or inter prediction unit 155 as with reference to image etc.

Figure 34 be a diagram that the block diagram of the main configuration example of the enhancement layer image coding unit 132 of Figure 32. As shown in figure 34, enhancement layer image coding unit 132 has the substantially similar configuration of the base layer image coding unit 131 with Figure 33.

Namely, as shown in figure 34, enhancement layer image coding unit 132 comprise screen rearrange buffer memory 162, calculate unit 163, orthogonal transform unit 164, quantifying unit 165, lossless encoding unit 166, accumulation buffer memory 167, inverse quantization unit 168 and anti-quadrature conversion unit 169. Further, enhancement layer image coding unit 132 comprises calculating unit 170, loop filter 171, frame memory 172, selection unit 173, intraprediction unit 174, inter prediction unit 175, predicted picture selection unit 176 and Rate control unit 177.

Screen rearranges buffer memory 162 to Rate control unit 177, and to rearrange buffer memory 142 to Rate control unit 157 with the screen of Figure 33 corresponding, and performs the process similar with corresponding processing unit. Noting, the unit of enhancement layer image coding unit 132 performs the process about enhancement layer image information instead of basal layer. Therefore, it is possible to apply the description of description as the process that screen is rearranged buffer memory 162 to Rate control unit 177 that the screen to Figure 33 rearranges buffer memory 142 to Rate control unit 157. In this case, pending data need to be the data of enhancement layer, instead of the data of basal layer. Further, the input source of data and the processing unit exporting point of destination need suitably to be used in the respective handling unit that screen rearranges in buffer memory 162 to Rate control unit 177 and substitute, and are read as the respective handling unit rearranging in buffer memory 162 to Rate control unit 177 at screen.

Further, frame memory 172 obtains and stores information, such as, from the basal layer of base layer image coding unit 131 offer decoded picture etc. If desired, by selection unit 173, this information is supplied to intraprediction unit 174 or inter prediction unit 175 as with reference to image etc. Noting, if desired, the enhancement layer of storage decoded picture can be supplied to base layer image coding unit 131 (Figure 32) by frame memory 172.

Utilizing above configuration, picture coding device 100 can improve coding efficiency more easily.

Next, the flow process of each process performed by picture coding device 100 will be described. First, the example of the flow process of RAW image coded treatment is described with reference to the schema of Figure 35.

In step S101, when starting RAW image coded treatment, the camera sensor 101 (Figure 29) of picture coding device 100 is by making the imagings such as object obtain RAW data. In step s 102, the RAW data got in step S101 are packed by packaged unit 102, as described in the first embodiment. The details of packing process will be described below.

In step s 103, the RAW data of packing are encoded by image coding unit 103 by the process of step S102. In step S104, position is flowed (encoded stream) by any recording medium and transmission medium and is transferred to decoding side by transmission unit 104, and this position stream (encoded stream) comprises coded data and the package information of the RAW data of the packing generated by the process of step S103.

When terminating the process of step S104, also terminate RAW image coded treatment.

Next, the example of the flow process of the packing process performed in step S102 (Figure 35) is described with reference to the schema of Figure 36.

In step S111, when starting packing and process, separating unit 111 based on the pixel data of dependency separation RAW image, as described in the first embodiment. In step S112, rearrange the pixel data group that unit 112 rearranges separation, as described in the first embodiment. In step S113, package information generates the package information described in the first embodiment that unit 113 generates relevant packing process. When terminating the process of step S113, also terminate packing process, and make process return Figure 35.

Next, the example of the flow process of the picture coding process performed in step S103 (Figure 35) is described with reference to the schema of Figure 37.

When starting picture coding and process, setting unit 121 (Figure 31) arranges various parameter collection, CFAPSEI etc. based on the package information generated in step S113, as described in the first embodiment.

In step S122, the view data packed is encoded by coding unit 122 with the use of the parameter collection arranged in step S121 etc., as described in the first embodiment. The details of coding will be described below.

When terminating the process of step S122, also terminate picture coding process, and make process return Figure 35.

Next, the example of the flow process of the coded treatment performed in step S122 (Figure 37) is described with reference to the schema of Figure 38.

In step 131, when starting coded treatment, control unit 134 controls the unit from base layer image coding unit 131 path multiplexer 133 at the most based on parameter collection etc.

In step S132, the basal layer of the RAW data of packing is encoded by base layer image coding unit 131. In step S133, the enhancement layer of the RAW data of packing is encoded by enhancement layer image coding unit 132.

In step S134, multiplexed device 133 in step S132 generate the encoded stream of base layer image and in step S133 generate the encoded stream of enhancement layer image (namely, the position stream of corresponding layer) carry out multiplexed, and the encoded stream of the layered image generating a system. Noting, multiplexed device 133 comprises Additional Information, such as, if desired, and various parameter collection in layered image coded data and CFAPSEI. Encoded for layered image stream is exported and transfers to decoding side by multiplexed device 133.

When terminating the process of step S134, also terminate coded treatment, and make process return Figure 37. Each picture is performed this kind of coded treatment by coding unit 122 respectively. Note, suitably omit the process that need not perform based on picture one by one.

Next, the example of the flow process of the basal layer coded treatment performed by base layer image coding unit 131 in the step S132 of Figure 38 with reference to the schema description of Figure 39.

In step s 141, when starting basal layer coded treatment, the screen of base layer image coding unit 131 rearranges the image of buffer memory 142 according to the frame (picture) of the sequence store input motion image of display, and performs from the shown sequence of picture to coded sequence to rearrange.

In step S142, intraprediction unit 154 performs intra-prediction process under intra prediction mode.

In step S143, inter prediction unit 155 performs inter prediction process under inter-frame forecast mode: perform motion prediction, motion compensation etc.

In step S144, predicted picture selection unit 156 selects predicted picture based on cost function value etc. That is, predicted picture selection unit 156 selects the predicted picture generated by the infra-frame prediction of step S142 or the predicted picture generated by the inter prediction of step S143.

In step S145, calculate unit 143 and calculate the difference between the input picture rearranged with the order of frame in the process by step S141 and the predicted picture selected by the process of step S144. That is, the view data of the difference image that unit 143 is created between input picture and predicted picture is calculated. Compared with raw image data, the data volume of the view data of the difference image obtained as described above reduces. Therefore, compared with situation about intactly image being encoded, it is possible to amount of compressed data.

In step S146, the view data of the difference image that the process by step S145 is generated by orthogonal transform unit 144 carries out orthogonal transformation.

In step S147, quantifying unit 145 makes the orthogonal transform coefficient obtained by the process of step S146 quantize with the use of the quantization parameter calculated by Rate control unit 157.

In step S148, inverse quantization unit 148 utilizes the characteristic corresponding with the characteristic of quantifying unit 145 to make to be quantized by the process of step S147 and the coefficient coefficient of quantification (this coefficient can also the be called) inverse quantization that generates.

In step S149, the orthogonal transform coefficient that the process by step S148 is obtained by anti-quadrature conversion unit 149 carries out anti-quadrature conversion.

In step S150, calculate unit 150 and add, by the predicted picture process by step S144 selected, the view data that the difference image recovered by the process of step S149 generates the image reconfigured to.

In step S151, the view data of the image reconfigured that the process by step S150 is generated by loop filter 151 performs loop filtering process. Therefore, eliminate the block distortion etc. of the image reconfigured.

In step S152, frame memory 152 stores the data of the decoded picture (basal layer is decoded picture) obtained by the process of step S151 and the image reconfigured obtained by the process of step S150.

In step S153, the coefficient of the quantification that the process by step S147 is obtained by lossless encoding unit 146 encodes. That is, the data corresponding with difference image are performed harmless coding, such as, Variable Length Code and arithmetic coding.

Further, now, the information that the predictive mode of the predicted picture selected with the process by step S144 is relevant is encoded by lossless encoding unit 146, and adds encoded information to obtain by being encoded by difference image coded data. Namely, lossless encoding unit 146 also information to the best intraprediction mode information provided from intraprediction unit 154 or according to the best inter-frame forecast mode provided from inter prediction unit 155 encode, and add encoded information to coded data.

In step S154, accumulation buffer memory 147 accumulates the coded data etc. obtained by the process of step S153. The coded data etc. being accumulated in accumulation buffer memory 147 is suitably read as the encoded stream of base layer image, and encoded for this base layer image stream is supplied to multiplexed device 133 and utilizes the encoded stream of enhancement layer image to carry out multiplexed to it, and then transmit it to decoding side by transmission line and recording medium.

In step S155, Rate control unit 157 controls the quantization operation of quantifying unit 145 based on the size of code (size of code of generation) of the coded data etc. being accumulated in accumulation buffer memory 147 by the process of step S154, not generate overflow or underflow. Further, Rate control unit 157 vectorization unit 145 provides the information relevant with quantization parameter.

In step S156, frame memory 152 provides, to enhancement layer coding process, the information comprising the basal layer basal layer of decoded picture.

When terminating the process of step S156, also terminate basal layer coded treatment, and make process return Figure 38.

Next, the example of the flow process of the enhancement layer coding process performed by enhancement layer image coding unit 132 in the step S133 of Figure 38 with reference to the schema description of Figure 40.

In step S161, when starting enhancement layer coding and process, frame memory 172 obtains and stores the information of basal layer, the basal layer that the information of this basal layer is provided from base layer image coded treatment by the process (Figure 39) of step S156 decoded picture etc.

Process from step S162 to step S176 is corresponding with the process from step S141 to step S155 of basal layer coded treatment (Figure 39), and is performed substantially similarly with corresponding process. But, although basal layer being performed the process of basal layer coded treatment, but enhancement layer is performed the process of this enhancement layer coding process.

When terminating the process of step S176, also terminate enhancement layer image coded treatment, and make process return Figure 38.

By performing process described above, picture coding device 100 can improve coding efficiency more easily when RAW data being encoded.

Note, in the above description, described picture coding device 100 to the situation about encoding through two layerings (two layers) the RAW data of packing process of basal layer and enhancement layer. But, the quantity (quantity of layer) of the layering of the RAW data encoded by picture coding device 100 is arbitrary. That is, the coding unit 122 of picture coding device 100 can only have the coding unit of the quantity (quantity of layer) of the layering of RAW data.

Such as, when the RAW data of a layering (only basal layer) being encoded, basal layer can only be encoded by picture coding device 100. Therefore, coding unit 122 can omit enhancement layer image coding unit 132.

Further, such as, when the RAW data of three layerings (three layers) are encoded, coding unit 122 can comprise a base layer image coding unit 131 and two enhancement layer image coding units 132, and basal layer can be encoded by base layer image coding unit 131, and enhancement layer different from each other can be encoded by two enhancement layer image coding units 132.

Further, such as, when the RAW data of N number of layering (N layer) are encoded, such as, coding unit 122 can only comprise a base layer image coding unit 131 and (N-1) individual enhancement layer image coding unit 132, and basal layer can only be encoded by base layer image coding unit 131, and the layer different from each other of (N-1) individual enhancement layer can be encoded by (N-1) individual enhancement layer image coding unit 132.

Next, the decoding coded data describing coding as described above carried out. Figure 41 be a diagram that the block diagram of the main configuration example of the picture decoding apparatus corresponding with picture coding device 100, and picture decoding apparatus is the image processing equipment of a kind of form applying this technology. The decoded data generated by picture coding device 100 is decoded by the picture decoding apparatus 200 shown in Figure 41 by the coding/decoding method corresponding with coding method.

As shown in figure 41, picture decoding apparatus 200 comprises reception unit 201, image decoding unit 202, unwrapper unit 203 and development treatment unit 204.

Receive unit 201 and obtain the encoded stream of layered image come from the transmission of coding side. The encoded stream of this layered image is position stream, this stream comprises: as the encoded RAW data through rearranging the packing process as each pixel data in the RAW data performing the view data before demosaicing processes according to the degree of correlation coded data, process the SEI of package information of relevant information, parameter collection etc. including as with packing, as described in the first embodiment and the second embodiment. Receive unit 201 and encoded for the layered image received stream is supplied to image decoding unit 202.

Image decoding unit 202 is by the method substantially similar with situation about being decoded by the encoded stream as the coded picture data after demosaicing process, the encoded stream of layered image provided is decoded, as described in the first embodiment. Noting, as described below, image decoding unit 202 extracts the package information being included in the encoded stream of layered image. The RAW data through packing process obtained by decoding and the package information extracted are supplied to unwrapper unit 203 by image decoding unit 202.

Unwrapper unit 203 performs Xie Bao process based on package information: for the RAW data processed through packing, pixel data is returned to and is performing the layout before packing processes, as described in the first embodiment. Therefore, recovered performing the RAW data before packing processes. The RAW data of acquisition are supplied to development treatment unit 204 by unwrapper unit 203.

For the RAW data provided, development treatment unit 204 performs development treatment, and such as, demosaicing and automatic dressing, as described in the first embodiment. Such as, development treatment unit 204 exports the view data of the development outside picture decoding apparatus 200. The image of view data is shown on a monitor, predetermined image process is applied to view data, and view data is encoded, recorded in the recording medium and transmit it to another device by transmission medium.

Noting, image decoding unit 202 and unwrapper unit 203 can be configured to a processing unit (RAW data decoding unit 205). Such as, that is, this RAW data decoding unit 205 (image decoding unit 202 and unwrapper unit 203) can be configured to an image processing equipment.

Figure 42 be a diagram that the block diagram of the main configuration example of the image decoding unit 202 of Figure 41. As shown in figure 42, image decoding unit 202 comprises decoding unit 211 and package information extraction unit 212.

For the encoded stream of layered image provided, decoding unit 211 performs the decoding process corresponding with the coded treatment performed by coding unit 122. As described in the first embodiment, the encoded stream of layered image, by the method substantially similar with situation about being decoded by the coded data as the coded picture data after development treatment, is decoded by decoding unit 211. Such as, the parameter collection of the RAW data after packing processes obtained, coded data and SEI (CFAPSEI) etc. are supplied to package information extraction unit 212 by decoding unit 211.

Such as, package information extraction unit 212 is extracted in package information to be used when Xie Bao processes from the data (CFAPSEI) provided. Such as, the encoded stream of layered image comprises various types of package information, as described in the first embodiment. Package information extraction unit 212 extracts package information, and together with the RAW data after processing in packing, package information is supplied to unwrapper unit 203.

Figure 43 be a diagram that the block diagram of the main configuration example of the decoding unit 211 of Figure 42. The coding/decoding method of decoding unit 211 is arbitrary, as long as the method is corresponding with the coding method of coding unit 122. Such as, unit 211 is decoded to as performing layered image decoding (scalable decoding) through the coded data of the RAW data after packing processes of apparatus of layered picture coding apparatus of picture (scalable coding). In Figure 43, using describe decoding unit 211 to as the RAW data through apparatus of layered picture coding apparatus of picture (scalable coding) coded data perform layered image decoding (scalable decoding) when, RAW data are made up of basal layer and enhancement layer two layerings. As shown in figure 43, decode unit 211 and comprise demultiplexer 221, base layer image decoding unit 222, enhancement layer image decoding unit 223 and control unit 224.

Demultiplexer 221 receives the encoded stream of layered image come from the transmission of coding side, and it is multiplexed to extract the encoded stream of base layer image and the encoded stream of enhancement layer image that the encoded stream of layered image carries out solution. Further, demultiplexer 221 extracts the information being included in the encoded stream of layered image, such as, and parameter collection and SEI (such as, CFAPSEI), and this information is supplied to control unit 224.

The encoded stream of the base layer image extracted by demultiplexer 221 is decoded the data (also referred to as base layer image) obtaining the basal layer of the RAW data after packing processes by base layer image decoding unit 222. The base layer image of acquisition is supplied to package information extraction unit 212 (Figure 42) by base layer image decoding unit 222.

The encoded stream of the enhancement layer image extracted by demultiplexer 221 is decoded the data (also referred to as enhancement layer image) obtaining the enhancement layer of the RAW data after packing processes by enhancement layer image decoding unit 223. The enhancement layer image of acquisition is supplied to package information extraction unit 212 (Figure 42) by enhancement layer image decoding unit 223.

Such as, control unit 224 controls base layer image decoding unit 222 and enhancement layer image decoding unit 223 (controlling the decoding to corresponding layer) by demultiplexer 221 based on information (the parameter collection provided). Such as, information (the parameter collection provided by demultiplexer 221) is supplied to package information extraction unit 212 (Figure 42) by control unit 224.

Figure 44 be a diagram that the block diagram of the main configuration example of the base layer image decoding unit 222 of Figure 43. As shown in figure 44, base layer image decoding unit 222 comprise accumulation buffer memory 231, lossless decoding unit 232, inverse quantization unit 233, anti-quadrature conversion unit 234, calculate unit 235, loop filter 236 and screen rearrange buffer memory 237. Further, base layer image decoding unit 222 comprises frame memory 239, selection unit 240, intraprediction unit 241, inter prediction unit 242 and predicted picture selection unit 243.

Accumulation buffer memory 231 is also the reception unit of the coded data (the encoded stream of the base layer image provided by demultiplexer 221) receiving transmission. Accumulation buffer memory 231 receives the coded data of transmission, accumulation coded data, and in predetermined timing place, coded data is supplied to lossless decoding unit 232. Such as, necessary information (prediction mode information) will be decoded and add coded data to.

The information encoded by lossless encoding unit 146 and provide from accumulation buffer memory 231 is decoded by lossless decoding unit 232 in the decode system corresponding with coding scheme. Lossless decoding unit 232 provides the coefficient of the quantification of the difference image obtained by decoding to inverse quantization unit 233.

Further, lossless decoding unit 232 determines whether to have selected intra prediction mode or have selected inter-frame forecast mode as optimum prediction mode, and is defined as, to intraprediction unit 241 and inter prediction unit 242, the information that the selected pattern selected provides relevant with optimal selection pattern. Such as, that is, when have selected intra prediction mode as the optimum prediction mode of coding side, the information (intraprediction mode information) relevant with optimum prediction mode is provided to intraprediction unit 241. Further, such as, when have selected inter-frame forecast mode as the optimum prediction mode of coding side, the information (inter-frame forecast mode information) relevant with optimum prediction mode is provided to inter prediction unit 242.

Further, lossless decoding unit 232 extracts the necessary information of inverse quantization from decoded data, such as, and quantization matrix and quantization parameter, and this information is supplied to inverse quantization unit 233.

The coefficient data of the quantification obtained by lossless decoding unit 232 is carried out inverse quantization by inverse quantization unit 233 in the system corresponding with the quantization system of quantifying unit 145. Noting, inverse quantization unit 233 is the processing unit similar with inverse quantization unit 148. The coefficient data (orthogonal transform coefficient) obtained is supplied to anti-quadrature conversion unit 234 by inverse quantization unit 233.

If desired, the orthogonal transform coefficient provided by inverse quantization unit 233 is carried out anti-quadrature conversion by anti-quadrature conversion unit 234 in the system corresponding with the orthogonal transformation system of orthogonal transform unit 144. Noting, anti-quadrature conversion unit 234 converts the similar processing unit of unit 149.

The view data of difference image is recovered by anti-quadrature conversion process. The view data of the recovery of difference image is corresponding with the view data of the difference image before the execution orthogonal transformation of coding side. Hereinafter, the view data of the recovery of the different images obtained by the anti-quadrature conversion process of anti-quadrature conversion unit 234 is also referred to as decoding residual data. Decoding is remained data and is supplied to calculating unit 235 by anti-quadrature conversion unit 234. Further, the view data of predicted picture is provided from intraprediction unit 241 or inter prediction unit 242 to calculating unit 235 by predicted picture selection unit 243.

Calculate unit 235 to obtain as the view data of the difference image added and the image reconfigured of predicted picture with the use of the view data of decoding residual data and predicted picture. This image reconfigured is with corresponding by calculating the input picture before unit 143 subtracts predicted picture. Calculate unit 235 and the image reconfigured is supplied to loop filter 236.

The loop green glow process comprising block elimination filtering process, adaptive loop filter process etc. is suitably applied to the image reconfigured of offer to generate decoded picture by loop filter 236. Such as, loop filter 236 removes block distortion by the image reconfigured performs block elimination filtering process. Further, such as, loop filter 236 processes with the use of Wiener wave filter execution loop filtering and block elimination filtering result (having eliminated the image reconfigured of block distortion) performs picture quality raising.

The type of the filtering process performed by loop filter 236 is arbitrary, and can perform the filtering process except processing above. Further, loop filter 236 can perform filtering process with the use of filter factor that is that provide from coding side and that use in the filtering process of the execution when encoding. Further, loop filter 236 can omit this kind of filtering process and input-output data and do not perform filtering process.

Loop filter 236 rearranges buffer memory 237 to screen and frame memory 239 is provided as the decoded picture (image reconfigured) of filtering process result.

Screen rearranges the order that buffer memory 237 rearranges the frame about decoded picture. That is, screen rearranges buffer memory 237 image rearranging the frame that buffer memory 142 rearranges taking the order encoded by screen rearranges as original display order by having. Namely, screen rearranges the view data of the decoded picture of the frame of the order offer that buffer memory 237 is waited to encode with this sequence store, and read the view data of the decoded picture of the frame with the sequence store encoded according to the order of display, and the view data read is supplied to package information extraction unit 212 (Figure 42).

Frame memory 239 stores the decoded picture provided, and predetermined timing place or based on extraneous (such as, intraprediction unit 241 and inter prediction unit 242) request, by selection unit 240 using the decoded picture stored as being supplied to intraprediction unit 241 and inter prediction unit 242 with reference to image.

Selection unit 240 selects the offer point of destination of the reference picture provided from frame memory 239. Such as, when infra-frame prediction, the reference picture provided from frame memory 239 is supplied to intraprediction unit 241 by selection unit 240. Further, such as, when inter prediction, the reference picture provided from frame memory 239 is supplied to inter prediction unit 242 by selection unit 240.

Suitably intraprediction mode information etc. is provided to intraprediction unit 241 from lossless decoding unit 232. Intraprediction unit 241 performs infra-frame prediction with generation forecast image under for the intra prediction mode (optimum frame inner estimation mode) of intraprediction unit 154. Now, intraprediction unit 241 by selection unit 240, with the use of the view data of the image reconfigured provided from frame memory 239 to perform infra-frame prediction. That is, intraprediction unit 241 uses the image reconfigured as with reference to image (peripheral pixels). The predicted picture of generation is supplied to predicted picture selection unit 243 by intraprediction unit 241.

Optimum prediction mode information, movable information etc. are provided to inter prediction unit 242 from lossless decoding unit 232. Inter prediction unit 242, by under the inter-frame forecast mode (best inter-frame forecast mode) of the optimum prediction mode information instruction got from lossless decoding unit 232, performs inter prediction with the use of the decoded picture (reference picture) got from frame memory 239.

The predicted picture provided from intraprediction unit 241 or the predicted picture that provides from inter prediction unit 242 are supplied to and calculate unit 235 by predicted picture selection unit 243. Calculating in unit 235, adding the predicted picture and decoding residual data (difference image information) that convert unit 234 from anti-quadrature, and obtain the image reconfigured.

Further, if desired, frame memory 239 decodes unit 223 (Figure 43) to enhancement layer image and provides the information stored, such as, and basal layer decoded picture etc. Noting, frame memory 239 can obtain and store information, such as, decodes, from enhancement layer image, the information that unit 223 provides, such as, and enhancement layer decoded picture etc. If desired, by selection unit 241, this information is supplied to intraprediction unit 241 or inter prediction unit 242 as with reference to image etc.

Figure 45 be a diagram that the block diagram of the main configuration example of the enhancement layer image decoding unit 223 of Figure 43. As shown in figure 43, enhancement layer image decoding unit 223 has the substantially similar configuration of the decoding unit 222 of the base layer image with Figure 44.

As shown in figure 45, that is, enhancement layer image decoding unit 223 comprise accumulation buffer memory 251, lossless decoding unit 252, inverse quantization unit 253, anti-quadrature conversion unit 254, calculate unit 255, loop filter 256 and screen rearrange buffer memory 257. Further, enhancement layer image decoding unit 223 comprises frame memory 259, selection unit 260, intraprediction unit 261, inter prediction unit 262 and predicted picture selection unit 263.

Accumulation buffer memory 251 to predicted picture selection unit 263 is corresponding with accumulation buffer memory 231 to the predicted picture selection unit 243 of Figure 44, and performs the process similar with corresponding processing unit. Noting, the unit of enhancement layer image decoding unit 223 performs about the process of enhancement layer image information instead of the decoding of basal layer. Therefore, it is possible to apply the description of description as the process to accumulation buffer memory 251 to predicted picture selection unit 263 of accumulation buffer memory 231 to the predicted picture selection unit 243 to Figure 44. In this case, pending data need to be the data of enhancement layer, instead of the data of basal layer. Further, the processing unit of the input source of data and output point of destination needs suitably to substitute with the alignment processing unit in accumulation buffer memory 251 to predicted picture selection unit 263, and the alignment processing unit being read as in accumulation buffer memory 251 to predicted picture selection unit 263.

Further, frame memory 259 obtains and stores and decode unit 222 (Figure 44) information that provides from base layer image, such as, and basal layer decoded picture etc. If desired, by selection unit 260, this information is supplied to intraprediction unit 261 or inter prediction unit 262 as with reference to image etc. Noting, if desired, the enhancement layer of storage decoded picture can be supplied to base layer image decoding unit 222 (Figure 44) by frame memory 259.

Figure 46 be a diagram that the block diagram of the main configuration example of the unwrapper unit 203 of Figure 41. As shown in figure 46, unwrapper unit 203 comprises package information resolution unit 271, separating unit 272 and rearranges unit 273.

Package information resolution unit 271 resolves the package information provided from image decoding unit 202 (Figure 41). Analysis result is supplied to separating unit 272 together with RAW data that are that provide from image decoding unit 202 (Figure 41) and that process through packing by package information resolution unit 271.

Separating unit 272 based on the analysis result separation of package information through the pixel data of RAW data of packing process. Pixel data after separation is supplied to together with the analysis result of package information and rearranges unit 273 by separating unit 272.

Rearrange the analysis result of unit 273 based on package information to be rearranged by the pixel data of separation and be put back into and performing the layout before packing processes. That is, recovered performing the RAW data before packing processes. Rearrange unit 273 and the RAW data of acquisition are supplied to development treatment unit 204 (Figure 41).

Utilizing configuration described above, picture decoding apparatus 200 is by performing decoding more easily and correctly corresponding to the method for the coding/decoding method being associated with packing process, and can perform Xie Bao process. That is, picture decoding apparatus 200 can improve coding efficiency more easily.

Next, the flow process of the process performed by picture coding device 200 described above will be described. First, the example of the flow process of RAW image decoding process is described with reference to the schema of Figure 47.

In step s 201, when starting RAW image decoding and process, image decoding fills the reception unit 201 received bit stream (the encoded stream of layered image) of 200, this stream comprise that come from the transmission of coding side and as the coded data of the encoded RAW data processed through packing.

In step S202, the encoded stream of the layered image received in step s 201 is decoded by image decoding unit 202.

In step S203, RAW data that are that obtained by the process of step S202 and that process through packing are carried out Xie Bao process by unwrapper unit 203.

In step S204, the RAW data that obtained by the process of step S203 are performed development treatment by development treatment unit 204, and this development treatment comprises demosaicing process, finishing process etc. to color, brightness, tone etc. When obtaining the view data after demosaicing processes, terminate RAW image decoding process.

Next, the example of the flow process of the image decoding process performed in the step S202 of Figure 47 is described with reference to the schema of Figure 48. In step S211, when starting image decoding and process, the encoded stream of layered image is decoded by the decoding unit 211 of image decoding unit 202. Such as, in step S212, package information extraction unit 212 extracts package information from the SEI (CFAPSEI) being included in the encoded stream of layered image. As described in the first embodiment, such as, CFAPSEI comprises various types of package information. Package information extraction unit 212 extracts package information, and this package information is supplied to Xie Bao process. When comprising package information in parameter collection, header information etc., package information extraction unit 212 can extract package information from it.

When terminating the process of step S212, also terminate image decoding process, and make process return Figure 47.

Next, the example of the flow process of the decoding process performed in the step S211 of Figure 48 is described with reference to the schema of Figure 49.

In step S221, when starting decoding and process, it is multiplexed that the encoded stream of layered image come from the transmission of coding side is carried out solution by the demultiplexer 221 of picture decoding apparatus 200 based on layer one by one.

Such as, in step S222, control unit 224 controls the decoding to each layer based on the information (parameter collection and SEI) extracted in step S221.

In step S223, the encoded stream of base layer image is decoded by base layer image decoding unit 222. In step S224, the encoded stream of enhancement layer image is decoded by enhancement layer image decoding unit 223.

When terminating the process of step S224, also terminate decoding process.

Next, the example of the flow process of the basal layer decoding process performed in the step S223 of Figure 47 is described with reference to the schema of Figure 50.

In step S231, when starting basal layer decoding and process, the accumulation buffer memory 231 of base layer image decoding unit 222 accumulates the encoded stream of basal layer that transmission comes. In step S232, the encoded stream of basal layer provided from accumulation buffer memory 231 is decoded by lossless decoding unit 232. That is, the view data of the I section encoded by lossless encoding unit 146, P section and B section is decoded. Such as, now, also the various types of information in addition to the image data comprised in stream in place (header information) are decoded.

In step S233, the coefficient of the quantification obtained by the process of step S232 is carried out inverse quantization by inverse quantization unit 233.

In step S234, the coefficient of inverse quantization in step S233 is carried out reverse orthogonal transformation by anti-quadrature conversion unit 234.

In step S235, intraprediction unit 241 and inter prediction unit 242 perform prediction processing, and generation forecast image. That is, under the predictive mode being applied to coding, prediction processing is performed, it has been determined that this predictive mode in lossless decoding unit 232. Specifically, such as, when infra-frame prediction is applied to coding, intraprediction unit 241 is generation forecast image under intra prediction mode, when encoding it has been determined that this intra prediction mode is best. Further, such as, when inter prediction is applied to coding, inter prediction unit 242 is generation forecast image under inter prediction, when encoding it has been determined that this inter-frame forecast mode is best.

In step S236, calculate unit 235 and the predicted picture generated in step S235 is added to the difference image obtained by the anti-quadrature conversion in step S234. Therefore, it is possible to obtain the view data of the image reconfigured.

In step S237, the view data of the image reconfigured obtained by the process of step S236 is suitably performed loop filtering process by loop filter 236, and the process of this loop filtering comprises block elimination filtering process, adaptive loop filter process etc.

In step S238, screen is rearranged buffer memory 237 and performs rearranging of the frame to the decoded picture obtained by the filtering process of step S237. That is, the order of frame rearranged when encoding is rearranged as original display order. Screen rearranges the frame that buffer memory 237 reads decoded picture with the order rearranged, and frame is supplied to package information extraction unit 212.

In step S239, data that frame memory 239 stores the decoded picture obtained by the process of step S237, the data etc. reconfigured obtained by the process of step S226.

In step S240, the information etc. of the basal layer of storage decoded picture is supplied to enhancement layer decoder process by frame memory 239.

When terminating the process of step S240, also terminate base layer image decoding process, and make process return Figure 49.

Next, the example of the flow process decoding the enhancement layer decoder process that unit 223 performs in the step S224 of Figure 49 by enhancement layer image is described with reference to the schema of Figure 51.

In step S251, when starting enhancement layer decoder and process, frame memory 259 obtains and stores the information of basal layer, the basal layer that the information of this basal layer is provided by basal layer decoding process by the process of step S240 (Figure 50) decoded picture etc.

Process from step S252 to step S260 is corresponding with the process of the step S231 to step S239 processing (Figure 50) from basal layer decoding, and is performed substantially similarly with corresponding process. But, although when basal layer has been performed the process of basal layer decoding process, but enhancement layer is performed the process of enhancement layer decoder process.

When terminating the process of step S260, also terminate enhancement layer decoder process, and make process return Figure 49.

Next, the example of the flow process of the Xie Bao process performed in the step S203 of Figure 47 is described with reference to the schema of Figure 52.

In step S271, when starting Xie Bao and process, the package information resolution unit 271 (Figure 46) of unwrapper unit 203 resolves the package information extracted by the process of step S212 (Figure 48).

In step S272, separating unit 272 is based on the analysis result of the package information obtained by the process of step S271, by the process of step S211 (Figure 48), it is separated the pixel data group of RAW data that are that obtain and that process through packing by being decoded by coded data.

In step S273, rearrange unit 273 based on the package information obtained by the process of step S271 analysis result by step S272 separation pixel data rearrange into packing process before layout. By process, the RAW data before packing processes are recovered.

When terminating the process of step S273, also terminate understanding bag process, and make process return Figure 48.

By performing process recited above, the coded data as encoded RAW data can easily and correctly be decoded by picture decoding apparatus 200. That is, picture decoding apparatus 200 can improve the coding efficiency when RAW data being encoded more easily.

Note, in the above description, described picture decoding apparatus 200 to based on the situation about decoding through the coded data of two layerings (two layers) the encoded RAW data of packing process of layer and enhancement layer. But, the quantity (quantity of layer) of the layering of the RAW data of the coded data decoded by picture decoding apparatus 200 is arbitrary. That is, the decoding unit 211 of picture decoding apparatus 200 can only comprise the coding unit of the quantity (quantity of layer) of the layering of RAW data.

Such as, when the coded data of the encoded RAW data as a layering (only basal layer) being decoded, basal layer can only be encoded by picture decoding apparatus 200. Therefore, enhancement layer image decoding unit 223 can be omitted in decoding unit 211 (Figure 43).

Further, when the coded data of the encoded RAW data as three layerings (three layers) is decoded, decoding unit 211 can only comprise a base layer image decoding unit 222 and two enhancement layer image decoding unit 223, and basal layer can only be encoded by base layer image decoding unit 222, and enhancement layer different from each other can only be encoded by two enhancement layer image decoding unit 223.

Namely, such as, when the coded data of the encoded RAW data as N number of layering (N layer) is decoded, decoding unit 211 can only comprise a base layer image decoding unit 222 and (N-1) individual enhancement layer image decoding unit 223, and basal layer can only be encoded by base layer image decoding unit 222, and the layer different from each other of (N-1) individual enhancement layer can only be encoded by (N-1) individual enhancement layer image decoding unit 223.

About the range of application of this technology, it is possible to this technology to be applied to any picture coding device and the picture decoding apparatus that RAW data can be carried out coding/decoding.

Further, such as, this technology can be applied to: for by satellite broadcasting, wire television, internet or network medium (such as, mobile phone device) receive by orthogonal transformation and motion compensation, such as, the picture coding device of the graphic information (position stream) that discrete cosine transform (such as MPEG, H.26x etc.) is compressed and picture decoding apparatus. Further, it is possible to this technology is applied to: picture coding device and picture decoding apparatus for storage media (such as CD, disk or flash memory) execution being processed are put.

< application in multi-view image coding/multi-view image decodes)

This series of processes described above can be applied to the decoding of multi-view image coding/multi-view image. Figure 53 illustrates the example of multi-view image coding scheme.

As shown in Figure 53, multi-view image comprises the image of multiple viewpoint (view). Multiple views of multi-view image are made up of basis view and non-basic view, when not using the information of other view, image only by the view using oneself this basis view is performed coding/decoding, and the information with the use of other view this non-basic view is performed coding/decoding. The coding/decoding carried out by non-basic view can use the information of basis view or can use the information of another non-basic view.

That is, the referring-to relation between the view in multi-view image coding/decoding and the referring-to relation between the layer in apparatus of layered picture coding apparatus of picture/decoding are similar. Therefore, when the multi-view image of such as Figure 53 is carried out coding/decoding, it is possible to apply method described above. Like this, when RAW data being encoded when multi-view image, it is possible to improve coding efficiency more easily.

<multi-view image coding device>

Figure 54 be a diagram that the schematic diagram of the multi-view image coding device performing multi-view image described above coding. As shown in Figure 54, multi-view image coding device 600 comprises coding unit 601, coding unit 602 and multiplexed device 603.

Basis view image is encoded with the encoded stream of formation base view image by coding unit 601. Non-basic view image is encoded to generate the encoded stream of non-basic view image by coding unit 602. The encoded stream of non-basic view image of the encoded stream of basic view image generated in coding unit 601 and generation in coding unit 602 is carried out multiplexed to generate the encoded stream of multi-view image by multiplexed device 603.

As the coding unit 122 of the picture coding device 100 described in a second embodiment, it is possible to apply this kind of multi-view image coding device 600. Like this, multi-view image can be encoded by picture coding device 100 by the method described in the first embodiment and the second embodiment, and in this multi-view image, the image of respective viewpoints is RAW data. That is, picture coding device 100 can improve coding efficiency more easily when being encoded by multi-view image, and in this multi-view image, the image of corresponding views is RAW data.

<multi-view image decoding device>

Figure 55 be a diagram that the schematic diagram of the multi-view image decoding device performing multi-view image described above decoding. As shown in fig. 55, multi-view image decoding device 610 comprises demultiplexer 611, decoding unit 612 and decoding unit 613.

It is multiplexed to extract the basis encoded stream of view and the encoded stream of non-basic view image that the encoded stream of multi-view image as the multiplexed encoded stream of basic view image and the encoded stream of non-basic view image is carried out solution by demultiplexer 611. The encoded stream of basic view image extracted by demultiplexer 611 is decoded to obtain basis view image by decoding unit 612. The encoded stream of non-basic view image extracted by demultiplexer 611 is decoded to obtain non-basic view image by decoding unit 613.

As the decoding unit 211 of the picture decoding apparatus 200 described in the third embodiment, it is possible to apply this kind of multi-view image decoding device 610. Like this, the coded data as multi-view image can be decoded by picture decoding apparatus 200 by the method described in the first embodiment or the 2nd embodiment, and in this multi-view image, the image of respective viewpoints is RAW data. That is, picture decoding apparatus 200 can improve coding efficiency more easily when being encoded by multi-view image, and in this multi-view image, the image of respective viewpoints is RAW data.

This series of process described above can be performed by hardware, or can perform by software. When this series of processes is performed by software, the program being configured to software is installed in a computer. Herein, computer comprises the computer being incorporated in specialized hardware and can perform the general purpose personal computer of various function by installing various program.

Figure 56 be a diagram that and utilizes program to perform the block diagram of the configuration example of the hardware of the computer of this series of processes described above.

In computer 800 shown in Figure 56, central processing unit (CPU) 801, read-only storage (ROM) 802 and random access memory (RAM) 803 are connected to each other by bus 804.

Input/output interface 810 is also connected to bus 804. Input unit 811, output unit 812, storage unit 813, communication unit 814 and driving mechanism 815 are connected to input/output interface 810.

Input unit 811 is made up of keyboard, mouse, microphone, contact panel, input end station etc. Output unit 812 is made up of indicating meter, loud speaker, output terminal etc. Storage unit 813 is made up of hard disk, ram disc, nonvolatile memory etc. Such as, the unit 814 that communicates is made up of network interface. Driving mechanism 815 drives move media 821, such as, and disk, CD, magneto-optic disk or semiconductor memory.

In the computer of configuration as described above, such as, the program being stored in storage unit 813 is loaded on RAM803 by input/output interface 810 and bus 804 by CPU801, and performs this program, thus performs this series of processes described above. Necessary data etc. suitably will be stored in RAM803 when performing various types of process by CPU801.

Such as, it is possible to using the program performed by computer (CPU801) as packing medium recording in move media 821, and this program can be applied. In this case, it is possible to by move media 821 is installed to driving mechanism 815, by input/output interface 810, program is installed to storage unit 813.

Further, it is possible to by wired or wireless transmission medium, such as, local area network, internet or digital satellite broadcasting provide program. In this case, program can be received by communication unit 814 and be installed to storage unit 813.

In addition, program can be pre-installed to ROM802 or storage unit 813.

Noting, the program performed by computer can be the program processed according to the time series execution of the order described in this manual, or can be parallel ground or the program in necessary timing place execution process when called.

Further, in this manual, coding and record the process that step in the recording medium is not only included in the time series according to said sequence to perform, also comprise not necessarily process but the process dripping separately or be performed parallel in time series.

Further, in this manual, system refers to the group of multiple configuration element (device, module (assembly) etc.), and no matter whether all configuration elements are contained in identical cabinet. Therefore, the multiple device being contained in independent cabinet and connected by network and the device comprising the multiple modules being contained in a cabinet are all systems.

Further, the configuration being described to a device (or a processing unit) can be divided into and be configured to multiple device (or processing unit). In contrast with the above, the configuration being described to multiple device (or processing unit) can be configured to a device (or a processing unit) jointly. Further, it is possible to the configuration except configuration described above is added to the configuration of device (or processing unit). Further, it is possible to the configuration at another device (or another processing unit) comprises a part for the configuration of certain device (or processing unit), as long as configuration or operation as whole system are identical substantially.

As described above, advantageous embodiment of the present disclosure is described in detail with reference to accompanying drawing. But, technical scope of the present disclosure is not limited to these examples. Obviously, those of ordinary skill in technical field of the present disclosure can be described in detail in the claims technical conceive scope in reach various change example or modified example, it is appreciated that obviously, and, these change examples or modified example belong to technical scope of the present disclosure.

Such as, this technology can have the configuration of cloud computing, and the configuration of this cloud computing processes a function by multiple device in a shared manner or by the interoperable mode of network.

Further, the step described in schema described above is except being performed by a device, it is also possible to perform in a shared manner by multiple device.

Further, when comprising multiple process in one step, the multiple process comprised in one step are except being performed by a device, it is also possible to perform in a shared manner by multiple device.

Picture coding device and picture decoding apparatus according to embodiment can be applied to various electronic installation, such as, in satellite broadcasting, electrophone (such as, wire television), distribution on internet or by cellular communication such as, to the transmitter in the distribution of terminal or receptor, image the is recorded in medium recording unit in (CD, disk or flash memory) and the reproduction device making the image repetition from these storage medias. Hereinafter, four application examples will be described.

Figure 57 illustrates the example of the illustrative configurations of the TV device applying embodiment described above. TV device 900 comprises antenna 901, tuner 902, demultiplexer 903, demoder 904, video signal processing unit 905, display unit 906, audio signal processing unit 907, loud speaker 908, outside interface (I/F) unit 909, control unit 910, user interface (I/F) unit 911 and bus 912.

Tuner 902 extracts the signal expecting channel from the broadcast signal received by antenna 901, and the signal that demodulation is extracted. Then tuner 902 exports the encoded position stream obtained by demodulation to demultiplexer 903. That is, the transmission unit that tuner 902 is used as in TV device 900, this transmission unit receives the encoded stream as encoded image.

The video flowing of Television programme seen from encoded position stream and audio frequency stream are treated in demultiplexer 903 separation, and export the stream of separation to demoder 904. Further, demultiplexer 903 extracts auxiliary data from encoded position stream, such as, and electronic programming guide (EPG) etc., and the data extracted are supplied to control unit 910. Noting, when encoded position stream is carried out scrambler, encoded position stream can be carried out solution and disturb by demultiplexer 903.

The video flowing inputted from demultiplexer 903 and audio frequency stream are decoded by demoder 904. The video data that then demoder 904 is generated being processed by decoding export video signal processing unit 905 to. Further, the audio frequency data that demoder 904 is generated being processed by decoding export audio signal processing unit 907 to.

Video signal processing unit 905 make from demoder 904 input video data reappear, and in display unit 906 display video. Further, video signal processing unit 905 can show the application screen provided by network in display unit 906. Further, video data can be performed additional processing according to arranging by video signal processing unit 905, such as, and denoising. Further, video signal processing unit 905 can generate the image of graphic user interface (GUI) (such as, menu, button or cursor), and is overlapped on output image by the image of generation.

Display unit 906 is driven by the actuate signal provided from video signal processing unit 905, and such as, display video or image on the video screen of display unit (liquid-crystal display, plasma display or display of organic electroluminescence (OELD) (OLED display)).

The audio frequency data inputted from demoder 904 are performed reproduction process by audio signal processing unit 907, such as, and D/A conversion and amplification, and from loud speaker 908 output audio. Further, audio frequency data can be performed additional processing by audio signal processing unit 907, such as, and denoising.

External interface unit 909 is for connecting TV device 900 and the interface of outer part device or network. Such as, the video flowing or the audio frequency stream that are received by external interface unit 909 can be decoded by demoder 904. That is, the transmission unit that external interface unit 909 is also used as in TV device 900, this transmission unit receives the encoded stream as encoded image.

Such as, such as, control unit 910 comprises treater (CPU) and storer (RAM and ROM). Storer store perform by CPU program, program data, EPG data, the data etc. that got by network. Such as, the program stored in memory is read by CPU when TV device 900 is started shooting and performs. CPU, by steering routine, controls to operate the operation of TV device 900 according to the operation signal inputted from user interface elements 911.

User interface elements 911 is connected with control unit 910. User interface elements 911 comprises the reception unit that user is used for operating the button of TV device 900 and switch and Long-distance Control signal. The operation that user interface elements 911 detects user by configuring element is to generate operation signal, and exports the operation signal of generation to control unit 910.

Tuner 902, demultiplexer 903, demoder 904, video signal processing unit 905, audio signal processing unit 907, external interface unit 909 and control unit 910 are connected to each other by bus 912.

In TV device 900 as described above, demoder 904 has the function of the picture decoding apparatus 200 according to embodiment described above. That is, demoder 904 is by the method for description in the first embodiment or the 3rd embodiment, correctly is decoded by the coded data of the RAW data encoded as the method by describing in the first embodiment or the 2nd embodiment. Therefore, TV device 900 can improve coding efficiency more easily when RAW data being encoded.

Figure 58 illustrates the example of the illustrative configurations of the mobile phone device applying embodiment described above. Mobile phone device 920 comprises antenna 921, communication unit 922, audio codec 923, loud speaker 924, microphone 925, camera unit 926, graphics processing unit 927, multiplexed/separating unit 928, record/reproduction unit 929, display unit 930, control unit 931, operation unit 932 and bus 933.

Antenna 921 is connected to communication unit 922. Loud speaker 924 and microphone 925 are connected to audio codec 923. Operation unit 932 is connected to control unit 931. Communication unit 922, audio codec 923, camera unit 926, graphics processing unit 927, multiplexed/separating unit 928, record/reproduction unit 929, display unit 930 and control unit 931 are connected to each other by bus 933.

Mobile phone device 920 is executable operations under various operator scheme, such as, the record of the transmission/reception of sound signal, e-mail or the transmission/reception of view data, the imaging of image and data, this various operator scheme comprises voice call mode, data communication mode, seizure pattern and videophone pattern.

Under voice call mode, provide the simulated audio signal generated by microphone 925 to audio codec 923. Simulated audio signal is converted to audio frequency data by audio codec 923, and the audio frequency data of conversion is performed A/D conversion and compress this audio frequency data. Then the audio frequency data of compression are exported to communication unit 922 by audio codec 923. Audio frequency data are encoded and modulate to generate transmission signal by communication unit 922. Then the transmission signal of generation is transferred to base station (not shown) by antenna 921 by communication unit 922. Further, the unit 922 that communicates is amplified the wireless signal received by antenna 921 and performs frequency inverted to obtain Received signal strength. Then communication unit 922 carries out demodulation to received signal and decodes to generate audio frequency data, and exports the audio frequency data of generation to audio codec 923. Audio codec 923 launches audio frequency data and performs D/A and change to generate simulated audio signal. Then the sound signal of generation is supplied to loud speaker 924 with output audio by audio codec 923.

Further, in a data communication mode, such as, control unit 931 generates text data, and text data configure e-mail by operation unit 932 according to the operation of user. Further, control unit 931 shows text in display unit 930. Further, control unit 931 generates e-mail data by operation unit 932 according to the transmission instruction from user, and the e-mail data of generation exports to communication unit 922. E-mail data is encoded and modulates to generate transmission signal by communication unit 922. Then the transmission signal of generation is transferred to base station (not shown) by antenna 921 by communication unit 922. Further, the unit 922 that communicates is amplified the wireless signal received by antenna 921 and performs frequency inverted to obtain Received signal strength. Then communication unit 922 carries out demodulation to received signal and decodes to recover e-mail data, and exports the e-mail data after recovering to control unit 931. Control unit 931 shows the content of e-mail in display unit 930, e-mail data is supplied to record/reproduction unit 929, and is write in storage media by e-mail data.

Record/reproduction unit 929 comprises any storage media can read/can write. Such as, storage media can be built-in storage media (such as, RAM or flash memory), or can be exterior formula storage media, such as, hard disk, disk, magneto-optic disk, CD, general serial bus (USB) storer or storage card.

Further, under seizure pattern, such as, camera unit 926 makes image objects with synthetic image data, and exports the view data of generation to graphics processing unit 927. The view data inputted from camera unit 926 is encoded by graphics processing unit 927, and encoded stream is supplied to record/reproduction unit 929, and is write in storage media by encoded stream.

Further, under image display mode, record/reproduction unit 929 reads the encoded stream being recorded in storage media, and exports the encoded stream read to graphics processing unit 927. The encoded stream inputted from record/reproduction unit 929 is decoded by graphics processing unit 927, and view data is supplied to display unit 930, and shows its image.

Further, under videophone pattern, such as, the audio frequency stream of multiplexed/separating unit 928 to the video flowing coded by graphics processing unit 927 and from audio codec 923 input carries out multiplexed, and exports the stream after multiplexed to communication unit 922. This stream is encoded and modulates to generate transmission signal by communication unit 922. Then the transmission signal of generation is transferred to base station (not shown) by antenna 921 by communication unit 922. Further, the unit 922 that communicates is amplified the wireless signal received by antenna 921 and performs frequency inverted to obtain Received signal strength. Encoded position stream can be included in these transmission signal and Received signal strength. Then communication unit 922 carries out demodulation to received signal and decoding is flowed to recover, and exports the stream after recovering to multiplexed/separating unit 928. Multiplexed/separating unit 928 by video flowing and audio frequency stream and input flow point from, and export video flowing to graphics processing unit 927 and export audio frequency stream to audio codec 923. Video flowing is decoded to generate video data by graphics processing unit 927. There is provided video data to display unit 930, and show a series of images by display unit 930. Coding and decoding video device 923 launches audio frequency stream and performs D/A and change to generate simulated audio signal. Then the sound signal of generation is supplied to loud speaker 924 with output audio by audio codec 923.

In the mobile phone device 920 of configuration as described above, graphics processing unit 927 has the function of the picture coding device 100 according to embodiment described above and picture decoding apparatus 200. Namely, RAW data can be encoded by graphics processing unit 927 by the method described in the first embodiment or the 2nd embodiment, and correctly can be decoded by the coded data obtained by encoding by the method described in the first embodiment or the 3rd embodiment. Therefore, mobile phone device 920 can improve coding efficiency more easily when RAW data being encoded.

Figure 59 illustrates the example of the illustrative configurations of the data recording/reproducing device applying embodiment described above. Such as, data recording/reproducing device 940 to the audio frequency data of Radio programme and coding video data, and by coded data record in the recording medium. Further, such as, data recording/reproducing device 940 can to the audio frequency data got from another device and coding video data, and by coded data record in the recording medium. Further, such as, data recording/reproducing device 940 reappears record data in the recording medium according to the instruction of user on watch-dog and loud speaker. Now, data recording/reproducing device 940 is to audio frequency data and video decoding data.

Data recording/reproducing device 940 comprises the indicating meter (OSD) 948 on tuner 941, outside interface (I/F) unit 942, encoder 943, hard disk drive (HDD) 944, disc driver 945, selector switch 946, demoder 947, screen, control unit 949 and user interface (I/F) unit 950.

Tuner 941 expects the signal of channel from the broadcast signal extraction received by antenna (not shown), and the signal that demodulation is extracted. Then tuner 941 exports the encoded position stream obtained by demodulation to selector switch 946. That is, the transmission unit that tuner 941 is used as in record/reproduction device 940.

External interface unit 942 is for connecting record/reproduction device 940 and the interface of outer part device or network. External interface unit 942 it may be that such as, electrics and electronics engineering Shi Xiehui (IEEE) 1394 interface, network interface, USB interface or flash interface. Such as, the video data received by external interface unit 942 and audio frequency data are input to encoder 943. That is, the transmission unit that external interface unit 942 is used as in record/reproduction device 940.

When not yet the video data inputted from external interface unit 942 and audio frequency data being encoded, video data and audio frequency data are encoded by encoder 943. Encoder 943 exports encoded position stream to selector switch 946.

HDD944 using as the encoded bitstream recording of the content-data of Audio and Video of compression, various program and other data in built-in hard disk. Further, HDD944 reads the data from hard disk when reproducing video and audio frequency.

Data are recorded to the recording medium installed and read this data by hard disk drive 945. The recording medium installed in disc driver 945 is, such as, digital versatile disc (DVD), disk (DVD-video, DVD-random access memory (DVD-RAM), DVD-can record (DVD-R), DVD-can rewrite (DVD-RW), DVD+ can record (DVD+R), DVD+ can rewrite (DVD+RW) etc.), blue light (registered trademark) disk etc.

Selector switch 946 is selected when recording of video and audio frequency from the encoded position stream that tuner 941 or encoder 943 input, and exports the encoded position stream selected to HDD944 or disc driver 945. Further, selector switch 946 exports the encoded position stream inputted from HDD944 or disc driver 945 to demoder 947 when reproducing video and audio frequency.

Encoded position stream is decoded to generate video data and audio frequency data by demoder 947. Then demoder 947 exports the video data of generation to OSD948. Further, demoder 947 exports the audio frequency data of generation to external loudspeaker.

OSD948 reappears the video data inputted from demoder 947, and shows this video. Further, OSD948 can on video to be shown such as, the image of overlapping GUI (menu, button or cursor).

Such as, such as, control unit 949 comprises treater (CPU) and storer (RAM and ROM). Storer stores the program performed by CPU, program data etc. The program in memory that stores is read by CPU when record/reproduction device 940 is started shooting and performs. CPU, such as, by steering routine, controls the operation of record/reproduction device 940 according to the operation signal inputted from user interface elements 950.

User interface elements 950 is connected with control unit 949. User interface elements 950 comprises, and such as, user is used for operating the reception unit etc. of the button of TV device 940 and switch, Long-distance Control signal. The operation that user interface elements 950 detects user by these configuration elements is to generate operation signal, and exports the operation signal of generation to control unit 949.

In record/reproduction device 940 as described above, encoder 943 has the function of the picture coding device 100 according to embodiment described above. That is, RAW data can be encoded by encoder 943 by the method described in the first embodiment or the 2nd embodiment. Further, demoder 947 has the function of the picture decoding apparatus 200 according to embodiment described above. That is, the coded data of the RAW data as the method coding by describing in the first embodiment or the 2nd embodiment correctly can be decoded by demoder 947 by the method described in the first embodiment or the 3rd embodiment. Therefore, record/reproduction device 940 can improve coding efficiency more easily when RAW data being encoded.

Figure 60 illustrates the example of the illustrative configurations of the imaging device applying embodiment described above. Imaging device 960 makes image objects, synthetic image, view data is encoded and by coded picture data record in the recording medium.

Imaging device 960 comprises optics block 961, image-generating unit 962, signal processing unit 963, graphics processing unit 964, display unit 965, outside interface (I/F) unit 966, storage location 967, media-driven device 968, OSD969, control unit 970, user interface (I/F) unit 971 and bus 972.

Optics block 961 is connected to image-generating unit 962. Image-generating unit 962 is connected with signal processing unit 963. Display unit 965 is connected with graphics processing unit 964. User interface elements 971 is connected with control unit 970. Graphics processing unit 964, external interface unit 966, storage location 967, media-driven device 968, OSD969 and control unit 970 are connected to each other by bus 972.

Optics block 961 comprises focusing lens, diaphragm assembly etc. The imaging that the optical imagery of object is focused on image-generating unit 962 by optics block 961 is on the surface. Image-generating unit 962 comprises image sensor (such as, charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS)), and be that figure image signal is as electrical signal using the optical imagery opto-electronic conversion focused on imaging surface. Then image-generating unit 962 exports figure image signal to signal processing unit 963.

The figure image signal inputted from image-generating unit 962 is performed the process of various types of camera signal by signal processing unit 963, such as, and the correction of curvature correction, gray scale and color correction. Signal processing unit 963 exports the view data processed through camera signal to graphics processing unit 964.

The view data inputted from signal processing unit 963 is encoded to generate coded data by graphics processing unit 964. Then the coded data of generation is exported to external interface unit 966 or media-driven device 968 by graphics processing unit 964. Further, the coded data inputted from external interface unit 966 or media-driven device 968 is decoded with synthetic image data by graphics processing unit 964. Then graphics processing unit 964 exports the view data of generation to display unit 965. Further, graphics processing unit 964 can export the view data inputted from signal processing unit 963 to display unit 965 and show image in display unit 965. Further, graphics processing unit 964 can on the image to be output to display unit 965 the overlapping display data got from OSD969.

OSD969 generates, such as, and the image of GUI (such as, menu, button or cursor), and export the image of generation to graphics processing unit 964.

Such as, external interface unit 966 is configured to USB I/O terminal. Such as, external interface unit 966 connects into picture device 960 and printer when print image. Further, if desired, driving mechanism is connected to external interface unit 966. Move media (such as, disk or CD) is installed to driving mechanism, and the program read from move media can be mounted to imaging device 960. Such as, further, external interface unit 966 can be configured to be connected to the network interface of network (LAN or internet). That is, the transmission unit that external interface unit 966 is used as in imaging device 960.

The recording medium being mounted to media-driven device 968 can be any move media can read/can rewrite, such as, and disk, magneto-optic disk, CD or semiconductor memory. Further, recording medium can be mounted to media-driven device 968 in a fixed manner to be configured to the non-transhipment storage unit such as internal hard disk driving mechanism or solid-state device (SSD).

Such as, such as, control unit 970 comprises treater (CPU) and storer (RAM and ROM). Storer stores the program performed by CPU, program data etc. Such as, the program in memory that stores is read by CPU when TV device 960 is started shooting and performs. CPU, such as, by steering routine, is controlled to the operation of picture device 960 according to the operation signal inputted from user interface elements 971.

User interface elements 971 is connected with control unit 970. User interface elements 971 comprises, and such as, user is used for operating button and the switch of imaging device 960. The operation that user interface elements 971 detects user by configuring element is to generate operation signal, and exports the operation signal of generation to control unit 970.

In the imaging device 960 of configuration as described above, graphics processing unit 964 has the function of the picture coding device 100 according to embodiment described above and picture decoding apparatus 200. Namely, RAW data can be encoded by graphics processing unit 964 by the method described in the first embodiment or the 2nd embodiment, and correctly can be decoded by the coded data obtained by encoding by the method described in the first embodiment or the 3rd embodiment. Therefore, imaging device 960 can improve coding efficiency more easily when RAW data being encoded.

Note, this technology can be applied to HTTP streaming transmission, such as, and MPEGDASH, this HTTP stream is selected suitable coded data from pre-prepd multiple coded data and is used this coded data in units of section, and the plurality of coded data has resolving power different from each other. That is, between multiple this kind of coded data, it is possible to share the information relevant with Code And Decode.

Describe the example of device and the system applying this technology. But, this technology is not limited to these examples, and may be implemented as and any it be configured to be installed in this kind of device or be configured in the device of this kind of system, such as, as system large-scale integrated (LSI) treater, use the module of multiple treater, use the unit of multiple module, the set (that is, the configuration of a part for device) by adding other function to unit further and obtain.

With reference to Figure 61, the example when this technology is implemented as set is described. Figure 61 illustrates the example of the illustrative configurations of the video set applying this technology.

In recent years, electronic installation has become multifunction. When a part for its configuration is researched and developed or is implemented as in manufacture listing at it or be on sale, not only it is embodied as the configuration with a kind of function, and there is the correlation function of combination and implement the various configurations of multiple function, be all general.

Video set 1300 shown in Figure 61 is multifunction configuration, and is have the device of the function relevant with the Code And Decode of image (this function can with the one in Code And Decode or two kinds about) and have the combination of the device of another function relevant with this function.

As shown in Figure 61, video set 1300 comprises: module group, and it comprises video module 1311, exterior storage device 1312, power supply management module 1313, front-end module 1314 etc.; And there is the device of correlation function, such as, connect system 1321, photographic camera 1322 and sensor 1323.

Module is the assembly with consistent function, is wherein integrated with the assembly function that some are relevant each other. Concrete physical configuration is arbitrary. But, for example, it is contemplated that other device that there is multiple treaters of corresponding function, electronic circuit component (such as, resistance and electrical condenser) and be arranged on integrated wiring board etc. Further, it may be considered that by the new module being combined to form of module and another module, treater etc.

In the example of Figure 61, video module 1311 is the combination of the configuration with the function relevant with image procossing, and comprises and answer purpose processor, video processing device, broadband modem 1333 and RF module 1334.

Treater is the configuration with predetermined function by system on sheet (SoC) on a semiconductor die integrated, and can be called system large-scale integrated (LSI) etc. The configuration with predetermined function can be logical circuit (Hardware configuration), or the program (software arrangements) that can be CPU, ROM, RAM etc. and perform with the use of aforementioned components, or can be their combination. Such as, treater comprises logical circuit, CPU, ROM, RAM etc., and part of functions can be realized by logical circuit (Hardware configuration), and other function can be realized by the program (software arrangements) performed by CPU.

The purpose processor 1331 of answering of Figure 61 is the treater performing the application relevant with image procossing. Such as, the application performed in purpose processor 1331 is being answered not only to perform computing, if desired, it is also possible to control inner at video module 1311 (video processing device 1332) and outside being configured to and realize predetermined function.

Video processing device 1332 is the treater with the function relevant with the coding/decoding of image (one in this function and Code And Decode or two kinds about).

Broadband modem 1333 digitally modulates the data (numerary signal) by wired or wireless (or both) transmission of broadband communications to convert data to simulating signal, and the simulating signal that demodulation is received by broadband connections to be converted to data (numerary signal) by simulating signal, such as, this broadband connections is performed by broadband line (internet or public telephone spider lines). Broadband modem 1333 processes any information, such as, and the view data handled by video processing device 1332, the stream as coded picture data, application program and setting data.

RF module 1334 is to the module being performed frequency inverted, modulating/demodulating, amplification, filtering process etc. by radio frequency (RF) signal of antenna transmission/receive. Such as, the baseband signal generated in broadband modem 1333 is performed frequency inverted etc. to generate RF signal by RF module 1334. Further, such as, the RF signal that received by front-end module 1314 is performed frequency inverted etc. to generate baseband signal by RF module 1334.

Note, as shown in the dotted line 1341 in Figure 61, answer purpose processor 1331 and video processing device 1332 to be integrated, and be configured to a treater.

Exterior storage device 1312 is the module comprising the storing device that video module 1311 uses, and outside being provided in video module 1311. The storing device of exterior storage device 1312 can be realized by any physical configuration. But, usually, storing device usually for storing the Large Volume Data such as view data in units of frame. Such as, therefore, storing device is desirably realize by the relatively inexpensive large-capacity semiconductor storer of such as dynamic RAM (DRAM).

Power supply management module 1313 manages and the power supply supply (configuration in video module 1311) that controls to video module 1311.

Front-end module 1314 is the module providing front-end functionality (antenna side transmission/reception terminal circuit) to RF module 1334. As shown in Figure 61, front-end module 1314 comprises, such as, and antenna unit 1351, wave filter 1352 and amplification circuit 1353.

Antenna unit 1351 comprises antenna and the periphery configure thereof of transmission/reception wireless signal. Antenna unit 1351 transmits as wireless signal from the signal amplifying unit 1353 offer, and as electrical signal (RF signal), the wireless signal received is supplied to wave filter 1352. The RF signal received by antenna unit 1351 is performed filtering process etc. by wave filter 1352, and treated RF signal is supplied to RF module 1334. Amplify unit 1353 and the RF signal provided from RF module 1334 is provided, and the RF signal after amplifying is supplied to antenna unit 1351.

Connection system 1321 is the module having and connecting relevant function to the external world. The physical configuration of connection system 1321 is arbitrary. Such as, connection system 1321 comprises the configuration with the communication function except the communication standard that broadband modem 1333 is supported, outside I/O terminal etc.

Such as, connection system 1321 can comprise having and meets wireless communication standard, such as, bluetooth (registered trademark), IEEE802.11 are (such as, Wireless Fidelity (Wi-Fi, registered trademark)), the module of the communication function of near-field communication (NFC) or IrDA (Infrared Data Association), or the antenna that transmission/reception meets the signal etc. of this standard can be comprised. Further, such as, connection system 1321 can comprise having and meets wired communications standards, such as, the module of the communication function of general serial bus (USB) or high-definition media interface (HDMI, registered trademark) or meet the terminal of this standard. Further, such as, connection system 1321 can have another data (signal) transfer function, such as, and analog input/output terminal.

Noting, connection system 1321 can comprise the device of the transmission destination of data (signal). Such as, connection system 1321 can comprise driving mechanism and (not only comprise the driving mechanism of move media, also comprise hard disk, solid-state drive (SSD), network is attached gives storage (NAS) etc.), such as, this driving mechanism can read data from recording medium (disk, CD, magneto-optic disk or semiconductor memory) or these data are write to this recording medium. Such as, further, connection system 1321 can comprise the take-off equipment (watch-dog or loud speaker) of image or video.

Photographic camera 1322 is the module of the function with the view data making image objects and obtain this object. The view data obtained by imaging by photographic camera 1322 is supplied to video processing device 1332, such as, and this view data is encoded.

Sensor 1323 be have any sensor function (such as, audio sensor, ultrasonic sensing device, optical sensing device, illumination sensor, infrared sensor, image sensor, turn-sensitive device, angle sensor device, circular frequency sensor, velocity sensor, acceleration transducer, inclination sensor, magnetic identification sensor, collision sensor, temperature sensor) module. Data detected by sensor 1323, such as, be provided to answer purpose processor 1331 and by application wait operation.

The configuration being described as module above may be implemented as treater. In contrast, the configuration being described as treater may be implemented as module.

In the video set 1300 with above configuration, it is possible to video processing device 1332 described below this technology is applied to. Therefore, video set 1300 may be implemented as the set applying this technology.

Figure 62 illustrates the example of the illustrative configurations of the video processing device 1332 (Figure 61) applying this technology.

In example in Figure 62, video processing device 1332 have when the input receiving these signals to the function of the vision signal in reservation system and coding audio signal and to encoded video data and audio frequency decoding data and reappear and the function of output video signal and sound signal.

As shown in Figure 62, video processing device 1332 comprises video input processing unit 1401, the first Nonlinear magnify/reducing unit 1402, two Nonlinear magnify/reducing unit 1403, video frequency output processing unit 1404, frame memory 1405 and memory control unit 1406. Further, video processing device 1332 comprises coding/decoding engine 1407, video ES (substantially flowing) buffer memory 1408A and 1408B and audio ES buffer memory 1409A and 1409B. Further, video processing device 1332 comprises audio coder 1410, audio decoder device 1411, multiplexed device (MUX) 1412, demultiplexer (DMUX) 1413 and stream buffer memory 1414.

Such as, video input processing unit 1401 obtains the vision signal inputted from connection system 1321 (Figure 61) etc., and vision signal is converted to Digital Image Data. View data is performed the zoom in/out etc. of format conversion, image by the first Nonlinear magnify/reducing unit 1402. Two Nonlinear magnify/reducing unit 1403 by video frequency output processing unit 1404 according at the form exported in point of destination, view data being performed the zoom in/out process of image and process format conversion and the zoom in/out process of similar image with the format conversion of the image of the first Nonlinear magnify/reducing unit 1402 and zoom in/out. View data is performed format conversion, conversion etc. to simulating signal by video frequency output processing unit 1404, and using analog signal output to connecting system 1321 grade as the vision signal reappeared.

Frame memory 1405 is for the storer by video input processing unit 1401, the first Nonlinear magnify/reducing unit 1402, two Nonlinear magnify/reducing unit 1403, video frequency output processing unit 1404 and the normally used view data of coding/decoding engine 1407. Such as, frame memory 1405 is implemented as semiconductor memory (DRAM).

Memory control unit 1406, when receiving the synchronizing signal from coding/decoding engine 1407, controls the access of the write to frame memory 1405 or the reading from frame memory 1405 according to the access schedule for frame memory 1405 in write access management table 1406A. Access management table 1406A is upgraded according to the process performed in coding/decoding engine 1407, the first Nonlinear magnify/reducing unit 1402, two Nonlinear magnify/reducing unit 14403 etc. by memory control unit 1406.

Coding/decoding engine 1407 performs the coded treatment of view data and the decoding process of the video flowing as coded picture data. Such as, the view data read from frame memory 1405 is encoded by coding/decoding engine 1407, and view data is sequentially write in video ES buffer memory 1408A as video flowing. Further, such as, coding/decoding engine 1407 sequentially reads video flowing from video ES buffer memory 1408A, and sequentially writes in frame memory 1405 video flowing as view data. Frame memory 1405 is used as work area when Code And Decode by coding/decoding engine 1407. Further, synchronizing signal is exported to memory control unit 1406 by coding/decoding engine 1407 when starting to process each grand piece.

The video flowing that video ES buffer memory 1408A buffer memory coding/decoding engine 1407 generates, and video flowing is supplied to multiplexed device (MUX) 1412. The video flowing that video ES buffer memory 1408B buffer memory is provided by demultiplexer (DMUS) 1413, and video flowing is supplied to coding/decoding engine 1407.

The audio frequency stream that audio ES buffer memory 1409A buffer memory audio coder 1410 generates, and audio frequency stream is supplied to multiplexed device (MUX) 1412. The audio frequency stream that audio ES buffer memory 1409B buffer memory is provided by demultiplexer (DMUS) 1413, and audio frequency stream is supplied to audio decoder device 1411.

Audio coder 1410, such as, the sound signal inputted from connection system 1321 etc. is carried out digital conversion, and to the coding audio signal in reservation system (such as, mpeg audio system or Audiocode numbering 3 (AC3) system). Audio frequency stream as encoded audio signal is sequentially write in audio ES buffer memory 1409A by audio coder 1410. The audio frequency stream provided by audio ES buffer memory 1409B is decoded by audio decoder device 1411, the circulation of this audio frequency is changed to simulating signal, and as the sound signal reappeared, simulating signal is supplied to connection system 1321 etc.

Video flowing and audio frequency stream are carried out multiplexed by multiplexed device (MUX) 1412. Multichannel multiplexing method (that is, by multiplexed and generate position stream form) be arbitrary. Further, when multiplexed, predetermined header information etc. can be added the stream that puts in place by multiplexed device (MUX) 1412. That is, multiplexed device (MUX) 1412 can by the multiplexed form changing stream. Such as, video flowing and audio frequency stream are carried out multiplexed and this video flowing and the circulation of this audio frequency are changed to Transport Stream by multiplexed device (MUX) 1412, and this Transport Stream is the position stream of the form for transmitting. Further, such as, video flowing and audio frequency stream are carried out multiplexed and this video flowing and the circulation of this audio frequency are changed to the data (file data) for the file layout recorded by multiplexed device (MUX) 1412.

Position stream as multiplexed video flowing and audio frequency stream is undertaken separating multiplexed by demultiplexer (DMUS) 1413 by the multiplexed corresponding method that carries out with multiplexed device (MUX) 1412. That is, demultiplexer (DMUS) 1413 extracts video flowing and audio frequency stream (position stream is separated into video flowing and audio frequency stream) from the position stream that stream buffer memory 1414 reads. That is, demultiplexer (DMUS) 1413 can change the form of stream by separating multiplexed (inverse transformation conversion carried out by multiplexed device (MUX) 1412). Such as, demultiplexer (DMUS) 1413 obtains, by stream buffer memory 1414, the Transport Stream provided from connection system 1321, wideband codec 1333 etc., and can by carrying out Transport Stream separating multiplexed video flowing and the audio frequency stream of Transport Stream being converted to. Further, such as, demultiplexer (DMUS) 1413 by stream buffer memory 1414 obtain by, such as, the file data that connection system 1321 reads from various recording medium, and can by carrying out Transport Stream separating multiplexed video flowing and the audio frequency stream of Transport Stream being converted to.

Stream buffer memory 1414 cache bit stream. Such as, the stream Transport Stream that provides from multiplexed device (MUX) 1412 of buffer memory 1414 buffer memory, and in predetermined timing place or based on the request from the external world etc., Transport Stream is supplied to connection system 1321, wideband codec 1333 etc.

Further, such as, the stream file data that provides from multiplexed device (MUX) 1412 of buffer memory 1414 buffer memory, is supplied to file data connection system 1321 etc. in predetermined timing place or based on the request from the external world etc., and is recorded in various recording medium by file data.

Further, the stream Transport Stream that got by connection system 1321, wideband codec 1333 etc. of buffer memory 1414 buffer memory, and in predetermined timing place or based on the request from the external world etc., Transport Stream is supplied to demultiplexer (DMUS) 1413.

Further, the file data that stream buffer memory 1414 buffer memory reads from the various recording medium connected system 1321 grade, and in predetermined timing place or based on the request from the external world etc., file data is supplied to demultiplexer (DMUS) 1413.

Next, the example of the operation of the video processing device 1332 with this kind of configuration will be described. Such as, in video input processing unit 1401, the vision signal being input to video processing device 1332 from connection system 1321 etc. is converted to the Digital Image Data in reservation system (such as, 4:2:2Y/Cb/Cr system), and it is sequentially write to frame memory 1405. Digital Image Data is read by the first Nonlinear magnify/reducing unit 1402 or two Nonlinear magnify/reducing unit 1403, perform by 4:2:0Y/Cb/Cr system to the format conversion of reservation system and zoom in/out process, and data are write to again frame memory 1405. View data is encoded by coding/decoding engine 1407, and is written into video ES buffer memory 1408A as video flowing.

Further, the sound signal being input to video processing device 1332 from connection system 1321 etc. is encoded by audio coder 1410, and is written into audio ES buffer memory 1409A as audio frequency stream.

Video flowing in video ES buffer memory 1408A and the audio frequency stream in audio ES buffer memory 1409A are read with multiplexed by multiplexed device (MUX) 1412, and are converted into Transport Stream, file data etc. The Transport Stream that multiplexed device (MUX) 1412 generates is buffered in stream buffer memory 1414, and then by connection system 1321, wideband codec 1333 etc. to this Transport Stream is exported outside network. Further, the file data that multiplexed device (MUX) 1412 generates is buffered in stream buffer memory 1414, exports connection system 1321 etc. to, and be recorded in various recording medium.

Further, it is buffered in by connection system 1321, wideband encoding modulator 1333 etc. stream buffer memory 1414 from the Transport Stream that outside network is input to video processing device 1332, and then is undertaken by demultiplexer (DMUX) 1413 separating multiplexed. Further, such as, the file data that will read from the various recording medium in connection system 1321 grade and be input to video processing device 1332 is buffered in stream buffer memory 1414, and then is undertaken separating multiplexed by demultiplexer (DMUX) 1413. That is, by demultiplexer (DMUX) 1413, the Transport Stream or file data that are input to video processing device 1332 are separated into video flowing and audio frequency stream.

There is provided audio frequency stream by audio ES buffer memory 1409B to audio decoder device 1411, and it is decoded, thus sound signal is reappeared. Further, video flowing is write to video ES buffer memory 1408B, then sequentially reads this video flowing by coding/decoding engine 1407 and it is decoded, and write to frame memory 1405. By two Nonlinear magnify/reducing unit 1403, decode image data is carried out zoom in/out process, write to frame memory 1405. Then, read decoded picture by video frequency output processing unit 1404, it is reservation system by its format conversion, such as, 4:2:2Y/Cb/Cr system, and it is converted to further simulating signal, thus make vision signal reproduction and export this vision signal.

When this technology is applied to the video processing device 1332 of configuration as described above, can only be applied to coding/decoding engine 1407 according to this technology of embodiment. Such as, that is, coding/decoding engine 1407 can only have the function of the picture coding device 100 according to embodiment and picture decoding apparatus 200. Like this, video processing device 1332 can have the effect similar with the effect described above with reference to Fig. 1 to Figure 52.

Note, in coding/decoding engine 1407, this technology is (namely, picture coding device according to embodiment and the function of picture decoding apparatus) can by hardware (such as, logical circuit) realize, can realize by software (such as, consolidation procedure), or can realize by both software and hardwares.

Figure 63 illustrates another example of the illustrative configurations of the video processing device 1332 applying this technology. In the example of Figure 63, video processing device 1332 has the function to the coding video data/decoding in reservation system.

Specifically, as shown in Figure 63, video processing device 1332 comprises control unit 1511, display interface 1512, display engine 1513, image processing engine 1514 and storage inside device 1515. Further, video processing device 1332 comprises encoding and decoding engine 1516, storer interface 1517, multiplexer/demultiplexer (MUXDMUX) 1518, network interface 1519 and video interface 1520.

Such as, control unit 1511 controls the operation of the processing unit in video processing device 1332 (showing interface 1512, display engine 1513 and image processing engine 1514) and encoding and decoding engine 1516.

As shown in Figure 63, control unit 1511 comprises main CPU1531, sub-CPU1532 and central controller 1533. Main CPU1531 performs the program etc. of the operation for the processing unit controlled in video processing device 1332. Main CPU1531 generates control signal according to program etc., and control signal is supplied to processing unit (that is, the operation of control treatment unit). Main CPU1531 is played booster action by sub-CPU1532. Such as, sub-CPU1532 performs sub-process or the subroutine of the program etc. performed by main CPU1531. Central controller 1533 controls main CPU1531 and the operation of sub-CPU1532, such as, specifies the program performed by main CPU1531 and sub-CPU1532.

Display interface 1512 exports view data to connection system 1321 etc. under the control of control unit 1511. Such as, the view data of numerical data is converted to simulating signal by display interface 1512, and using analog signal output to the vision signal of monitor apparatus as reproduction connecting system 1321 grade, or intactly as the view data of numerical data.

Display engine 1513 is treated the view data that the hardware specification with monitor apparatus mates under the control of control unit 1511 and is performed various types of conversion process, such as, the conversion of format conversion, size, color region conversion etc., show the image of view data on this monitor apparatus.

View data application predetermined image is processed by image processing engine 1514 under the control of control unit 1511, such as, for improving the filtering process of picture quality.

The storer that storage inside device 1515 is to provide in video processing device 1332, and usually use by display engine 1513, image processing engine 1514 and encoding and decoding engine 1516. The data that storage inside device 1515 performs for being transmitted between display engine 1513, image processing engine 1514 and encoding and decoding engine 1516. Such as, storage inside device 1515 stores the data provided from display engine 1513, image processing engine 1514 and encoding and decoding engine 1516, and, if desired (such as, in response to request), data are supplied to display engine 1513, image processing engine 1514 or encoding and decoding engine 1516. Storage inside device 1515 can be realized by any storing device. But, because storage inside device 1515 is generally used for storing little capacity data in units of block, such as, view data or parameter, so storage inside device 1515 is desirably by having little capacity but have high response speed (such as, compared with exterior storage device 1312) semiconductor memory realize, such as static RAM (SRAM).

Encoding and decoding engine 1516 performs the coding with view data or decodes relevant process. The system of the coding/decoding that encoding and decoding engine 1516 is supported is arbitrary, and can adopt one or more system. Such as, encoding and decoding engine 1516 can have the codec functions of multiple coding/decoding system, and by view data being encoded from the system wherein selected or coded data is decoded.

In the example shown in Figure 63, encoding and decoding engine 1516 comprises, such as, functional block as the process relevant with codec of MPEG-2 video 1541, AVC/H.2641542, HEVC/H.2651543, HEVC/H.265 (can expand) 1544, HEVC/H.265 (multi views) 1545 and MPEG-DASH1551.

MPEG-2 video 1541 is the functional block that the view data in MPEG-2 system carries out Code And Decode. AVC/H.2641542 is the functional block that the view data in AVC system carries out Code And Decode. HEVC/H.2651543 is the functional block that the view data in HEVC system carries out Code And Decode. It is the functional block that the view data in HEVC system can carry out Code And Decode with expanding that HEVC/H.265 (can expand) 1544. View data multi views in HEVC system is carried out the functional block of Code And Decode by HEVC/H.265 (multi views) 1545.

MPEG-DASH1551 is the functional block by the view data in the dynamic self-adaptation streaming of HTTP (MPEG-DASH) system transfers/reception MPEG. MPEG-DASH is the streamed technology for performing video with the use of HTTP (HTTP), and it is characterized in that: from pre-prepd in units of section and have and multiple coded datas of resolving power different from each other are selected a suitable coded data, and transmit the coded data of this selection. MPEG-DASH1551 control meets the generation of stream and the transmission of stream of this standard, and uses MPEG-2 video 1541 to HEVC/H.265 (multi views) 1545 described above to carry out the coding/decoding of view data.

Storer interface 1517 is the interface for exterior storage device 1312. By storer interface 1517, the data provided from image processing engine 1514 and encoding and decoding engine 1516 are supplied to exterior storage device 1312. Further, by storer interface 1517, the data read from exterior storage device 1312 are supplied to video processing device 1332 (image processing engine 1514 or encoding and decoding engine 1516).

Multiplexer/demultiplexer (MUXDMUX) 1518 is to carry out multiplexed reconciliation multiplexed with image-related various data (encoded data bits stream, view data and vision signal). Carry out multiplexed/to separate multiplexed method be arbitrary. Such as, when carrying out multiplexed, multiple data integration is not only data by multiplexer/demultiplexer (MUXDMUX) 1518, it is also possible to add to the data such as predetermined header information. Further, when carrying out separating multiplexed, data are not only divided into multiple data by multiplexer/demultiplexer (MUXDMUX) 1518, it is also possible to predetermined header information etc. is added to the data after each segmentation. That is, multiplexer/demultiplexer (MUXDMUX) 1518 can by multiplexed/separate multiplexed come translation data form. Such as, it is Transport Stream by the format conversion of data that multiplexer/demultiplexer (MUXDMUX) 1518 can be undertaken multiplexed by contraposition stream, and this Transport Stream transmits the position stream of form or the data (file data) of file layout for recording. Obviously, multiplexed by separating, inverse transformation is possible.

Network interface 1519 is the interface for broadband modem 1333, connection system 1321 etc. Video interface 1520 is the interface for connecting system 1321, photographic camera 1322 etc.

Next, the example of the operation of video processing device 1332 will be described. Such as, when receiving the transform stream from outside network by connection system 1321, broadband modem 1333 etc., by network interface 1519 Transport Stream is supplied to multiplexer/demultiplexer (MUXDMUX) 1518 and carries out this Transport Stream separating multiplexed, and by encoding and decoding engine 1516, this Transport Stream is decoded. The view data decoding undertaken by encoding and decoding engine 1516 obtained by image processing engine 1514 carries out predetermined image process, by display engine 1513, it is carried out predetermined map, and provide it to connection system 1321 etc. by display interface 1512, and its image is shown on a monitor. Further, such as, the view data decoding undertaken by encoding and decoding engine 1516 obtained by encoding and decoding engine 1516 carries out recompile, by multiplexer/demultiplexer (MUXDMUX) 1518, it is carried out multiplexed and it is converted into file data, exported to connection system 1321 etc. by video interface 1520, and it is recorded in various recording medium.

Further, such as, by video interface 1520, the file data of the coded data as the coded picture data read from recording medium (not shown) by connection system 1321 grade is supplied to multiplexer/demultiplexer (MUXDMUX) 1518 and this file data carries out solution multiplexed, and by encoding and decoding engine 1516, it is decoded. The view data decoding undertaken by encoding and decoding engine 1516 obtained by image processing engine 1514 carries out predetermined image process, by display engine 1513, it is carried out predetermined map, and provide it to connection system 1321 etc. by display interface 1520, and its image is shown on a monitor. Further, such as, the view data decoding undertaken by encoding and decoding engine 1516 obtained by encoding and decoding engine 1516 carries out recompile, by multiplexer/demultiplexer (MUXDMUX) 1518, it is carried out multiplexed and it is converted into Transport Stream, provide it to connection system 1321, broadband modem 1333 etc. by network interface 1519, and it is sent to another device (not shown).

Note, with the use of, such as, storage inside device 1515 and exterior storage device 1312 perform to transmit view data and other data between the processing unit in video processing device 1332. Further, power supply management module 1313, such as, controls the power supply supply to control unit 1511.

When this technology is applied to the video processing device 1332 of configuration as described above, can only be applied to encoding and decoding engine 1516 according to this technology of embodiment. Such as, that is, encoding and decoding engine 1516 can only comprise realization according to the picture coding device 100 of embodiment described above and the functional block of picture decoding apparatus 200. Like this, video processing device 1332 can obtain the effect similar with the effect described above with reference to Fig. 1 to Figure 52.

In encoding and decoding engine 1516, this technology is (namely, picture coding device according to embodiment and the function of picture decoding apparatus) can by hardware (such as, logical circuit) realize, can by software (such as, the program being incorporated to) realize, or can realize by both software and hardwares.

Two examples of the configuration of video processing device 1332 are described. But, the configuration of video processing device 1332 is arbitrary, and can adopt the configuration except above two examples. Further, video processing device 1332 can be configured to a semi-conductor chip, or can be configured to multiple semi-conductor chip. Such as, video processing device 1332 can be three-dimensional laminated LSI, and in the LSI that this is three-dimensional laminated, layer is pressed with multiple semi-conductor. Further, video processing device 1332 can be realized by multiple LSI.

Video set 1300 can be incorporated in the various devices of image data processing. Such as, video set 1300 can be incorporated in TV device 900 (Figure 57), mobile phone device 920 (Figure 58), record/reproduction device 940 (Figure 59), imaging device 960 (Figure 60) etc. By being incorporated to of video set 1300, device can obtain the effect similar with the effect described above with reference to Fig. 1 to Figure 52.

The part configuration of video set 1300 may be implemented as the configuration applying this technology, as long as this configuration comprises video processing device 1332. Such as, only video processing device 1332 may be implemented as the video processing device applying this technology. Further, such as, may be implemented as, by the illustrated treater of dotted line 1341, video module 1311 etc., the treater or module that apply this technology. Further, such as, video module 1311, exterior storage device 1312, power supply management module 1313 and front-end module 1314 are combined, and may be implemented as the video unit 1361 applying this technology. In any one configuration, it is possible to obtain the effect similar with the effect described above with reference to Fig. 1 to Figure 52.

That is, any configuration can be incorporated in the various devices of image data processing, as long as this configuration comprises video processing device 1332, is similar to the situation of video set 1300. Such as, video processing device 1332, can be incorporated to by the illustrated treater of dotted line 1341 and video unit 1361 in TV device 900 (Figure 57), mobile phone device 920 (Figure 58), record/reproduction device 940 (Figure 59), imaging device 960 (Figure 60) etc. Further, by applying being incorporated to of any configuration of this technology, device can obtain the effect similar with the effect described by Fig. 1 to Figure 52, is similar to the situation of video set 1300.

Further, in this manual, the example various types of information being multiplexed as encoded stream and it is transferred to from coding side decoding side has been described. But, it is not limited to this example for transmitting the technology of information. Such as, it is possible to transmit this information or be recorded as and the independent data that encoded position stream is associated, this information is not multiplexed as encoded position stream. Herein, term " association " instigates the image (can be a part for image, such as, cut into slices or block) comprised in stream in place and the information corresponding with image can contact together when decoding. Namely, it is possible to upload transmission information at the independent transmission line different from the transmission line for image (or position stream). Further, it is possible to record the information in the independent recording medium (or the independent posting field in identical recording medium) different from the recording medium for image (or position stream). Such as, further, information and image (or position stream) can be associated with each other with arbitrary unit (in units of the part in multiple frame, a frame or frame).

Noting, this technology can adopt following configuration.

(1) a kind of image processing equipment, this image processing equipment comprises:

Packing processing unit, this packing processing unit is configured to perform packing process: according to the degree of correlation, rearranges as the pixel data in the RAW data performing the view data before demosaicing processes; And

Coding unit, this coding unit is configured to the RAW data to the packing process through performing by packing processing unit and encodes.

(2) according to (1) and (3) to the image processing equipment according to any one of (9), wherein, this packing processing unit comprises:

Separating unit, this separating unit is separated the pixel data of RAW data to have one by one based on the data of high correlation,

Rearranging unit, this rearranges unit and rearranges by separating unit to have the pixel data group of separation based on the data of high correlation one by one, and

Generating unit, this generation unit generates the package information of relevant packing process.

(3) according to (1), (2) and (4) to the image processing equipment according to any one of (9), wherein, this separating unit carrys out separate pixel data based on individual element, and this pixel is assigned the wave filter of identical type.

(4) according to (1) to (3) and (5) to the image processing equipment according to any one of (9), wherein, this separating unit carrys out separate pixel data based on individual element, and this pixel is assigned the color filter of same color.

(5) according to (1) to (4) and (6) to the image processing equipment according to any one of (9), wherein, this rearranges unit and rearranges the component of pixel data group as predetermined color space, and

Component is encoded by this coding unit together or independently of one another.

(6) according to (1) to (5) and (7) to the image processing equipment according to any one of (9), wherein, component is encoded by this coding unit based on the region of part one by one of the image of RAW data.

(7) according to (1) to (6) and (8) to the image processing equipment according to any one of (9), wherein, this rearranges unit and rearranges the different from each other part region of pixel data group as one or more image.

(8) according to the image processing equipment according to any one of (1) to (7) and (9), wherein, this rearranges unit and rearranges the data of pixel data group as the layering different from each other of stratification view data.

According to (1) to the image processing equipment according to any one of (8), wherein, (9) this rearranges unit and the view data processed through demosaicing is re-arranged to part layering.

(10) a kind of image processing method, this image processing method comprises:

Execution packing processes: according to the degree of correlation, rearranges as the pixel data in the RAW data performing the view data before demosaicing processes; And

The RAW data processed through packing are encoded.

(11) a kind of image processing equipment, this image processing equipment comprises:

Decoding unit, this decoding cell location is decoded by the coded data being the encoded RAW data as the view data before performing demosaicing process; And

Separate bag processing unit, this solution bag processing unit is configured to perform Xie Bao process: for the RAW data of the packing process through rearranging pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, the decoding that these RAW data are undertaken by decoding unit obtains.

(12) according to (11) and (13) to the image processing equipment according to any one of (19), wherein, this packing processing unit comprises:

Resolution unit, this resolution unit resolves the package information of relevant packing process;

Separating unit, this separating unit resolves the pixel data of the RAW data that the package information separation obtained processes through packing based on resolution unit; And

Rearranging unit, this rearranges unit and resolves, based on resolution unit, the package information that obtains and return to have the pixel data of separation based on the data of high correlation one by one by separating unit and to perform the layout before packing processes.

(13) according to (11), (12) and (14) to the image processing equipment according to any one of (19), wherein, this separating unit is separated the pixel data rearranged based on individual element, this pixel is assigned the wave filter of identical type by packing process, and

Rearrange unit and rearrange pixel data according to the array of wave filter.

(14) according to (11), (13) and (15) to the image processing equipment according to any one of (19), wherein, this separating unit is separated the pixel data rearranged based on individual element, this pixel is assigned the color filter of same color by packing process, and

Rearrange unit and rearrange pixel data according to the array of color filter.

(15) according to (11) to (14) and (16) to the image processing equipment according to any one of (19), wherein, this separating unit is to have the pixel data being separated based on the data of high correlation and rearranging the component into predetermined color space one by one.

(16) according to (11) to (15) and (17) to the image processing equipment according to any one of (19), wherein, coded data is decoded by this decoding unit based on part region one by one, and this coded data is the encoded component based on the region of part one by one of the image of RAW data.

(17) according to (11) to (16) and (18) to the image processing equipment according to any one of (19), wherein, this separating unit is to have the pixel data being separated based on the data of high correlation and rearranging in the part region different from each other of one or more image one by one.

(18) according to the image processing equipment according to any one of (11) to (17) and (19), wherein, this separating unit is to have the pixel data being separated based on the data of high correlation and rearranging in the layering different from each other of stratification view data one by one.

(19) according to (11) to the image processing equipment according to any one of (18), wherein, the separation of this separating unit rearranges the pixel data in other layering except part layering, and the view data through demosaicing process is arranged on the portion.

(20) a kind of image processing method, this image processing method comprises:

The coded data being the encoded RAW data as the view data before performing demosaicing process is decoded; And

Perform Xie Bao process: for the RAW data of the packing process through rearranging pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, these RAW data obtain by decoding.

Element symbol list:

100 picture coding devices

101 camera sensor

102 packaged units

103 image coding unit

104 transmission unit

105RAW data encoding unit

111 separating units

112 rearrange unit

113 package information generate unit

121 setting units

122 coding units

131 base layer image coding units

132 enhancement layer image coding units

133 multiplexed devices

134 control units

200 picture decoding apparatus

201 reception unit

202 image decoding unit

203 unwrapper unit

204 development treatment unit

211 decoding unit

212 package information extraction units

221 demultiplexers

222 base layer image decoding unit

223 enhancement layer image decoding unit

224 control units

271 package information resolution unit

272 separating units

273 rearrange unit

Claims

1. an image processing equipment, described image processing equipment comprises:

Packing processing unit, described packing processing unit is configured to perform packing process: according to the degree of correlation, rearranges as the pixel data in the RAW data performing the view data before demosaicing processes; And

Coding unit, described coding unit is configured to the described RAW data to the described packing process through performing by described packing processing unit and encodes.

2. image processing equipment according to claim 1, wherein, described packing processing unit comprises:

Separating unit, described separating unit is separated the described pixel data of described RAW data to have one by one based on the data of high correlation,

Rearrange unit, described in rearrange unit and rearrange by described separating unit to have based on the data of high correlation the pixel data group of separation one by one, and

Generating unit, described generation unit generates the package information about described packing process.

3. image processing equipment according to claim 2, wherein, described separating unit is separated described pixel data based on individual element, and described pixel is assigned the wave filter of identical type.

4. image processing equipment according to claim 3, wherein, described separating unit is separated described pixel data based on individual element, and described pixel is assigned the color filter of same color.

5. image processing equipment according to claim 2, wherein, described in rearrange unit and rearrange the component of described pixel data group as predetermined color space, and

Described component is encoded by described coding unit together or independently of one another.

6. image processing equipment according to claim 5, wherein, described component is encoded by described coding unit based on the region of part one by one of the image of described RAW data.

7. image processing equipment according to claim 2, wherein, described in rearrange unit and rearrange the different from each other part region of described pixel data group as one or more image.

8. image processing equipment according to claim 2, wherein, described in rearrange unit and rearrange the data of described pixel data group as the layering different from each other of stratification view data.

9. image processing equipment according to claim 8, wherein, described in rearrange unit the described view data processed through described demosaicing is re-arranged to a part layering.

10. an image processing method, described image processing method comprises:

The described RAW data processed through described packing are encoded.

11. 1 kinds of image processing equipments, described image processing equipment comprises:

Decoding unit, described decoding cell location is decoded by the coded data being the encoded RAW data as the view data before performing demosaicing process; And

Separate bag processing unit, described solution bag processing unit is configured to perform Xie Bao process: for the described RAW data of the described packing process through rearranging described pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, the decoding that described RAW data are undertaken by described decoding unit obtains.

12. image processing equipments according to claim 11, wherein, described solution bag processing unit comprises:

Resolution unit, described resolution unit resolves the package information about described packing process;

Separating unit, described separating unit resolves the described pixel data of the described RAW data that the described package information separation obtained processes through described packing based on described resolution unit; And

Rearrange unit, described in rearrange unit and resolve, based on described resolution unit, the described package information that obtains and return to have the described pixel data of separation based on the data of high correlation one by one by described separating unit and to perform the layout before described packing processes.

13. image processing equipments according to claim 12, wherein, described separating unit is separated the described pixel data rearranged based on individual element, and described pixel is assigned the wave filter of identical type by described packing process, and

The described unit that rearranges rearranges described pixel data according to the array of described wave filter.

14. image processing equipments according to claim 13, wherein, described separating unit is separated the described pixel data rearranged based on individual element, and described pixel is assigned the color filter of same color by described packing process, and

The described unit that rearranges rearranges described pixel data according to the array of described color filter.

15. image processing equipments according to claim 12, wherein, described separating unit is to have the described pixel data being separated based on the data of high correlation and rearranging the component into predetermined color space one by one.

16. image processing equipments according to claim 15, wherein, described coded data is decoded by described decoding unit based on part region one by one, and described coded data is the encoded component based on the region of part one by one of the image of described RAW data.

17. image processing equipment according to claim 12, wherein, described separating unit is to have the described pixel data being separated based on the data of high correlation and rearranging in the part region different from each other of one or more image one by one.

18. image processing equipments according to claim 12, wherein, described separating unit is to have the described pixel data being separated based on the data of high correlation and rearranging in the layering different from each other of stratification view data one by one.

19. image processing equipments according to claim 18, wherein, the separation of described separating unit rearranges the described pixel data in other layering except part layering, and the described view data processed through described demosaicing is arranged on the mentioned parts.

20. 1 kinds of image processing methods, described image processing method comprises:

Perform Xie Bao process: for the described RAW data of the described packing process through rearranging described pixel data according to the degree of correlation, being returned to by pixel data and performing the layout before packing processes, described RAW data obtain by decoding.