CN102986221B - Picture coding device and integrated circuit thereof and method for encoding images - Google Patents

Abstract

When encoding an image, blurring can be applied to the image according to the distance of the subject, and a block size as a unit of encoding can be appropriately determined. Blurring is applied to the image using disparity information per pixel unit or distance information per pixel unit of the input image, and an arbitrary blurring condition, and the block size of the encoding target area is determined according to the blurring amount of each unit area, The image is divided into encoding target areas according to the determined block size and encoded. By increasing the block size of an image region with a large amount of blur, it is possible to reduce the amount of search processing for a reference block, thereby speeding up processing and reducing the amount of processing.

Description

Image encoding device, integrated circuit thereof, and image encoding method

technical field

The present invention relates to an image encoding technique for encoding an image by adding blur.

Background technique

As processing for expressing a three-dimensional effect that makes the main subject stand out, a process of focusing only on a part of the subject of the image and intentionally blurring the other foreground and background is performed. In addition, in a stereoscopic video signal for stereoscopic viewing, when the viewer's attention is directed to the distant view, the viewer feels strong fatigue or the problem of stereoscopic damage occurs (see, for example, Non-Patent Document 1). In order to prevent this problem, a process of blurring the background of the stereoscopic video signal so that the viewer's attention is not drawn to the background is performed.

On the other hand, in image coding, in order to remove collective redundancy and improve compression coding efficiency, a large number of pixels are collectively divided into blocks. Therefore, in the predictive coding that utilizes the inter-block correlation to increase the compression rate, if inter-frame prediction and intra-frame prediction are performed on a complex image in small block units, and frame prediction is performed on a flat image in large block units, Both complex images and flat images can be efficiently predicted by inter-prediction and intra-frame prediction.

Patent Document 1 is disclosed as a conventional example of an encoding device that changes a block size according to the characteristics of an encoding target region. In general, in an ultrasonic diagnostic apparatus, the closer to the focus position, the higher the image resolution, and the farther away from the focus position, the lower the image resolution. Therefore, by determining the block size using the distance information and the focus position information, both high-quality image quality in a high-resolution area and high compression in a low-resolution area are achieved.

In addition, Patent Document 2 is disclosed as another conventional example. In the technique of Patent Document 2, the predicted block size is increased to generate a compressed image in which flicker is suppressed, focusing on the fact that a static region with a high flatness is likely to flicker.

prior art literature

patent documents

Patent Document 1: JP-A-2007-289556

Patent Document 2: JP-A-2007-67469

non-patent literature

Non-Patent Document 1: Tesho Honda "Stereoscopic Image Technology—The Latest Trend of Spatial Expression Media—" CMC Publishing, July 31, 2008 (P61-P62)

Summary of the invention

The problem to be solved by the invention

In the technique of Patent Document 1, the block size for encoding is determined based on distance information and focus position information. However, for various blurring processes, the block size determined only based on the focus position information and the distance information is not necessarily a block size suitable for the blurred image.

On the other hand, in the technique of Patent Document 2, the block size is determined according to the flatness of the image area. However, the block size determined only by the determination of flatness is not necessarily a block size suitable for the blurred image.

Contents of the invention

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image encoding device and an image encoding method for encoding by selecting an appropriate block size after applying various blurring processes based on disparity information or distance information.

means of solving problems

In order to solve the above-mentioned conventional problems, the image encoding device of the present invention is characterized in that it includes: an image acquisition unit that acquires an image; at least one of the above; the blur amount determination unit determines the blur amount of each pixel unit of the above-mentioned image according to the parallax of each pixel unit or the distance information of each pixel unit; the blur processing unit determines the above-mentioned The image is subjected to blurring processing; the block size determination unit determines, based on the amount of blurring, a block size for cutting out the encoding target area from each image after the blurring process is performed from among a plurality of block sizes; and the encoding unit determines according to the above-mentioned The block size of , the image after the above-mentioned blurring process is encoded in block units.

Invention effect

According to this configuration, when encoding an image, the block size is determined using the amount of blur, so that an appropriate block size according to the amount of blur can be used for blur processing.

Description of drawings

FIG. 1 is a block diagram of an image encoding device according to Embodiment 1 of the present invention.

FIG. 2 is a diagram showing an imaging optical system and a blur amount according to Embodiment 1 of the present invention.

FIG. 3 is a diagram showing an example of a fuzzy matrix according to Embodiment 1 of the present invention.

4 is a flowchart showing the operation of the image coding device according to Embodiment 1 of the present invention.

5 is a flowchart showing the operation of generating an encoding target signal in the image encoding device according to Embodiment 1 of the present invention.

6 is a flowchart of an operation of block size determination in intra prediction in the image coding device according to Embodiment 1 of the present invention.

7 is a flowchart showing the operation of block size determination in motion compensation prediction in the image coding device according to Embodiment 1 of the present invention.

8 is a flowchart of an operation of block size determination in parallax compensation prediction in the image coding device according to Embodiment 1 of the present invention.

FIG. 9 is a block diagram of an image encoding device according to Modification 1 of Embodiment 1 of the present invention.

Fig. 10 is a block diagram of an image encoding device according to Embodiment 2 of the present invention.

11 is a flowchart showing the operation of the image encoding device according to Embodiment 2 of the present invention.

FIG. 12 is a flowchart showing the block size determination operation of the image coding device according to Embodiment 2 of the present invention.

FIG. 13 is a block diagram of an image encoding device according to Embodiment 3 of the present invention.

FIG. 14 is a flowchart showing the operation of the image encoding device according to Embodiment 3 of the present invention.

Fig. 15 is a block diagram of an image encoding device according to Embodiment 4 of the present invention.

16 is a diagram showing a positional relationship between a subject, a focal point, and an imaging element according to Embodiment 4 of the present invention.

Fig. 17 is a block diagram of an image coding system according to Embodiment 5 of the present invention.

FIG. 18 is a block diagram of an image encoding device as an integrated circuit according to Embodiment 1 of the present invention.

FIG. 19 is a block diagram of an ultrasonic diagnostic apparatus including a conventional encoding unit.

Fig. 20 is a flowchart illustrating a conventional coding control method.

detailed description

<The process of completing the invention>

As one of image processing, processing of blurring based on the distance of the subject is an important technique. For example, as processing for making the main subject stand out and expressing a three-dimensional effect, only a part of the subject of the image is brought into focus, and the other foreground and background are intentionally blurred. In such blurring processing, the foreground and background may be blurred by reproducing “bokeh” that occurs when shooting with an optical system with a shallow subject depth. In addition, there are various processes such as changing the blur strength between blurring for the background and blurring for the foreground, or focusing on multiple people at different distances and making the foreground of the closest person Blur the background of the person and the person in the innermost side, or focus on both the person and the building behind it to make both the person and the building stand out, and blur the other subjects.

In addition, in the stereoscopic video signal for stereoscopic viewing, as described in Non-Patent Document 1, when the viewer's attention is directed to the distant view, if the distance indicated by the parallax information does not match the focal distance, the viewer will feel that the Strong fatigue, or problems such as loss of stereoscopic vision. In particular, when a stereoscopic video signal produced on the premise of being projected on a small screen is enlarged and projected on a larger screen, the distance between the right eye and the left eye assumed by the stereoscopic video signal is also enlarged. Therefore, for one subject, when the distance between the subject image in the image for the right eye and the image of the subject in the image for the left eye becomes larger than the distance between the right eye and the left eye (backward divergence) , the parallax of the subject cannot be the parallax state when seeing a single object, so the stereoscopic vision is broken. In order to prevent this problem, a process of blurring the background of the stereoscopic video signal so that the viewer's attention is not drawn to the background is performed.

On the other hand, when encoding an image, many pixels are collectively divided into blocks in order to remove collective redundancy and improve compression encoding efficiency. Therefore, in the predictive coding that utilizes the inter-block correlation to increase the compression rate, if inter-frame prediction and intra-frame prediction are performed on a complex image in small block units, and frame prediction is performed on a flat image in large block units, Both complex images and flat images can be efficiently predicted by inter-prediction and intra-frame prediction.

Here, the inventors paid attention to the fact that the region to which strong blur is applied is a flat region without a complicated pattern, which is suitable for a large block size. In addition, the following idea was obtained: if an area subject to strong blurring can be obtained without analyzing the blurred image, it is not necessary to select the block size of the area when encoding the blurred image. selection, it is possible to reduce the computational load for encoding. This is because, in this way, it is not necessary to perform a block size selection process of re-detecting an area subject to strong blurring from a blurred image.

On the other hand, in the conventional block size selection technology, the image to be coded itself is analyzed. Therefore, when the conventional block size selection technique is applied to an image before blurring processing, there is a problem that the selected block size is not always suitable for encoding of the image after blurring processing.

Patent Document 1 is an ultrasonic diagnostic apparatus for selecting a block size using distance information and focus information, and FIG. 19 is a block diagram of the ultrasonic diagnostic apparatus of Patent Document 1. This ultrasonic diagnostic apparatus encodes a reflected signal as depth information as input image data. Since the image resolution of an ultrasonic diagnostic device is determined by the distance from the focus position, the block size is determined using the distance information and the focus position information, thereby achieving both high image quality and resolution in areas with high resolution. High compression of lower areas.

However, even if the distance and focus position are determined, as long as the contents of the blurring process are not determined, the intensity of blurring will not be uniquely determined. That is, in the case of blurring the same subject that is not in focus in the same image, in the case of blurring only the foreground at the focus position, and in the case of blurring only the background at the focus position Of course, the area of the subject in the blurred image is different between the former case and the latter case. That is, even if the image before the blurring process is the same, the appropriate block size for coding the blurred image is not necessarily the same as long as the content of the blurring process is different. Therefore, in the method of Patent Document 1 in which the block size is selected only based on the distance and the focus position for various blurring processes, it is not always possible to obtain a block size suitable for the blurred image.

Furthermore, Patent Document 2 is an encoding control method for suppressing flicker based on the flatness of an image, and FIG. 20 is a flowchart showing the encoding control method of Patent Document 2. In the technique of Patent Document 2, it is determined whether or not it is a still area, and then, for an area determined to be a still area, the degree of flatness of the area is calculated from the dispersion of pixel values or the like. Then, based on the calculated flatness and quantization step size, the degree of flicker is judged, and the block size of a region with a high degree of flicker is increased to generate a compressed image with suppressed flicker.

However, as mentioned above, even flat areas of complex images will fit into smaller block sizes. A larger block size is suitable for areas that have been blurred strongly, but since there may be flat areas with complex patterns in areas that are not blurred, the block size is determined only by whether the area is flat or not. In the technique of Patent Document 2, an appropriate block size cannot be selected. Furthermore, in the technique of Patent Document 2, it is not possible to detect a strongly blurred region that occurs after the blurring process and is suitable for a large block size from the image before the blurring process.

Therefore, the inventors completed the present invention, which is characterized in that the intensity of blurring per pixel unit is maintained during blurring processing, and the block size is determined using the maintained intensity of blurring. As a result, an appropriate block size can be selected for an area to which strong blur is applied during encoding after adding blur, and it is no longer necessary to use a smaller block when performing encoding that performs picture prediction. The size of the picture is predicted, so the computational load of encoding can be reduced without analyzing the blurred image.

<implementation mode>

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(implementation mode 1)

FIG. 1 is a block diagram of an image encoding device according to Embodiment 1 of the present invention.

In FIG. 1 , the stereoscopic photography encoding system photographs a subject so that stereoscopic viewing is possible, and encodes a stereoscopic video signal generated by photography. Such a stereoscopic imaging coding system includes cameras 100 a and 100 b that capture a main image and a sub image with parallax, an image coding device 200 , a blur parameter input unit 300 , and a storage unit 400 .

<Camera 100>

The cameras 100 a and 100 b are arranged at a distance of 6.5 cm, which is the average distance between human eyes, from each other. The cameras 100 a and 100 b each photograph a subject, and output video signals obtained by the photography to the image encoding device 200 .

Hereinafter, for simplicity, the video signals respectively output from the cameras 100 a and 100 b are collectively referred to as stereoscopic video signals. In addition, the image represented by the video signal output from the camera 100a is called a main image, and the image represented by the video signal output from the camera 100b is called a sub image.

The image of the subject is stereoscopically viewed through the main image and the sub image. The cameras 100a and 100b generate and output pictures at the same timing (time) respectively.

<Fuzzy parameter input part 300>

The blur parameter input unit 300 is for inputting a blur parameter for adding a bokeh different from the bokeh caused by the actual imaging lens of the camera 100a and the camera 100b to the stereoscopic video signal of the main image and the sub image. The user interface consists of a touch panel and cross keys.

As input blur parameters, input the blur adjustment amount (S _n , S _f ) at the position in front of or behind the subject, each weight coefficient (W _n , W _f ) in front of the subject and behind the subject, and the subject The focal position information (P), the baseline length (l) between the cameras 100a and 100b, the pixel size (m), the focal length fp, the lens information different from the information of the actual photographic lens, that is, the imaginary focal length fv and the aperture value (F value).

For the blur adjustment amounts (S _n , S _f ) and weighting coefficients (W _n , W _f ) of the front and back of the subject, in the case of stereoscopic viewing, it is necessary to suppress the jumping out of the front. When expressing a sense of depth and perspective, increase the amount of blur in front of the subject by increasing the blur adjustment amount S _n or weight coefficient W _n in the front, or decreasing the blur adjustment amount S _f or weight coefficient W _f in the rear . Conversely, when expressing a three-dimensional effect that makes the front subject stand out by emphasizing the front subject and intentionally blurring the background, increase the rear blur adjustment amount S _f or the weight coefficient W _f , or decrease the front blur adjustment amount S f or the weight coefficient W f The blur adjustment amount S _n or the weight coefficient W _n increases the amount of blur behind the subject.

In addition, if the blur adjustment amount (S _n , S _f ) is increased, it is possible to uniformly apply blur processing to the front or back of the focus position regardless of the subject distance. On the other hand, if the weight coefficients (W _n , W _f ) are increased, the blur becomes stronger as the subject moves away from the focus position, and blur processing depending on the subject distance can be performed.

The input of the lens information and the weighting coefficient is performed by calling the setting item menu from the touch panel and the cross key and setting each item. The focus position is input by specifying the position of the subject in the main image of the camera 100a displayed on the touch panel.

The blur parameter input unit 300 receives the settings of these blur parameters and outputs them to the image coding device 200 . In this way, the blurring processing unit 202 can perform blurring processing that can express any bokeh situation that cannot be generated by a virtual imaging lens or an actual lens.

<storage unit 400>

The storage unit 400 is a recording medium for storing image data output from the image encoding device 200 (locally decoded image signal described later).

<Image Coding Device 200 >

The image encoding device 200 encodes the stereoscopic video signals output from the cameras 100 a and 100 b to generate and output encoded video signals. Here, in this embodiment, it is assumed that the image encoding device 200 performs blur processing on both the main image and the sub image of the stereoscopic video signal, and encodes the stereoscopic video signal. Also, when encoding two video signals included in a stereoscopic video signal, the image coding device 200 codes these video signals for each region constituting a picture. The image coding device 200 is a device that encodes a blurred image signal output from the camera 100a in the H.264MVC (Multi View Coding) format, and uses each picture included in the video signal as an I picture and a P picture, respectively. and B picture coding.

In addition, when encoding an I picture and a P picture, the image encoding device 200 performs intra-frame predictive encoding.

In addition, when encoding P pictures and B pictures, the image encoding device 200 performs inter-frame predictive encoding (motion compensation predictive encoding).

On the other hand, the image coding device 200 performs parallax-compensated predictive coding when coding a blurred video signal output from the camera 100b, and codes each picture included in the video signal as a P picture.

That is, the image encoding device 200 predicts a picture of the sub-image generated at the same timing as the picture from the picture of the main image, and encodes the picture of the sub-image based on the prediction result.

Such an image encoding device 200 includes an image acquisition unit 206, a parallax acquisition unit 207, a blur processing unit 202, a selector 203, switches 204a, 204b, and 204c, a subtractor 205, block size determination units 210a and 210b, a block division unit 212a and 212b, an intra prediction unit 213, a detection unit 214, a compensation unit 215, a transformation unit 216, a quantization unit 217, a variable length coding unit 218, an inverse quantization unit 219, an inverse transformation unit 220, and an adder 221.

<Image Acquisition Unit 206>

The image acquisition unit 206 acquires the image signals output from the cameras 100 a and 100 b and outputs them to the blur processing unit 202 .

<parallax acquisition unit 207>

The parallax acquisition unit 207 acquires the parallax value or the distance value between the subject and the camera 100 for each element unit on the image signal received by the image acquisition unit 206, and sends the parallax value or the distance value to the blur processing unit 202. output. In the present embodiment, the parallax acquisition unit 207 includes a parallax detection unit 201 , and the parallax detection unit 201 generates a parallax value.

<Parallax detection unit 201>

The parallax detection unit 201 detects parallax between main image and sub image data input from the cameras 100a and 100b. The detection of parallax is performed by finding the corresponding position between the main image and the sub image, and detecting the deviation in the horizontal direction between the unit area of one image and the corresponding area of the other image as the number of pixels. The parallax detection unit 201 outputs the detected parallax information d to the blur processing unit 202 .

In this embodiment, detection of parallax is performed using a block matching method. In the block matching method, a unit area of 16×16 pixels is first cut out from the comparison source image. Next, a plurality of identical 16×16 areas are cut out from the comparison target search area, and for each cut out area, the sum of the absolute values of brightness differences (SAD; Sum of Absolute Difference) from the comparison source area is obtained . Then, by finding the cutout position with the minimum SAD, the position corresponding to the comparison source image is obtained in units of regions.

<blurring unit 202>

The blur processing unit 202 uses the blur parameter input from the blur parameter input unit 300 and the disparity information or distance information input from the disparity acquisition unit 207 to determine the pixel unit to be added to the main image and the sub image input from the image acquisition unit 206. fuzzy matrix. In this embodiment, the blur processing unit 202 uses the parallax information input from the parallax acquisition unit 207 . The blur processing unit 202 outputs, to the block size determination units 210 a and 210 b , the blur amount for each pixel unit calculated from the blur matrix.

Also, the blur processing unit 202 performs blur processing on the main image or sub image data based on the blur matrix, and outputs each blurred image data of the main image and sub image resulting from the processing to the selector 203 .

In addition, in the present embodiment, a pixel unit is constituted by one pixel.

The flow of obfuscation is as follows. The blur processing unit 202 obtains the distance distribution from the camera to the subject using the disparity information d, generates a blur matrix based on the parameters from the blur parameter input unit 300 and the distance of the subject, and passes each pixel of the main image or sub-image data Area overlay blur matrix to add blur. At the same time, the blur processing unit 202 calculates the blur amount from the blur matrix, and outputs it to the block size determination units 210a and 210b.

It demonstrates using FIG. 2. First, the distance distribution from the camera to the subject is obtained from the parallax information d of each pixel unit. As shown in Fig. 2(a), if d is the parallax information and m is the length in the horizontal direction of one pixel of the image sensor used in photography, the distance D between the subject and the camera is as follows:

[Equation 1]

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; d</span>}}$

In addition, as described above, _fp is the focal length of the lens, and l is the base line length between the cameras 100a and 100b.

By performing such calculations for all pixel units, the distance distribution of the subject captured in the main image or the sub image is obtained.

The image of a point at a position deviated from the focal point P by the distance from the camera, that is, a bokeh circle is called a circle of confusion, but in this embodiment, the size of the blur matrix is determined using a virtual circle of confusion diameter σ.

The distance L between the focus position P and the camera is obtained as a value of D by substituting the parallax information at the focus position P into d in the calculation formula of D above. Similarly, the distance X between an arbitrary subject and the camera is obtained as a value of D by substituting the parallax information of the subject into d in the calculation formula of D above.

The diameter of the circle of confusion at a distance X from the camera is calculated as follows using the focal distance fv, the F value F, the distance L between the focal position and the camera, and the distance X between the camera and the subject, which are characteristic values of the optical system.

In the present embodiment, a virtual value input from the fuzzy parameter input unit 300 is used for the F value F. In addition, using the blur adjustment amounts S _f , S _n , and weight coefficients W _f , W _n input from the blur parameter input unit 300 , the diameters σ of the circle of confusion behind and in front of the comparison focus position are weighted. Blur adjustment amounts S _f , S _n and weight coefficients W _f , W _n are as described above. If S _f >S _n or W _f >W _n , the rear blur is stronger and the front blur is weaker. If S If _f <S _n or W _f <W _n , the front blur is stronger and the rear blur is weaker. Also, if the blur adjustment amounts S _f and S _n are both 0, there is no adjustment amount, and bokeh by the virtual optical system is formed, and the larger the value, the more uniform the blur can be applied to the front or rear of the focus position. In addition, if the weight coefficients W _f and W _n are both 1, there will be no weighting, and the bokeh itself generated by the virtual optical system will become larger. The larger the value, the stronger the blur that depends on the subject distance will be generated. If both If it is 0, the blur to the front or back no longer depends on the subject distance.

If the calculation formula of the circle of confusion is approximated and the adjustment amount is added and weighted, the diameter σ of the imaginary circle of confusion behind the focus position is as follows:

[Formula 2]

$\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}<span\; class="notranslate">\; \&CenterDot;</span>{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; FLX</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; f</span>}}$

The diameter σ of the imaginary circle of confusion ahead of the focus position is as follows:

[Formula 3]

$\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; FLX</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}$

The blur matrix can be obtained by inputting the disparity information d into the formula of D above for each pixel unit on the main image and the sub-image to obtain the distance X, and inputting the distance X into the formula for the diameter σ of the imaginary circle of confusion. the size of. In FIG. 2( b ), the relationship between the diameter σ of the imaginary circle of confusion and the distance based on Equation 2 and Equation 3 is shown. Next, as shown in FIG. 3( a ), a fuzzy matrix is created in which the inside of a circle whose diameter is the virtual confusion circle diameter σ, which is the size of the fuzzy matrix, is set to filter coefficient 1, and the outside is set to filter coefficient 0.

Next, blurring is added by superimposing the generated blur matrix on each pixel area of the main image or sub image data. At this time, the sum of all the filter coefficients in the fuzzy matrix is first calculated, and the value is used to divide all the filter coefficients in the matrix to generate a superposition matrix in which the sum of all the filter coefficients inside is 1. Use this overlay matrix for the overlay.

Also, based on the blur matrix for each pixel unit obtained above, the blur amount for each pixel unit is calculated. The blur amount is a value indicating that the filter coefficients in the blur matrix are in the range above the threshold. For values indicating a range, eg the diameter in the case of a circle, the length of the diagonal in the case of a square.

In this embodiment, the fuzzy matrix is a matrix in which the inside of a circle with the diameter σ of the imaginary circle of confusion is set as a filter coefficient 1, and the outside is set as a filter coefficient 0. Therefore, a threshold value of 1 is set as a circle with an indicated diameter σ. The value of , set the amount of blurring to σ.

The blur processing unit 202 outputs the blur amount per pixel obtained as described above to the block size determination units 210 a and 210 b.

<selector 203>

The selector 203 acquires video signals output from the cameras 100a and 100b, alternately switches these video signals, and outputs them to the switch 204a, the subtractor 205, and the block dividing unit 212a or 212b.

In this embodiment, the selector 203 performs switching in units of pictures. That is, when the selector 203 obtains the blurred main image and the sub image from the blur processing unit 202 at the same timing, first outputs the blurred main image, and then outputs the blurred sub image.

<Subtraction Department 205>

The subtractor 205 calculates, for each block, the difference between the encoding target image indicated by the video signal output from the selector 203 and the predicted image indicated by the predicted signal output from the switch 204b. The subtractor 205 outputs a prediction error signal indicating the difference to the switch 204a.

Here, the subtracter 205 calculates the above-mentioned difference in each picture in which the encoding target image is an I picture, a P picture, or a B picture (block), and outputs a prediction error signal.

<switch 204a>

The switch 204a connects the selector 203 to the conversion unit 216 when the encoding target image is the main image of an I picture and intra prediction is not performed, and transmits the video signal representing the encoding target image from the selector 203 to the conversion unit 216. .

In addition, the switch 204a switches the subtractor to the substractor when the encoding target image is a main image of an I picture and intra-frame prediction is performed, or when the encoding target image is an image (main image or sub image) of a P picture or a B picture. 205 is connected to the conversion unit 216 , and transmits each prediction error signal of an I picture, a P picture, or a B picture from the subtractor 205 to the conversion unit 216 .

<conversion unit 216>

The transform unit 216 acquires a video signal or a prediction error signal as an image signal for each block via the switch 204 a, and performs discrete cosine transform (DCT; Discrete Cosine Transform) as an orthogonal transform on the image signal. Thus, the conversion unit 216 converts the image signal into DCT coefficients that are frequency coefficients, that is, converts the domain of the image signal from the spatial domain to the frequency domain, and outputs the frequency coefficients to the quantization unit 217 .

<quantization part 217>

The quantization unit 217 acquires frequency coefficients from the conversion unit 216 and quantizes the frequency coefficients. That is, the quantization unit 217 generates a quantization value by dividing the frequency coefficient by the quantization step size. Here, the quantization unit 217 quantizes the coding target block using a determined quantization width in accordance with the image coding standard, H.264 in this embodiment.

<variable length coding unit 218>

The variable length coding unit 218 performs variable length coding on the quantization value generated by the quantization unit 217 and the motion vector or disparity vector output from the detection unit 214 to generate and output an encoded stereoscopic video signal. In addition, this variable length coding is reversible coding.

<Inverse Quantization Unit 219>

The inverse quantization unit 219 generates inverse quantization frequency coefficients by inverse quantizing the quantization value generated by the quantization unit 217 . That is, the inverse quantization unit 219 generates an inverse quantization frequency coefficient by multiplying the quantization width used in the quantization unit 217 by the quantization value.

In addition, the dequantized frequency coefficients generated at this time are different from the frequency coefficients generated by the conversion unit 216 and include quantization errors.

<Inverse conversion unit 220>

The inverse transform unit 220 performs an inverse orthogonal transform, which is an inverse operation of the orthogonal transform performed by the transform unit 216, on the inverse quantized frequency coefficients generated by the inverse quantization unit 219, and performs an inverse discrete cosine transform (iDCT; inverse Discrete Cosine Transform) in this embodiment. Transform). Accordingly, the inverse transform unit 220 transforms the inverse quantized frequency coefficients into an image signal, that is, transforms the region of the inverse quantized frequency coefficients from the frequency domain to the spatial domain, and outputs the image signal to the adder 221 .

<Adder 221>

The adder 221 stores the image signal output from the inverse transform unit 220 as a locally decoded image signal in the storage unit 400 when the encoding target image is a main image of an I picture and intra prediction is not performed.

In addition, the adder 221 performs intra prediction when the encoding target image is a main image of an I picture, or when the encoding target image is a P picture or a B picture (either a main image or a sub image). In this case, the image signal output from the inverse transform unit 220 is added to the prediction signal output from the switch 204c, and the addition result is output to the intra prediction unit 213 as a locally decoded image signal. At the same time, the addition result is stored in the storage unit 400 .

<block size determination unit 210a>

The block size determination unit 210a obtains the blurring amount of the coding target image area from the blurring processing unit 202 and compares it with a threshold when the coding target picture is an I picture, a P picture, or a B picture and performs intra prediction.

In the present embodiment, H.264 is used as the encoding method as described above, so in intra prediction, block sizes of 16×16 and 4×4 can be used. Here, the threshold is set to 4 pixels, and the block size of 4×4 is not used for the area whose blurring amount is larger than 4 pixels.

When the amount of blur is greater than the threshold, block size information that limits the block unit predicted by the intra prediction unit 213 to a larger block size of 16×16 is output to the block dividing unit 212 a.

Conversely, when the amount of blur is smaller than the threshold, block size information including a smaller block size of 4×4 in addition to 16×16 is output to the block division unit 212 a.

<block division unit 212a>

The block division unit 212a divides the encoding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determination unit 210a.

For example, when the block size information is limited to 16×16, the encoding target image is divided into only 16×16, and the divided image is output to the intra prediction unit 213 .

On the other hand, when the block size information is composed of two types of 16×16 and 4×4, an image obtained by dividing an encoding target image into a 16×16 block size, and an image obtained by dividing the same encoding target image into a 4×4 block size are prepared. The prepared two types of divided images are output to the intra prediction unit 213 respectively.

<Intra Prediction Unit 213>

For the locally decoded image signal in the same picture output from the adder 221, the intra prediction unit 213 calculates a total of four positions, which are left, upper, upper right, and upper left when viewed from the position of the encoding target image, to obtain the following information from the block division unit. Intra-frame prediction is performed in units of the divided block size output by 212a.

For example, when block information limited to a block size of 16×16 is received from the block dividing unit 212 a, a search for the same or similar block is performed with a block size of 16×16 for four positions.

On the other hand, in the case of receiving two types of block size information of 16×16 and 4×4, a search for the same or similar block is performed with two block sizes of 16×16 and 4×4 for 4 positions .

Then, the most similar prediction signal obtained by searching is output to switches 204b and 204c.

<block size determination unit 210b>

The block size determination unit 210b obtains the blurring amount of the encoding target image region from the blur processing unit 202 when the encoding target image is a main image of a P picture or a B picture, or a sub image of a P picture, and compares it with a threshold.

In the present embodiment, as described above, H.264 is used as an encoding method, so block sizes of 16×16 and 8×8 can be used for inter prediction. Here, the threshold is set to be 8 pixels, and the block size of 8×8 is not used for pixels whose blurring amount is larger than 8 pixels. In addition, block sizes of 16×8 and 8×16 can also be used in the H.264 standard, but they are not used in this embodiment.

When the amount of blur is greater than the threshold, block size information that limits the block unit for searching for a motion vector or a disparity vector by the detection unit 214 to a large block size of 16×16 is output to the block division unit 212b. Here, 8 pixels are used as the threshold.

Conversely, when the amount of blur is smaller than the threshold, block size information including a smaller block size of 8×8 in addition to 16×16 is output to the block division unit 212 b.

<block division unit 212b>

The block division unit 212b divides the encoding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determination unit 210b.

For example, when the block size information is limited to 16×16, the encoding target image is divided into only 16×16, and the divided image is output to the detection unit 214 .

On the other hand, when two kinds of block size information of 16×16 and 8×8 are received, an image obtained by dividing an encoding target image into a block size of 16×16, and an image divided into 8×8 using the same encoding target image are prepared. There are two types of divided images with a block size of 8, and the two divided images are output to the detection unit 214 .

<detection part 214>

The detection unit 214 detects a motion vector for each block of the encoding target image when the encoding target image block-divided by the block dividing unit 212 b is a main image of a P picture or a B picture.

That is, the detection unit 214 refers to the local decoded image signal representing another encoded and decoded main image (I picture or P picture) stored in the storage unit 400 as a reference image.

Then, the detection unit 214 detects the motion of the encoding target image as a motion vector by searching for a block identical to or similar to the encoding target image (encoding target block) from the reference image.

Then, the detection unit 214 outputs the motion vector to the compensation unit 215 and the variable length coding unit 218 .

On the other hand, when the encoding target image block-divided by the block dividing unit 212 b is a sub-picture of a P picture, the detection unit 214 detects a disparity vector for each block of the encoding target image.

That is, the detection unit 214 refers to the local decoded image signal representing the coded and decoded main image (I picture, P picture, or B picture) stored in the storage unit 400 as a reference image.

Then, the detection unit 214 detects the positional relationship between the encoding target block and the same or similar block as a disparity vector (disparity) by searching the reference image for the same or similar block as the encoding target image (coding target block).

Also, a picture including a sub-picture of a block to be coded and a reference picture (reference picture) that is a main picture are pictures generated at the same timing by the cameras 100a and 100b, respectively. Then, the detection unit 214 outputs the disparity vector to the compensation unit 215 and the variable length coding unit 218 .

In addition, the disparity vector expresses the above-mentioned positional relationship between blocks as a positional deviation in the left-right direction.

Also, in the case of both motion vectors and disparity vectors, when block information limited to a block size of 16×16 is received from the block dividing unit 212b, only The search for the same or similar blocks is performed with a block size of 16×16.

On the other hand, when two block size information of 16×16 and 8×8 is received, for the surrounding 32×32 area of the encoding target image, two kinds of block sizes of 16×16 and 8×8 are used. Search for identical or similar blocks.

Then, the most similar motion vector or disparity vector is output to the compensation unit 215 .

<Compensation Department 215>

When the compensation unit 215 acquires a motion vector from the detection unit 214 , it performs motion compensation using the motion vector and a reference image to generate a prediction signal, and outputs the prediction signal. That is, the aforementioned same or similar blocks included in the reference picture are spatially shifted according to the aforementioned motion vector.

On the other hand, when the compensation unit 215 acquires the disparity vector from the detection unit 214 , it performs disparity compensation using the disparity vector and the reference image, thereby generating a prediction signal, and outputs the prediction signal.

That is, the above-mentioned identical or similar blocks included in the reference image are spatially shifted according to the above-mentioned disparity vector.

<Switch 204c>

The switch 204c opens between the intra prediction unit 213 and the adder 221 and between the compensation unit 215 and the adder 221 when the encoding target image is the main image of an I picture and intra prediction is not performed.

In addition, the switch 204c connects the intra prediction unit 213 to the adder 221 when the encoding target image is an I-picture, P-picture, or B-picture image (main image or sub-image) to perform intra prediction, and the prediction signal The data is passed from the intra prediction unit 213 to the adder 221 .

On the other hand, the switch 204c connects the compensation unit 215 to the adder 221 when the coding target picture is a P picture or a B picture (main picture or sub picture), and sends the prediction signal from the compensation unit 215 to the adder 221. transfer.

<switch 204b>

The switch 204b connects the intra prediction unit 213 to the subtractor 205 when performing intra prediction when the encoding target image is an image of an I picture, a P picture, or a B picture (main image or sub image). The intra prediction unit 213 passes the information to the subtractor 205 .

On the other hand, the switch 204b connects the compensation unit 215 to the subtractor 205 when the coding target picture is a P picture or a B picture (main picture or sub picture), and sends the predicted signal from the compensation unit 215 to the subtracter 205. transfer.

Also, when the encoding target image is a P picture or a B picture (main image or sub image) and there are prediction signals from both the intra prediction unit 213 and the compensation unit 215, the prediction signal from the intra prediction unit 213 and the A more similar prediction signal among the prediction signals of the compensation unit 215 is passed to the subtractor 205 .

<action>

4 , 5 , 6 , 7 , and 8 are flowcharts showing operations of the image coding device 200 according to this embodiment.

The image coding device 200 sequentially codes the pictures included in the stereoscopic video signal with parallax.

At the time of encoding, the image encoding device 200 first sets arbitrary blurring parameters for blurring, such as lens information for blurring areas other than the main subject, in order to make the main subject stand out and give a three-dimensional effect. (S10).

The image coding device 200 acquires an image signal after the setting of the blur parameter is completed ( S20 ), and the parallax detection unit 201 detects the parallax between the two acquired images ( S30 ).

The image encoding device 200 selects an image to be encoded ( S40 ). In this embodiment, both the main image and the sub image are selected for coding by adding blur to both the main image and the sub image.

The blurring processing unit 202 calculates a blurring amount to be added to the stereoscopic video signal based on the blurring parameter acquired in step S10 and the parallax information acquired in S30 , and performs blurring processing of adding the calculated blurring amount to the stereoscopic video signal ( S50 ).

The image coding device 200 generates a signal to be coded in order to code a stereoscopic video signal composed of blurred images to which blur amounts have been added ( S60 ).

Here, a series of operations in S60 for generating an encoding target signal will be described. First, it is determined whether or not the image to be encoded is a main image ( S61 ). Here, it is determined by the control unit included in the image encoding device 200 .

Here, if it is determined that the encoding target image is the main image (YES in S61 ), the control unit further determines whether motion compensation predictive encoding should be performed on the encoding target image ( S62 ). For example, when the image encoding device 200 encodes the encoding target image as a block of a P picture or a B picture, it determines that motion compensation predictive encoding should be performed (Yes in S62), and when encoding the encoding target image as a block of an I picture, , it is determined that motion compensation predictive coding should not be performed (No in S62).

If the control unit determines that motion compensation predictive coding should be performed (Yes in S62), it controls the switch 204a to connect the subtractor 205 to the conversion unit 216, controls the switch 204b to connect the compensation unit 215 to the subtractor 205, The switch 204c is controlled to connect the compensation unit 215 and the adder 221 .

Next, the block size determination unit 210 b determines a block size for motion detection based on the blur amount of the encoding target image ( S64 b ). First, the blurring amount of the encoding target image region is acquired from the blurring processing unit 202 and compared with a threshold value ( S211 ). If it is judged to be larger than the threshold value, the block size information that limits the block unit in which the motion vector is searched by the detection unit 214 to a large block size of 16×16 is output to the block division unit 212 b ( S212 ). Conversely, if it is judged to be smaller than the threshold, block size information including a smaller block size of 8×8 in addition to 16×16 is output to the block division unit 212 b ( S213 ).

The block division unit 212b divides the coding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determination unit 210b (S65b).

Next, the detection unit 214 searches for the same or similar block with the block size received from the block division unit 212b in the encoding target image, and detects a motion vector. At this time, the most similar motion vector is output ( S661 ).

In the motion vector, when the block information limited to the block size of 16×16 is received from the block division unit 212b, only the block size of 16×16 is performed for the area of 32×32 around the encoding target image. Search for identical or similar blocks.

On the other hand, when two types of block size information of 16×16 and 8×8 are received, for the surrounding 32×32 area of the encoding target image, two kinds of block sizes of 16×16 and 8×8 are performed. Search for identical or similar blocks. Then, the most similar motion vector is output to the compensation unit 215 .

Furthermore, the compensation unit 215 performs motion compensation using the detected motion vector, and generates a prediction signal for a block of the encoding target image ( S662 ).

Furthermore, the subtracter 205 generates a prediction error signal by subtracting the image indicated by the prediction signal from the encoding target image ( S67 b ).

On the other hand, when the encoding target image is an I picture and it is determined in S406 that motion compensation predictive encoding should not be performed and intra-frame predictive encoding should be performed, or when the encoding target image is a P picture or a B picture, When it is determined that intra-frame predictive coding should be performed (Yes in S63), the switch 204a is controlled to connect the subtractor 205 to the conversion unit 216, the switch 204b is controlled to connect the intra-frame coding unit 213 to the subtractor 205, and the switch 204c performs control to connect between the intra coding unit 213 and the adder 221 .

Next, the block size determination unit 210 a determines a block size for performing intra prediction based on the blur amount of the encoding target image ( S64 a ). First, the blurring amount of the encoding target image region is acquired from the blurring processing unit 202 , and it is judged whether or not it is larger than a threshold value ( S201 ). If it is judged to be larger than the threshold, the block size information is output to the block division unit 212 a , which limits the block unit of the block search performed by the intra prediction unit 213 to a large block size of 16×16 ( S202 ). Conversely, if it is judged to be smaller than the threshold, block size information including a smaller block size of 4×4 in addition to 16×16 is output to the block division unit 212 a ( S203 ).

The block dividing unit 212 a divides the coding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determining unit 210 a ( S65 a ).

In intra prediction, when block information limited to a block size of 16×16 is received from the block division unit 212a, a total of 4 blocks are left, top, top right, and top left when viewed from the position of the encoding target image. positions, the search for the same or similar blocks is performed only with a block size of 16×16.

On the other hand, in the case where two kinds of block size information of 16×16 and 4×4 are received, the search for the same or similar block is performed with two kinds of block sizes of 16×16 and 4×4 for 4 positions .

Then, the most similar prediction signal is output to the switches 204b and 204c (S66).

Furthermore, the subtracter 205 generates a prediction error signal by subtracting the image indicated by the prediction signal from the encoding target image ( S67 a ).

On the other hand, if the control unit determines that the coding target picture is a sub picture (No in S61), it controls the switch 204a to connect the subtractor 205 to the conversion unit 216, controls the switch 204b to connect the compensation unit 215 and the subtractor 205 is connected, and the switch 204c is controlled to connect the compensating part 215 and the adder 221.

Next, the block size determination unit 210 b determines a block size for performing parallax detection based on the blur amount of the encoding target image ( S64 c ). First, the blurring amount of the encoding target image region is acquired from the blurring processing unit 202 , and it is judged whether it is larger than a threshold ( S221 ). If it is judged to be larger than the threshold, block size information that limits the block unit for searching for a disparity vector by the detection unit 214 to a larger block size of 16×16 is output to the block division unit 212 b ( S222 ).

Conversely, if it is judged to be smaller than the threshold, block size information including a smaller block size of 8×8 in addition to 16×16 is output to the block division unit 212 b ( S223 ).

The block division unit 212b divides the encoding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determination unit 210b (S65c).

Next, the detection unit 214 searches for the same or similar block with the block size received from the block division unit 212b in the encoding target image, and detects a disparity vector. At this time, the most similar disparity vector is output ( S663 ).

In the disparity vector, when the block information limited to the block size of 16×16 is received from the block division unit 212b, only the block size of 16×16 is performed for the area of 32×32 around the encoding target image. Search for identical or similar blocks.

On the other hand, when two types of block size information of 16×16 and 8×8 are received, for the surrounding 32×32 area of the encoding target image, two kinds of block sizes of 16×16 and 8×8 are performed. Search for identical or similar blocks. Then, the most similar disparity vector is output to the compensation unit 215 .

Furthermore, the compensation unit 215 performs parallax compensation using the detected parallax vector, thereby generating a prediction signal for a block of the encoding target image ( S664 ).

Also, the subtracter 205 generates a prediction error signal by subtracting the image indicated by the prediction signal from the encoding target image ( S67 c ).

Next, the switch 204 a outputs the generated prediction residual signal to the conversion unit 216 ( S70 ).

Here, a plurality of prediction residual signals may be generated. For example, when the encoding target image is a sub-picture and is a P picture, since it is a sub-picture, a prediction residual signal based on parallax compensation is generated (S67c), and since it is a P picture, a prediction residual signal based on motion compensation is generated. Since the difference signal ( S67 b ) can be intra-coded, a prediction residual signal by intra coding is generated ( S67 a ), and as a result, a total of three prediction residual signals are generated. In this case, the smallest prediction residual signal among the plurality of generated prediction residual signals is passed to the conversion unit 216 .

In contrast, sometimes no prediction residual signal is generated at all. For example, when the encoding target image is an I picture of the main image, none of intra prediction, motion compensation prediction, and parallax compensation prediction is performed, and therefore no prediction residual signal is generated. In this case, the switch 204 a outputs the encoding target image of the main image data output from the selector 203 to the conversion unit 216 without performing selection of the prediction residual signal.

Next, the transformation unit 216 generates frequency coefficients by performing orthogonal transformation on the encoding target image or the prediction residual signal output from the switch 204 a ( S80 ).

The quantization unit 217 quantizes the frequency coefficient to generate a quantized value ( S90 ), and the variable-length coding unit 218 performs variable-length coding on the quantized value ( S100 ).

<Summary>

In this way, in this embodiment, in determining the block size at the time of encoding, not only distance information and focus information based on disparity information but also blur information added by blur processing using arbitrary blur parameters are used to determine any The ambiguity situation can make the block size decision.

Thus, in encoding capable of determining a block size from a plurality of candidates, a region with a larger amount of blur is limited to a larger block size, so that the compression ratio of a region with a small amount of information and a large amount of blur can be improved.

In addition, in this embodiment, it is possible to perform encoding using intra prediction after determining a variable block size for any blurring situation.

Thus, in encoding based on intra prediction that improves the compression rate by utilizing the correlation between blocks in the same screen, it is determined that the high-frequency Flat images with reduced components, and limited to a large prediction block size for prediction, thereby reducing the amount of search processing for reference blocks with small prediction residuals in intra prediction, enabling faster processing and power saving . Also, conversely, in areas where the amount of blur is small or not blurred, prediction is performed using blocks including smaller blocks, so that there is no problem even if there is a complex image. In this way, both flat images and complex images can be efficiently predicted.

Furthermore, in the present embodiment, it is possible to perform encoding using inter-frame prediction based on motion compensation prediction or parallax compensation prediction after determining a variable block size for an arbitrary blur state.

As a result, in inter-frame prediction-based encoding that improves the compression rate by utilizing inter-block correlations between multiple frames, it is determined that high-frequency A flat image with reduced components is predicted in units of larger blocks, thereby reducing the amount of search processing for reference blocks with small prediction residuals in inter prediction, and enabling faster processing and power saving. Conversely, in an area where the amount of blur is small or not blurred, prediction is performed using blocks including small blocks, so that there is no problem even if there is a complex image. In this way, both flat images and complex images can be efficiently predicted. Furthermore, in the search process of the reference block in the inter prediction, a plurality of pictures are stored in the memory and referred to, so the number of times of data access and the amount of data transfer can be reduced by limiting the prediction to a larger block unit. , it is possible to save memory bandwidth.

(Modification 1)

In the above-mentioned embodiment, the image encoding device 200 generates parallax information from the main image and the sub-image, adds blurring to the main image and the sub-image respectively, and encodes them. The distance information of , append blur and encode it.

FIG. 9 shows an image coding device 500 according to this modification. In FIG. 9 , the same reference numerals are used for the same constituent elements as those in FIG. 1 , and description thereof will be omitted.

In FIG. 9 , the photographed image encoding system acquires distance information between the subject and the system while photographing the subject, and encodes an image signal blurred using the distance information. Such a captured image coding system includes the image coding device 500 of this modified example, the camera 100 , the distance acquiring unit 110 , and the blur parameter input unit 300 .

<Distance acquisition unit 110>

The distance acquiring section 110 measures the distance between the camera and the subject for each pixel on the image input from the camera 100 . The detection of the distance is performed by sending out ultrasonic waves or millimeter waves, etc., and measuring the time and phase difference until the reflection wave from the object arrives.

<Image encoding device 500>

The image encoding device 500 uses the distance information output from the distance obtaining section 110 to blur and encode the image signal output from the camera 100 , thereby generating and outputting an encoded image signal.

Such an image encoding device 500 includes an image acquisition unit 206, a parallax acquisition unit 501, a blur processing unit 202, a selector 203, switches 204a, 204b, and 204c, a subtractor 205, block size determination units 210a and 210b, a block division unit 212a and 212b, intra prediction unit 213, detection unit 214, compensation unit 215, transformation unit 216, quantization unit 217, variable length coding unit 218, inverse quantization unit 219, inverse transformation unit 220, and adder 221.

<parallax acquisition unit 501>

The parallax acquisition unit 501 does not include the parallax detection unit 201 , acquires parallax or distance information from the distance acquisition unit 110 , and outputs the parallax or distance information to the blur processing unit 202 . In the present embodiment, the parallax acquisition unit 501 acquires the distance between the camera and the subject for each pixel on the image input from the camera 100 from the distance acquisition section 110 .

<action>

The operation of the image encoding device 500 is the same as that of the above-mentioned embodiment, except that the blur amount is calculated using the input distance information instead of detecting the parallax, and the object of blurring and encoding is a single image, so the operation in FIG. 4 is used as it is. Flowchart, only the differences are explained.

The image acquisition unit 206 acquires an image signal from the camera 100 ( S20 ). The accepted video signal is a video signal of a single viewpoint.

The parallax acquisition unit 501 acquires distance information per pixel of the image signal from the distance acquisition unit 110 , and outputs it to the blur processing unit 202 ( S30 ).

The image encoding device 500 selects the image signal received by the image acquisition unit 206 as an encoding target image ( S40 ).

The blur processing unit 202 calculates the amount of blur using the distance information per pixel of the image signal, and adds blur to the encoding target image ( S50 ). At this time, since the blur processing unit 202 has obtained the distance information from the parallax acquisition unit 501 , the blur amount is calculated using the distance information from the parallax acquisition unit 501 without performing a process of calculating the distance information from the parallax using blur parameters.

The following coding process is the same as the operations in S60 to S100 except that a single picture without sub-pictures is coded, and thus description thereof will be omitted.

In this manner, in this modified example, it is possible to select a block size and encode a single image whose distance to a subject is known, using blur information added by blur processing using an arbitrary blur parameter and distance.

In addition, in this modified example, the parallax acquisition unit 501 acquires distance information from the distance acquisition unit 110 and outputs it to the blur processing unit 202 , but the parallax acquisition unit 501 may accept input of the parallax value itself such as a parallax image. In this case, the parallax acquisition unit 501 may output the acquired parallax to the blur processing unit 202 , or may convert the distance information into distance information using a blur parameter and output the distance information to the blur processing unit 202 . In addition, similarly, the parallax acquisition unit 501 may obtain distance information, convert the distance information into a parallax using a blur parameter, and output the parallax to the blur processing unit 202 .

(Modification 2)

In the above-described embodiment, the image encoding device 200 encoded the stereoscopic video signal composed of the main image and the sub image by adding blur, but it may also add blur and encode only one of the main image or the sub image.

In this case, for example, the blur parameter includes information indicating which of the main image and the sub image is to be encoded with blurring, and in S40 based on this information, the image to be encoded can be selected, so that only the image to be encoded can be selected. The image of the object is blurred and encoded. Alternatively, when only the main image is blurred and encoded, the image signal from the camera 100 b may be input only to the parallax detection unit 201 without being input to the image acquisition unit 206 or the blur processing unit 202 .

In this modified example, since there is a single image to be encoded by adding blur, the blur processing unit 202 performs blur processing only on an image to be encoded, and outputs single blur image data to the selector 203 . The blurring and encoding operations are the same as those in Modification 1 described above, so descriptions thereof are omitted.

(Embodiment 2)

FIG. 10 is a block diagram of an image encoding device 600 according to Embodiment 2 of the present invention. In FIG. 10 , the same reference numerals are used for the same constituent elements as those in FIG. 1 , and description thereof will be omitted.

The image coding device 600 of the present embodiment is characterized in that, after determining a variable block size for an arbitrary blur state, coding is performed using an orthogonal transform for converting to a frequency component.

In FIG. 10 , the stereoscopic imaging coding system photographs a subject so that stereoscopic viewing is possible, and encodes a stereoscopic video signal generated by imaging. Such a stereoscopic encoding system includes the image encoding device 600 according to Embodiment 2, cameras 100 a and 100 b that receive input of parallaxed main images and sub images, and a blur parameter input unit 300 .

<Image coding device 600>

The image encoding device 600 adds blur to the stereoscopic video signals output from the cameras 100 a and 100 b using the parallax output from the parallax detection unit 201 , and encodes them to generate and output encoded video signals. In addition, when the image encoding device 600 encodes two video signals included in the stereoscopic video signal, it encodes them as still images or moving images.

Such an image encoding device 600 includes an image acquisition unit 206, a parallax acquisition unit 207, a blur processing unit 202, a selector 203, a block size determination unit 610, a block division unit 611, a conversion unit 216, a quantization unit 217, and a variable length coding unit. 218.

In addition, in the present embodiment, the still image format JPEG2000 and the moving image format MotionJPEG2000 using discrete wavelet transform which is an orthogonal transform are used as encoding methods.

<block size determination unit 610>

The block size determination unit 610 obtains the blurring amount of the encoding target image region from the blur processing unit 202 for the encoding target image, and compares it with a threshold.

In this embodiment, one of 16×16 and 8×8 is used as the block size, and the threshold is set to 8 pixels.

When the amount of blur is larger than the threshold value, block size information that limits the block unit to be orthogonally transformed by the transformation unit 216 to a larger block size of 16×16 is output to the block division unit 611 .

On the contrary, when it is smaller than the threshold value, block size information limited to a small block size of 8×8 is output to the block division unit 611 .

<block division unit 611>

The block dividing unit 611 divides the encoding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determining unit 610 .

In the present embodiment, when the block size information is limited to 16×16, the encoding target image is divided only in the size of 16×16, and the divided image is output to the conversion unit 216 .

On the other hand, when the block size information limited to 8×8 is received, the encoding target image is divided into only the size of 8×8, and the divided image is output to the conversion unit 216 .

<action>

11 and 12 are flowcharts showing the operation of the image coding device 600 according to this embodiment.

In FIG. 11 , the same symbols are used for the same operations as those in FIG. 4 , and explanations thereof are omitted.

The block size determination unit 610 determines a block size for performing orthogonal transformation based on the blur amount of the encoding target image ( S310 ). First, the blurring amount of the encoding target image region is acquired from the blurring processing unit 202 , and it is judged whether it is larger than a threshold ( S311 ). If it is judged to be larger than the threshold value, the block size information block division unit 611 outputs a large block size that limits the block unit to be orthogonally transformed by the transformation unit 216 to 16×16 ( S312 ).

On the contrary, if it is judged to be smaller than the threshold, block size information limited to a smaller block size of 8×8 is output to the block dividing unit 611 ( S313 ).

The block division unit 611 divides the encoding target image output from the selector 203 according to the size of the block size information based on the block size information output from the block size determination unit 610 ( S320 ).

<Summary>

As described above, in this embodiment, blur information added by blur processing using an arbitrary blur parameter is used to determine a block size for performing orthogonal transform at the time of encoding. Accordingly, it is possible to determine the block size of the orthogonal transform for any blurring situation.

In addition, in an area with a large amount of blurring, high-frequency components are reduced by strong blurring, so by limiting the block size to a large size, high compression can be achieved.

In addition, the same modification as Modification 1 or Modification 2 of Embodiment 1 can also be performed on the image coding device 600 of Embodiment 2. The operation of the modified example is the same as that of the second embodiment except for the differences in S10 to 40 described in the modified example of the first embodiment, and the object of blurring and coding is a single image signal.

(Embodiment 3)

FIG. 13 is a block diagram of an image encoding device 700 according to Embodiment 3 of the present invention. In FIG. 13 , the same reference numerals are used for the same components as those in FIGS. 1 and 9 , and explanations thereof are omitted.

The image encoding device 700 of the present embodiment is characterized in that it generates two images that can be viewed stereoscopically based on one image and the distance information of the subject corresponding to the image, and adds an arbitrary blur using the distance information of the subject. coding.

In FIG. 13 , the stereoscopic video encoding system acquires distance information of the subject while photographing the subject, and generates and encodes a stereoscopic video signal from an image generated by photographing based on the distance information. Such a stereoscopic video coding system includes the image coding device 700 according to Embodiment 3, the camera 100 that accepts image input, the distance acquisition unit 110 , the blur parameter input unit 300 , and the storage unit 400 .

<Image coding device 700>

The image encoding device 700 includes a parallax acquisition unit 501, an image acquisition unit 701, a blur processing unit 202, a selector 203, switches 204a, 204b, and 204c, a subtractor 205, block size determination units 210a and 210b, block division units 212a and 212b, An intra prediction unit 213 , a detection unit 214 , a compensation unit 215 , a transformation unit 216 , a quantization unit 217 , a variable length coding unit 218 , an inverse quantization unit 219 , an inverse transformation unit 220 , and an adder 221 . Furthermore, the image acquisition unit 206 includes a stereoscopic image generation unit 701 .

<Fuzzy parameter input part 300>

The blur parameter input unit 300 is a user interface for inputting a stereoscopic image generation parameter for generating a stereoscopic image from an image captured by the camera 100 and a blur parameter for controlling the generated stereoscopic video signal. attach.

The parameters entered are described below. As stereoscopic image generation parameters, in addition to the pixel size m and focal length _fp of the camera 100, the base line length l between the two cameras assumed to capture a stereoscopic image, the parallax adjustment amount Sr, and the weighting coefficient _Wr are _input . As blur parameters, blur adjustment amounts S _n , S _f at positions in front of or behind the subject, weight coefficients W _n , W _f in front of and behind the subject, focus position information P of the subject, Lens information different from the information of the actual imaging lens, that is, a virtual focal length f _v and an aperture value (F number).

In addition, the parallax adjustment amount S _r and the weight coefficient W _r are coefficients that change the distance of the entire subject and the depth distance between the subjects, and produce the following effect: the larger the value, the closer the subject is to the camera and the subject is captured. The smaller the difference in the distance from the camera between the subjects is, the smaller the value is, the farther the subject is from the camera and the larger the difference in the distance from the camera between the subjects. In addition, when the parallax adjustment amount S _r is set to 0 and the weight coefficient W _r is set to 1, a sub image assumed to be captured at a position separated from the camera 100 by the base line length 1 is generated.

In response to the setting of these parameters, the stereoscopic image generation parameters are output to the stereoscopic image generation unit 701 , and the blurring parameters are output to the blurring processing unit 202 .

<Image acquisition unit 701>

The image acquisition unit 701 acquires an image from the camera 100 , generates a stereoscopic video signal, and outputs it to the blur processing unit 202 . The image acquisition unit 701 includes a stereoscopic image generation unit 702 that generates a stereoscopic video signal.

<Stereoscopic Image Generation Unit 702>

The stereoscopic image generation unit 702 generates a stereoscopic video signal including a left-eye image and a right-eye image based on the image input from the camera 100 and the distance information input from the parallax acquisition unit 501 .

A method of generating a stereoscopic image will be described. As described in Embodiment 1, if the parallax information is d, the relationship between the distance D between the subject and the camera is as in Equation 2 ( FIG. 2( a )), so the parallax information d is

[Formula 4]

$\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; D.</span>}}$

By performing such calculations for all the subjects, the parallax value for each subject in the stereoscopic image is obtained.

In this embodiment, the image acquired by the image acquisition unit 701 from the camera 100 is used as the main image as it is, and the pixels of the main image are shifted to use

[Equation 5]

d _r =W _r ·d+S _r

The obtained d _r is used to generate a secondary image.

Also, depending on the distribution of d _r , each pixel on the sub-image may correspond to a plurality of pixels on the main image, or there may be no corresponding pixel on the main image. When a pixel on the sub-image corresponds to a plurality of pixels on the main image, if it corresponds to a pixel closest to the camera among the corresponding pixels, it becomes a natural image. Conversely, for a blank pixel that does not exist in a pixel on the corresponding main image, by copying the data of the surrounding pixels of the pixel and filling the blank, it is possible to prevent an unnatural secondary image from being output.

The stereoscopic image generating unit 702 outputs the generated main image and sub image to the blur processing unit 202 .

<action>

FIG. 14 is a flowchart showing the operation of the image coding device 700 according to this embodiment.

In FIG. 14 , the same symbols are used for the same operations as those in FIG. 4 , and explanations thereof are omitted.

The image processing device 700 acquires an image signal after the setting of the blur parameter is completed ( S20 ), and acquires distance information on each pixel of the acquired image ( S410 ).

The stereoscopic image generator 702 calculates the amount of parallax that should exist in the stereoscopic video signal based on the blur parameter acquired in S10 and the distance information acquired in step S410 . The stereoscopic image generating unit 702 uses the image acquired in step S20 as a main image, and performs stereoscopic image generation for generating a sub image with the calculated parallax amount added to the main image ( S420 ). The stereoscopic image generation unit 702 outputs the generated stereoscopic image to the blurring processing unit 202 .

In this way, in the present embodiment, two images capable of stereoscopic viewing are generated based on one image and distance information of a subject corresponding to the image, and encoded by adding arbitrary blur using the distance information of the subject. Thereby, it is possible to generate a blurred encoded stereoscopic video signal based on the distance information between the image of one viewpoint and the subject.

(Embodiment 4)

FIG. 15 is a block diagram of an image encoding device 710 according to Embodiment 4 of the present invention. In FIG. 15 , the same reference numerals are used for the same components as those in FIG. 13 , and explanations thereof are omitted.

The image encoding device 710 of the present embodiment is characterized in that the distance information between the imaging device and the object is obtained by switching between two focus distances by zooming the camera 120 .

In Fig. 15, the stereoscopic image encoding system obtains two images with different focal distances, calculates the distance of the subject based on the disparity information of the two images, and generates a stereoscopic image signal from one captured image based on the calculated distance information and encodes it. . Such a stereoscopic video encoding system includes the image encoding device 710 according to Embodiment 4, the camera 120 having a zoom function and a variable focal length, the blur parameter input unit 300 , and the storage unit 400 .

<Image encoding device 710>

The image coding device 710 includes a distance detection unit 704 in the parallax acquisition unit 703 , and the distance detection unit 704 is connected to the camera 120 and the blur parameter input unit 300 .

<Fuzzy parameter input part 300>

The blur parameter input unit 300 accepts input of a focal distance f p for generating an image and a focal distance f _{p ′} for distance measurement (f _p′ >f _p ).

The blur parameter input unit 300 outputs two pieces of focus distance information to the distance detection unit 704 .

<Distance detection unit 704>

The distance detecting unit 704 receives the focal distance f _p and the focal distance f _p' from the blur parameter input unit 300, sets the focal distance f _p and the focal distance f _p' alternately for each frame of the camera 120, and measures the distance input from the camera 120. The distance between the camera 120 and the subject for each pixel on the image. The detection of the distance is performed by detecting parallax information of corresponding pixels on two types of images having different focal lengths, and using a measurement principle of triangulation. The distance detection unit 704 outputs the measured distance information to the stereo image generation unit 702 and the blur processing unit 202 .

Description will be made using FIG. 15 . At the focal distance f _p , the distance y _p between the pixel on the imaging element of the object and the optical axis, and the orientation θ _p at the focal distance f _p from the imaging element to the object based on the optical axis relationship is

[Formula 6]

$\mathrm{<span\; class="notranslate">\; the\; tan</span>}{\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}$

The distance y _p' between the pixel on the imaging element of the object and the optical axis at the focal distance f _{p '} and the orientation at the focal distance f p' from the imaging element to the object based on the optical axis The relation of θ _p' is also

[Equation 7]

${\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}}{{\mathrm{<span\; class="notranslate">\; f</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}}$

On the other hand, if the distance between the optical axis and the subject is y, and the distance from the focal point to the subject is z when the focal length f _p' is, then the relationship between θ _p and y, z is

[Formula 8]

${\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; z</span>}}$

Similarly, the relationship between θ _p' and y, z is

[Formula 9]

${\mathrm{<span\; class="notranslate">\; the\; tan</span>}\mathrm{<span\; class="notranslate">\; \&theta;</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; the\; y</span>}}{\mathrm{<span\; class="notranslate">\; z</span>}}$

According to Equation 6 to Equation 9, the distance Z from the subject to the imaging element is

[Formula 10]

$\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{{\mathrm{<span\; class="notranslate">\; p</span>}}^{<span\; class="notranslate">\; \&prime;</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}}$

The distance Z between the subject and the imaging element can be obtained by using Equation 10 from the obtained pixel positions y _p , y _p′ for each subject detection pixel position y _p , y _p′ . In addition, the detection of the pixel positions y _p and y _p' can be performed, for example, by the same method as the method in which the parallax detection unit 201 in Embodiment 1 detects parallax by the block matching method.

The obtained distance Z between the subject and the imaging device is output to the stereoscopic image generation unit 702 .

<Stereoscopic Image Generation Unit 702>

The stereoscopic image generation unit 702 generates a stereoscopic video signal composed of a left-eye image and a right-eye image based on the image input from the camera 120 and the distance information input from the parallax acquisition unit 703 . The operation is the same as that of Embodiment 3, so description thereof will be omitted.

<action>

After the setting of the blur parameter is completed, the image processing device 710 sets the focal distance of the camera 120 to f _p and acquires an image signal (S20), and then sets the focal distance of the camera 120 to f _p' by performing The acquisition of the image signal and the detection of the parallax of the two acquired images are performed to detect the distance information for each subject based on the parallax value ( S410 ).

Thus, in this embodiment, the distance information of the subject is detected from one image and the other image obtained by changing the focus distance, and two images capable of stereoscopic viewing are generated using the detected distance information of the subject. Encoding is performed by adding arbitrary blurring. Thus, it is possible to generate a blurred encoded stereoscopic video signal from two images of one viewpoint with different focal distances using one camera having a zoom function and a variable focal length.

(implementation mode 5)

In this embodiment, a system including the image encoding device according to the present invention will be described.

FIG. 17 is a block diagram showing the configuration of the image coding system according to this embodiment. In FIG. 17 , the same reference numerals are used for the same components as those in FIG. 1 , and explanations thereof are omitted.

The image encoding system includes an image encoding device 200 , a storage unit 400 , a camera 100 a , a camera 100 b , a display unit 800 , and an external recording device 900 .

The camera 100a includes a lens 101a, an imaging element 102a, a main control unit 103a, and a lens control unit 104a. The imaging element 102a is constituted by, for example, a CCD (Charge Coupled Device), acquires an optical signal through the lens 101a, converts the optical signal into an electrical signal, and outputs it to the main control unit 103a. The lens control unit 104a adjusts the focus and the like of the lens 101a under control from the main control unit 103a. The main control unit 103 a is composed of, for example, an IC (Integrated Circuit), acquires an electrical signal output from the imaging element 102 a, and outputs the electrical signal as a video signal to the image coding device 200 . Furthermore, the main control unit 103a performs shutter speed, gain adjustment, focus adjustment, and the like by controlling the imaging element 102a and the lens control unit 103a.

The camera 100b includes a lens 101b, an imaging element 102b, a main control unit 103b, and a lens control unit 104b. These constituent elements included in the camera 100 b are the same as the respective constituent elements included in the camera 100 a.

In addition, the main controllers 103a and 103b perform coordinated operations so that the focus, shutter speed, and the like of the cameras 100a and 100b are the same.

The display unit 800 includes, for example, a liquid crystal display or the like, acquires a stereoscopic video signal from the image coding device 200 , and displays a main image and a sub image represented by the stereoscopic video signal.

The external recording device 900 is configured to incorporate, for example, a BD-R (Blu-ray Disc Recordable). Then, the external recording device 900 acquires the coded video signal from the image coding device 200 and writes the coded video signal to the attached recording medium.

<Other modified examples of this embodiment>

(1) In Embodiment 1, a case was described where a region of 16×16 pixels is cut out, the sum of luminance differences (SAD) is obtained, and the cutout position is searched for as a parallax detection method by the block matching method. The invention is not necessarily limited to this case. For example, the cut-out area may be 8×8 pixels, or a cut-out area with a different aspect ratio such as 8×16 pixels may be used. Alternatively, instead of the sum of luminance differences (SAD), for example, sum of squared luminance differences (SSD; Sum of Squared Difference) or normalized cross-correlation (ZNCC; Zero-mean Normalized Cross-Correlation) may be used.

Alternatively, the method of finding the corresponding position between the main image and the sub-image may not be the block matching method, for example, a pattern cutting method may be used.

(2) In Embodiment 1, a case was described in which a luminance difference is used as a parallax detection method by the block matching method, but the present invention is not necessarily limited to this case. For example, when pixel values are expressed in RGB space, the difference in G value can be used, and when pixel values are expressed in YCbCr space and the brightness Y in the screen is approximately constant, the difference in Cb value or the difference in Cr value can be used. Difference.

Alternatively, for example, when pixel values are expressed in RGB space, SAD may be calculated for R, G, and B values, and the minimum value or average value thereof may be used, or a weighted average value may be used. For example, (SAD of R+SAD of 2×SAD of G+SAD of B)/4 may be used as the parallax value, and the cutout position with the smallest parallax value may be searched. Similarly, when pixel values are expressed in the YCbCr space, SAD may be calculated for each of Y, Cb, and Cr to use the minimum value, or an average value or a weighted average value may be used. In this way, parallax can be detected by any method using all or a part of pixel values.

(3) In Embodiment 1, a case was described in which the blur parameter input unit 300 receives the setting of the blur parameter from the touch panel and the cross key, specifies the position of the subject in the main image, and sets the focus position. The present invention is not necessarily limited to this case. For example, the fuzzy parameter input unit 300 may be any input device such as a keyboard or a dial. Alternatively, for example, if the camera 100a has an autofocus function, the blur parameter input unit 300 may acquire the focus position and focus position information P from the camera 100a.

(4) In Embodiment ₁ , the focus position information _P of the _subject , the camera ₁₀₀ a Baseline length l between 100b, focal distance _fp , pixel size m, virtual focal distance _fv , and aperture value (F value), but the present invention is not necessarily limited to this case. For example, the pixel size m may be acquired from the camera 100 a, and the baseline length (l) may be held in advance by the image encoding device 200 . In this way, sufficient blurring parameters necessary for blurring can be obtained and used.

Alternatively, the image encoding device 200 may also hold a plurality of corresponding modes of the distance from the focus position and the circle of confusion in advance, and present the blurred image using a certain mode to the user, and let the user choose which mode to use. In this way, it is possible to select and implement a blur suitable for the user.

(5) In Embodiment 1, the distance D between the camera and the subject was calculated using the focal distance _{f p of} the optical system used for shooting and the virtual focal distance f _v as blur parameters, However, the present invention is not necessarily limited to this case. For example, when the focus position P is obtained using the parallax value d instead of the actual distance L, the virtual focus distance f _v may be used instead of the focus distance f _p of the optical system used for imaging.

When calculating the distance D between the camera and the subject from the parallax value d, if f _v is used instead of f _p , the distance D' (hereinafter referred to as the virtual distance) calculated from the parallax value becomes the actual distance D is f _v /f _p times, so X and L become f _v /f _p times the actual value. Therefore, the value of the imaginary circle of confusion diameter σ' corresponding to the parallax value d is f _p /f _v times the actual value in the first term on the right in each of Equation 2 and Equation 3. Therefore, by calculating f _v /f _p into the weighting coefficients W _n and W _f in advance, it is possible to obtain a desired value of the virtual circle of confusion diameter σ corresponding to the parallax value d. In addition, the virtual distance L' between the camera and the focal position P is also f _v /f _p times the actual distance L, so the virtual distances from the camera of all subjects and the focal position P are f _v /f p times the actual distance f _p times. Therefore, the anteroposterior relationship between the subject and the focus position, and the anteroposterior relationship between the subjects have the same relationship with respect to the actual distance and the imaginary distance.

That is, except that the imaginary distance between the camera and all points is f _v /f _p times the actual distance, and the distance-dependent component of the imaginary circle of confusion σ' is f _p /f _v times, f will not be used instead of f _p the influence of _v . In addition, as mentioned above, as long as the weighting coefficients W _n and W _f are multiplied by _{f v} /f _p , the influence on σ' can be cancelled. Therefore, in the fuzzy matrix and the fuzzy amount, it is possible to eliminate the Effect of using f _v .

For the above reasons, f _v can also be used as a substitute for f _p . Thereby, it is no longer necessary to use f _p , and the blur parameter can be reduced by one.

(6) In Embodiment 1, the case where the imaginary circle of confusion diameter σ is calculated using Equation 2 when the focal position is farther than the focal position, and the imaginary circle of confusion diameter σ is calculated using Equation 3 when the focal position is closer, but The present invention is not necessarily limited to this case. For example, the diameter of the circle of confusion of the virtual optical system may be used as it is and set to

[Formula 11]

$\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; =</span>\frac{{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; x</span>}}$

It is also possible to perform only addition of an adjustment amount and a weight coefficient without performing approximation.

Alternatively, different σ calculation formulas may be used for each distance range of the subject. For example, as shown in FIG. 2( c ), when the distance is farther than the focal point, different calculation formulas for σ may be used when the distance is farther than an arbitrary distance X1 and when it is closer.

Alternatively, instead of imagining an ideal optical system and adding blur, σ may be calculated using an arbitrary function with X as a variable. For example, as shown in FIG. 2( d ), two focus positions L1 and L2 (L1<L2) may be used, and when the focus position is closer than the focus position L1, the focus position is farther than the focus position L2, and the focus position is farther than the focus position L1. When it is closer than L2, a different σ calculation formula is used. In this way, it is possible to focus on a plurality of subjects at different distances and to blur all subjects that do not exist at any focus position.

In addition, for example, it is also possible to set the weight coefficient as W _g , set the adjustment amount as S _g , and set the gain G of the camera sensor as

[Formula 12]

σ＝W _g ·G·|XL|+S _g

It is also possible to set the weight coefficient as W _iso and the adjustment amount as S _iso . For the ISO sensitivity ISO of the camera sensor, set it to

[Formula 13]

$\mathrm{<span\; class="notranslate">\; \&sigma;</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; W</span>}}_{\mathrm{<span\; class="notranslate">\; iso</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; log</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}\frac{\mathrm{<span\; class="notranslate">\; ISO</span>}}{\mathrm{<span\; class="notranslate">\; 100</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; L</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; iso</span>}}$

In this way, the focus position can be conspicuous without adding blur, and blur can be added for smoothing noise generated due to high gain and ISO sensitivity to make it inconspicuous at a position away from the focus position.

In this way, arbitrary blurring can be added.

(7) In Embodiment 1, as the fuzzy matrix, as shown in FIG. must be limited to this case. For example, as shown in Figure 3(b), it may be a matrix in which the closer to the center, the larger the filter coefficient is, or the filter coefficient inside the square whose diagonal length is the diameter σ of the imaginary circle of confusion may be 1, such that The external filter coefficient is an arbitrary matrix such as 0. Alternatively, a Gaussian filter with σ _g = σ/4 can also be used according to the calculated diameter σ of the imaginary circle of confusion

[Formula 14]

$\mathrm{<span\; class="notranslate">\; G</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}}}\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; g</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; )</span>$

Generate matrix.

These matrices may be freely input by the user, or may be selected by the user from matrices prepared in advance in the image encoding device 200 . In this way, it is possible to select and implement a blur suitable for the user.

(8) In Embodiment 1, a case was described in which the amount of blur is the value of the diameter σ of the virtual circle of confusion, but the present invention is not necessarily limited to this case. For example, in the case of using the fuzzy matrix as shown in FIG. 3( b ), the diameter of the range in which a certain proportion (here, 68.26%) of the filter coefficients is concentrated can also be used as the fuzzy amount.

In addition, also in the case of using a blur matrix as shown in FIG. 3( b ), the average value of filter coefficients may be used as a threshold, and the diameter of a range of values above the threshold may be used as a blur amount. In addition, the threshold may be set to the median value of the filter coefficient or half of the maximum value of the filter coefficient.

Alternatively, in the case of using the matrix using the Gaussian filter represented by Equation 14 of the modification (7), (σ/2) may be used as the threshold.

By doing so, it is possible to calculate the amount of blur from the characteristics of the matrix.

(9) In Embodiment 1, the case where the pixel unit of the blurring process is one pixel was described, but the present invention is not necessarily limited to this case. For example, it may be composed of four pixels whose pixel unit is 2×2 pixels. By doing so, it is possible to perform processing in a unit suitable for blurring processing, and to reduce the amount of processing.

(10) In Embodiment 1 and Embodiment 2, it was described that the block size determination units 210a, 210b, and 610 set the threshold value to 4 pixels or 8 pixels, based on 16×16 and 4×4, or 16×16 and 8× 8 and two block sizes to output block size information, but the present invention is not necessarily limited to this case.

For example, when the number of pixels of the threshold value is H as the length of the diagonal line of the imaging plane of the sensor of the camera, the allowable circle of confusion diameter is (H/N) (N is a value from 1000 to 1500), so it may be Allow twice the diameter of the circle of confusion (H/1300) as the threshold pixel number.

In addition, it is also possible to set a value exceeding twice the maximum value (f ² /(FL)) of the added circle of confusion diameter without approximation, adjustment amount, and weight coefficient, and the above-mentioned allowable circle of confusion diameter (H/1300 ) above 2 times any value is set as the number of pixels to be thresholded. In this way, the number of pixels of the threshold value can be determined according to the sensor size of the camera, and the block size information can be output.

The number of pixels of the threshold is not limited to this, and the number of pixels of the threshold may be set arbitrarily by the user.

(11) In Embodiment 1, the case where the block size determination unit 210 a sets the threshold to 4 pixels and outputs block size information based on two block sizes of 16×16 and 4×4 was described, but the present invention is not necessarily limited to in this case. For example, when a block size of 8×8 can also be used, the block can be output according to three block sizes of 16×16, 8×8, and 4×4 using two of 4 pixels and 8 pixels as thresholds. size information. By doing so, it is possible to increase the block size for an image region with a larger amount of blur.

Alternatively, when the amount of blur is smaller than the threshold, block size information limited to the smallest block size, for example, 4×4 may be output. In this way, it is no longer necessary to perform intra prediction for a plurality of block sizes for an area with a small amount of blur, and the amount of processing can be reduced.

(12) In Embodiment 1, the case where the block size determination unit 210b sets the threshold to 8 pixels and outputs block size information based on two block sizes of 16×16 and 8×8 was described, but the present invention is not necessarily limited to in this case.

For example, it is also possible to use 4 pixels as the threshold, and determine a 4×4 sub-block when the amount of blur is 4 pixels or less for a block determined to be 8×8. In this way, the block size can be changed in multiple stages according to the amount of blur.

Alternatively, when the amount of blur is smaller than the threshold, block size information limited to the smallest block size, for example, 8×8 may be output. In this way, it is no longer necessary to perform motion prediction and parallax prediction for a plurality of block sizes for an area with a small amount of blur, and it is possible to reduce the amount of processing.

In addition, since block sizes of 16×8 and 8×16 can also be used in the H.264 standard, for example, the threshold value can also be set to 8 pixels, and the pixels whose blurring amount is larger than 8 pixels do not use 8×8 or 16 pixels. ×8, 8×16 blocks.

Furthermore, for example, by setting the threshold to both 8 pixels and 6 pixels, the pixels whose blurring amount is larger than 8 pixels do not use 8×8, 16×8, and 8×16 blocks, and the blurring amount is larger than 6 pixels. Larger pixels do not use 8×8.

(13) In Embodiment 1, a case was described in which the detection unit 214 searches for the same or similar blocks in the surrounding 32×32 area of the encoding target image, but the present invention is not necessarily limited to this case. The search for the same or similar blocks can also be performed on a larger area, for example a surrounding 64×64 area. By doing so, it is possible to search for more similar blocks.

Alternatively, for a smaller area such as an 8x8 block, the search for the same or similar blocks may be performed in the surrounding 16x16 area. In this way, the amount of processing can be reduced.

(14) In Embodiment 1, it was explained that both motion compensation prediction and parallax compensation prediction are inter-frame prediction, and perform the same or similar processing, so the block size determination unit 210b, the block division unit 210b, the detection unit 214 and the compensation unit are shared. 215, but the present invention is not necessarily limited to this case. For example, the block size determination unit 210b may be divided into motion compensation prediction and parallax compensation prediction. In this way, the threshold value of the blurring amount used for block size determination can be set to a different value for each process, and an appropriate block size can be determined for each process.

(15) In Embodiment 1, the case where the detection unit 214 searches for the same or similar block with the block size received from the block division unit 212b in the encoding target image and detects a disparity vector has been described, but the present invention is not necessarily limited to this. Condition. For example, the parallax vector may be calculated using the parallax information detected by the parallax detection unit 201 . By doing so, it is possible to omit the block search process and reduce the amount of processing.

(16) In Embodiment 1, a case was described in which the transform unit 216 and the inverse transform unit 220 respectively perform DCT and iDCT, but the present invention is not necessarily limited to this case. For example, the transform unit 216 may perform discrete wavelet transform (DWT; Discrete Wavelet Transform), and the inverse transform unit 220 may perform inverse discrete wavelet transform. In this way, any encoding method using orthogonal transform can be used.

(17) In Embodiment 1, a case was described in which the image coding device 200 performs coding using H.264MVC using two methods of parallax compensation prediction and motion compensation prediction, but the present invention is not necessarily limited to this case. For example, an encoding method without parallax compensation prediction such as H.264 or MPEG-4 may be used. In this way, any encoding method can be used as long as the block size is not limited to one type of encoding method.

(18) In Modification 1 of Embodiment 1, a case was described in which distance obtaining section 110 measures the distance between the camera and the subject at each pixel on an image input from camera 100 using ultrasonic waves, millimeter waves, or the like. The invention is not necessarily limited to this case. For example, distance obtaining section 110 may obtain an image of camera 100 and an image of a viewpoint different from camera 100 , and obtain the distance using parallax. In addition, for example, the distance acquisition unit 110 may use a distance measurement unit of a camera having two or more focal lengths as described in the fourth embodiment. In this way, the parallax acquisition unit 501 acquires parallax or distance information acquired by an arbitrary method, and the image encoding device 500 can perform blur processing and encoding.

(19) In Embodiment 2, the case where the block size determination unit 610 sets the threshold to 8 pixels and outputs block size information based on two block sizes of 16×16 and 8×8 was described, but the present invention is not necessarily limited to in this case. For example, arbitrary sizes such as 4×4, 32×32, 8×16, and 64×32 may be used as the block size, and arbitrary values may be used for the threshold value depending on the block size used.

Alternatively, for example, two of 4 pixels and 8 pixels may be used as thresholds, and the block size information may be output according to three block sizes of 16×16, 8×8, and 4×4. By doing so, it is possible to make the block size larger for an image region with a larger amount of blur.

(20) In Embodiment 2, the case where JPEG2000 is used as the encoding method for still images and Motion JPEG2000 is used for moving images is described, but the present invention is not necessarily limited to this case. Any encoding method can be used as long as it allows a plurality of block sizes and uses an orthogonal transform.

(21) In Embodiment 3, the image encoding device 700 replaces the camera 100 a and the camera 100 b of the image encoding device 200 with the camera 100 , replaces the parallax acquisition unit 207 with the parallax acquisition unit 501 , and replaces the image acquisition unit 206 with the image The structure of the acquisition unit 701, but the present invention is not necessarily limited to this case. For example, in the image encoding device of Embodiment 3, the camera 100a and camera 100b of the image encoding device 600 may be replaced by the camera 100, the parallax acquisition unit 207 may be replaced by the parallax acquisition unit 501, and the image acquisition unit 206 may be replaced by the image acquisition unit. 701 structure. In this way, it is possible to perform encoding using orthogonal transform for transforming into frequency components on a stereoscopic video signal generated from one image and distance information of a subject corresponding to the image.

In addition, similarly, in the image encoding device 710 of Embodiment 4, the camera 100a and the camera 100b of the image encoding device 600 may be replaced by the camera 120, the parallax acquisition unit 207 may be replaced by the parallax acquisition unit 703, and the image acquisition unit 206 may be replaced by The configuration of the image acquisition unit 701 .

(22) In Embodiment 4, a case was described in which the focal distance f _p' for distance detection is longer than the focal distance f _p for image generation (f _p' > f _p ), but the present invention does not necessarily limit in this case. It may also be a case where the focal distance f _p' for distance detection is shorter than the focal distance f _p for image generation (f _p' <f _p ). In this way, as long as the focal distance f _{p '} different from the focal distance f p used for image generation can be set, distance detection can be performed from two images with different focal distances.

(23) In Embodiment 5, the case where the image coding system includes the image coding device 200 was described, but the present invention is not necessarily limited to this case. For example, the image coding system may include the image coding device 600 instead of the image coding device 200 . Also, for example, the image encoding system may include an image encoding device 500 or an image encoding device 700 connected to the camera 100 and the distance obtaining unit 110 instead of the cameras 100 a , 100 b and the image encoding device 200 . In addition, for example, the image encoding system may include an image encoding device 710 connected to the camera 120 instead of the cameras 100 a , 100 b and the image encoding device 200 .

(24) In Embodiment 5, a case was described in which the display unit 800 acquires a stereoscopic video signal from the image coding device 200 and displays the main image and sub-image represented by the stereoscopic video signal, but the present invention is not necessarily limited to this case. . For example, the display unit 800 may acquire a local decoded image signal from the image encoding device 200 and display the main image and sub image indicated by the locally decoded image signal. In this way, the encoded image signal can be monitored.

(25) In Embodiment 5, a case was described in which the external storage device 900 uses a BD-R as a recording medium, but the present invention is not necessarily limited to this case. For example, CD-R (Compact Disc Recordable), MO (Magnet Optical disk), DVD-R (Digital Versatile Disk Recordable), or a semiconductor memory such as an SDHC memory card or SDXC memory card may be used as the storage medium.

(26) The image coding devices of the above-mentioned embodiments can also be realized typically as an LSI (LargeScale Integration) which is an integrated circuit. Each circuit may be used individually as one chip, or may include all or part of the circuits to form one chip. For example, the blur processing unit 202 may be integrated into the same integrated circuit as other circuit units as shown in FIG. 18 , or may be a different integrated circuit.

Here, it is described as LSI, but depending on the degree of integration, it may be called IC (Integrated Circuit), system LSI, super LSI, or ultra-large-scale LSI.

In addition, the method of circuit integration is not limited to LSI, and it can also be realized by a dedicated circuit or a general-purpose processor. FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, and reconfigurable processors that can reconfigure the connection and settings of circuit cells inside the LSI can also be used.

Furthermore, if an integrated circuit technology that replaces LSI emerges due to progress in semiconductor technology or other derivative technologies, it is of course possible to integrate functional modules using this technology.

(27) The above-mentioned descriptions of the respective embodiments and modifications are merely examples of the present invention, and various improvements and modifications can be made without departing from the scope of the present invention.

Hereinafter, configurations and effects of the image encoding device, its integrated circuit, and image encoding method according to the embodiment will be described.

(a) The image encoding device according to the embodiment is characterized by comprising: an image acquisition unit that acquires an image; a disparity acquisition unit that acquires at least one of disparity per pixel or distance information per pixel of the image; The amount determination unit determines the blur amount of each pixel unit of the image according to the parallax of each pixel unit or the distance information of each pixel unit; the blur processing unit performs blur processing on the image according to the blur amount; The block size determination unit determines, from a plurality of block sizes, a block size for cutting out an encoding target area from each image subjected to the blur processing, based on the blur amount; The unit will encode the image after the blurring process described above.

In addition, the integrated circuit of the embodiment is characterized in that it includes: an image acquisition unit that acquires an image; a parallax acquisition unit that acquires at least one of parallax per pixel unit or distance information per pixel unit of the image; blur amount determination a unit that determines a blur amount of each pixel unit of the image according to the parallax of each pixel unit or the distance information of each pixel unit; a blur processing unit that performs blur processing on the above image according to the blur amount; the block size The determining unit determines, from a plurality of block sizes, a block size for cutting out the encoding target region from the image subjected to the blurring process based on the blur amount; Image encoding after blurring above.

In addition, the image encoding method of the embodiment is characterized in that it includes: an image acquisition step of acquiring an image; a disparity acquisition step of acquiring at least one of disparity per pixel or distance information per pixel of the image; The decision step is to determine the blurring amount of each pixel unit of the above-mentioned image according to the parallax of each pixel unit or the distance information of each pixel unit; the blurring processing step is to perform blurring processing on the above-mentioned image according to the above-mentioned blurring amount; In the size determining step, a block size for cutting out an encoding target area from the image subjected to the blurring process is determined from among a plurality of block sizes based on the blurring amount; Image encoding after performing the blurring process described above.

In addition, the image encoding system of the embodiment includes: a camera for taking images, an image encoding device for encoding the image, and a recording medium for recording compressed image data generated by the encoding of the image encoding device. The system is characterized in that the image encoding device includes: an image acquisition unit that acquires an image; a disparity acquisition unit that acquires at least one of disparity per pixel unit or distance information per pixel unit of the image; a blur amount determination unit, Determining a blurring amount of each pixel unit of the above-mentioned image according to the parallax of each pixel unit or the distance information of each pixel unit; the blurring processing unit performs blurring processing on the above-mentioned image according to the blurring amount; the block size determination unit , according to the amount of blurring, determine a block size for cutting out the encoding target area from the image after the blurring process is performed from a plurality of block sizes; The processed image is encoded.

According to these above configurations, when encoding an image, the blur amount is used to determine the block size, so that an appropriate block size corresponding to the blur amount per pixel of the image can be used.

(b) In addition, in the image encoding device of the above (a) of the embodiment, the blur amount may be determined by parallax per pixel or distance information per pixel of the image, and a blur parameter of the image.

Thereby, it is possible to perform various kinds of blurring processing using not only parallax information but also arbitrary blurring information such as focus information and blurring amount for each subject distance.

(c) In addition, in the image coding device of the above (b) of the embodiment, the blur parameter may include an optical characteristic value different from a characteristic value of an optical system that captures the image.

Thereby, it is possible to set a virtual optical system using blur information, and to create an image with the bokeh state of an arbitrary optical system added without being limited to the optical system actually used for imaging, or to create an image by blur processing. The image captured by the optical system.

(d) In addition, in the image coding device of the above (c) of the embodiment, the blur parameter may include focus position information of one or more subjects.

Thereby, blurring processing using focus information can be performed, for example, blurring processing that cannot be realized by an actual optical system, such as the presence of multiple focal points, can be performed.

(e) Furthermore, in the image coding device of the above (a) of the embodiment, the block size determination unit may determine, for a pixel unit in which the blur amount is larger than a threshold value, one of the plurality of block sizes that is not the smallest block size. block size.

As a result, in an encoding method that can use two or more block sizes, the block size used is limited to a larger block size for a region with a larger amount of blur. Compression ratio for large regions. In addition, when performing predictive coding processing, it is not necessary to select a predictive block size compared with the determined block size and block size, and it is possible to further reduce the amount of processing.

(f) In addition, in the image coding device of the above (a) of the embodiment, the block size determination unit may determine, for a pixel unit in which the blur amount is smaller than a threshold value, that the smallest block size among a plurality of block sizes is included in within more than one block size.

In this way, by using variable blocks including small blocks in areas where the amount of blur is small or not blurred, reduction in the amount of information can be prevented. Also, if it is determined to use a plurality of block sizes when performing predictive coding processing, predictive coding processing can be efficiently performed using a large block size when the reference block is limited to a large block size.

(g) In addition, in the image encoding device of the above (e) or the above (f) of the embodiment, the encoding unit may use the above block size determination when performing motion compensation prediction, parallax compensation prediction or intra prediction Similar blocks are searched for each block size determined locally, a block size that can obtain the block most similar to the image divided by the block unit is selected, and a residual signal corresponding to the selected block size is generated.

Thus, when motion prediction, parallax prediction, or intra-frame prediction processing is performed during encoding, a block size suitable for each processing can be determined from the determined variable blocks, and the residual signal can be minimized. , capable of high compression.

(h) In addition, in the image encoding device of the above (a) of the embodiment, the image acquisition unit acquires an image for a left eye and an image for a right eye for stereoscopic image display; the parallax acquisition unit includes detecting and a parallax detection unit for disparity between an area of the image and a corresponding area of the image for the right eye.

In this way, parallax information for each pixel can be generated from the right-eye image and the left-eye image used for stereoscopic image display, and the generated parallax can be used for the stereoscopic image composed of the right-eye image and the left-eye image. The information is obfuscated and encoded using an appropriate block size.

(j) In addition, in the image encoding device of the above (a) of the embodiment, the image acquisition unit may include a stereoscopic image generation unit, and the stereoscopic image generation unit may be based on one acquired image and the above-mentioned The parallax and distance information of the images are used to generate a stereoscopic image composed of an image for the left eye and an image for the right eye.

Thus, two images for stereoscopic viewing can be generated based on one image and the distance information of the subject involved in the image, and blurring and appropriate blocks can be used for the generated stereoscopic image using the distance information of the subject. size encoding.

Industrial Applicability

The present invention relates to an image encoding device for encoding an image signal. In particular, it is possible to add appropriate blur according to the distance of a subject and determine a block size according to the amount of blur, which contributes to high-efficiency encoding and throughput. cuts.

Therefore, it is effective for image recording devices such as digital video cameras and digital still cameras that process video signals. In addition, it is also effective for image distribution equipment, image editing equipment, and the like.

Symbol Description

100, 100a, 100b, 120 cameras

101a, 101b lens

102a, 102b imaging elements

103a, 103b main control unit

110 distance acquisition unit

200, 500, 600, 700, 710 image coding device

201 Parallax detection unit

202 Fuzzy processing department

203 selector

204a, 204b, 204c switches

205 Subtractor

206, 701 Image Acquisition Department

207, 501, 703 Parallax acquisition part

210a, 210b, 610 block size determination unit

212a, 212b, 611 block division unit

213 Intra prediction unit

214 Inspection Department

215 Compensation Department

216 Conversion Department

217 Quantification Department

218 variable length code

219 Inverse quantization part

220 Inverse conversion unit

221 adder

300 Fuzzy parameter input unit

400 storage department

702 Stereoscopic image generation department

704 Distance detection unit

800 Display

900 external storage devices

Claims (12)

1. An image coding device, characterized in that, comprising: The image acquisition unit acquires the image captured by the camera; a parallax acquisition unit that acquires at least one of the parallax per pixel of the image or the distance information between the camera and the object per pixel; a blur amount determination unit that determines a blur amount per pixel of the image based on the parallax per pixel or the distance information per pixel; The blur processing unit performs blur processing on the image according to the blur amount; a block size determining unit that determines, from a plurality of block sizes, a block size for cutting out an encoding target region from each image subjected to the blur processing, based on the blur amount; and The encoding unit encodes the image subjected to the blurring process in units of blocks according to the determined block size. 2. The image encoding device according to claim 1, wherein: The amount of blur is determined by the disparity per pixel of the image or the distance information per pixel of the image, and the blur parameter of the image. 3. The image encoding device according to claim 2, wherein: The aforementioned blur parameter includes an optical characteristic value different from a characteristic value of an optical system that captures the aforementioned image. 4. The image encoding device according to claim 3, wherein: The aforementioned blur parameter includes focus position information of one or more subjects. 5. The image encoding device according to claim 1, wherein: The block size determination unit determines a block size that is not the smallest block size among the plurality of block sizes for a pixel unit in which the blur amount is larger than a threshold value. 6. The image encoding device according to claim 1, wherein: The block size determination unit determines one or more block sizes including a smallest block size among a plurality of block sizes for a pixel unit in which the blur amount is smaller than a threshold value. 7. The image encoding device according to claim 5 or 6, wherein: When performing motion compensation prediction, parallax compensation prediction, or intra-frame prediction, the encoding unit searches for similar blocks using each block size determined by the block size determination unit, and selects an image that can be obtained by dividing an image corresponding to the block unit. The block size of the most similar block, generating a residual signal corresponding to the selected block size. 8. The image encoding device according to claim 1, wherein: The image acquisition unit acquires an image for a left eye and an image for a right eye for stereoscopic image display; The parallax acquisition unit includes a parallax detection unit that detects a parallax between a region of the left-eye image and a corresponding region of the right-eye image. 9. The image encoding device according to claim 1, wherein: The image acquisition unit includes a stereoscopic image generation unit that generates a left-eye image and a right-eye image based on the acquired one image and the parallax or distance information of the image acquired by the parallax acquisition unit. Stereoscopic image. 10. An integrated circuit, characterized in that, comprising: The image acquisition unit acquires the image captured by the camera; a parallax acquisition unit that acquires at least one of the parallax per pixel of the image or the distance information between the camera and the object per pixel; a blur amount determination unit that determines a blur amount per pixel of the image based on the parallax per pixel or the distance information per pixel; The blur processing unit performs blur processing on the image according to the blur amount; a block size determining unit that determines, from among a plurality of block sizes, a block size for cutting out an encoding target region from the blurred image based on the amount of blur; and The encoding unit encodes the image subjected to the blurring process in units of blocks according to the determined block size. 11. An image encoding method, comprising: The image obtaining step is to obtain the image taken by the camera; a parallax obtaining step of obtaining at least one of the parallax per pixel unit of the image or the distance information between the camera and the subject per pixel unit; The blur amount determining step is to determine the blur amount of each pixel unit of the above-mentioned image according to the above-mentioned parallax of each pixel unit or the above-mentioned distance information of each pixel unit; The blurring processing step is to perform blurring processing on the above-mentioned image according to the above-mentioned blurring amount; A block size determining step of determining, from among a plurality of block sizes, a block size for cutting out an encoding target region from the blurred image based on the amount of blur; and In the encoding step, the blurred image is encoded in units of blocks according to the determined block size. 12. An image encoding system comprising a camera for capturing images, an image encoding device for encoding the image, and a recording medium for recording compressed image data generated by encoding by the image encoding device, wherein: The above-mentioned image encoding device includes: The image acquisition unit acquires an image; a parallax acquisition unit that acquires at least one of the parallax per pixel of the image or the distance information between the camera and the object per pixel; a blur amount determination unit that determines a blur amount per pixel of the image based on the parallax per pixel or the distance information per pixel; The blur processing unit performs blur processing on the image according to the blur amount; a block size determining unit that determines, from among a plurality of block sizes, a block size for cutting out an encoding target region from the blurred image based on the amount of blur; and The encoding unit encodes the image subjected to the blurring process in units of blocks according to the determined block size.