JP2004274125A - Image processing apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display a title in an appropriate position. <P>SOLUTION: A title generating part 24 generates title data D, generates a distance parameter E indicating how long distance is secured from a user for displaying a title based on the title data D on a three-dimensional display device on a decoding side, namely, how long distance is secured from a display screen of the three-dimensional display device for locating superimposed characters, and supplies superimposed character data and the distance parameter to a multiplexing part 25. The multiplexing part 25 multiplexes the superimposed character data D and the distance parameter E supplied from the superimposed character generating part 24 on encoded image data C supplied from a stereo encoding part 23 on the basis of a predetermined format and transmits a multiplexed data stream F to a decoding system via a predetermined transmission path or medium. Thus, on the three-dimensional display device, the superimposed characters can be displayed to keep a predetermined distance of depth from the user. This invention can be applied to a stereo video camera. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method suitable for use in encoding and decoding stereo moving images, for example.
[0002]
[Prior art]
Conventionally, various methods have been proposed as encoding methods for stereo moving images. For example, when encoding left and right images using two cameras, one of the images uses a motion-compensated inter-frame prediction encoding method used in MPEG (Moving Picture Experts Group) or the like, while the other image uses the other. There has been proposed a method of encoding a disparity-compensated prediction or a disparity component of an image using an affine transformation to eliminate redundancy.
[0003]
Also, by defining an image of one viewpoint as a reference image, performing motion compensation on the reference image, performing disparity compensation on a disparity image other than the reference image, and selecting a compensation method for each block, Some are encoded (for example, see Patent Document 1).
[0004]
[Patent Document 1]
JP 10-191394 A
In the case of the technology disclosed in Patent Document 1 described above, a stereo moving image has a data rate that is twice that of a plane moving image. I have.
[0006]
However, in this technique, only reduction of the data rate on the encoding side is considered, and display on the decoding side is not considered at all.
[0007]
In recent years, advances in coding technology and improvements in the processing speed of processors have made it possible to reduce the data rate on the coding side, and the spread of ADSL (Asynchronous Digital Subscriber Loop) or the spread of recording media With the increase in recording capacity, reducing the data rate is not always important.
[0008]
[Problems to be solved by the invention]
By the way, when stereo display is performed on the decoding side, for example, subtitles may be superimposed on a stereo moving image and displayed.
[0009]
However, in such a case, since the stereo moving image is a natural image and originally has a parallax, when the stereo moving image is displayed on the display screen on the decoding side, the stereo moving image is displayed so as to give a sense of distance, but the caption is artificial. Since the image is an image and has no parallax in the first place, there is a problem that when the subtitle is displayed on the display screen on the decoding side, it is displayed as being at infinity.
[0010]
In addition, when a stereo moving image is displayed in stereo along with subtitles, the subtitles are displayed as a part of a distant background, and the user feels uncomfortable compared to the flat display in which the subtitles are always displayed in front of the image on the display screen. There was a problem of feeling.
[0011]
The present invention has been made in view of such a situation, and an object of the present invention is to enable a subtitle to be displayed at an appropriate position.
[0012]
[Means for Solving the Problems]
A first image processing apparatus according to the present invention includes: an obtaining unit that obtains image data of a stereo image; a first generating unit that generates synthetic data of a synthetic image to be synthesized with the stereo image; It is characterized by comprising a second generating means for generating a parameter representing a display position in the depth direction, and a multiplexing means for multiplexing the combined data and the parameter with the image data of the stereo image.
[0013]
The composite image may be a caption image.
[0014]
A first image processing method according to the present invention includes an obtaining step of obtaining image data of a stereo image, a first generating step of generating synthesized data of a synthesized image to be synthesized with the stereo image, and a step of generating a synthesized image based on the synthesized data. The method includes a second generation step of generating a parameter representing a display position in the depth direction, and a multiplexing step of multiplexing the combined data and the parameter with image data of a stereo image.
[0015]
The second image processing device according to the present invention is configured to convert, from input data, image data of a stereo image, synthetic data of a synthetic image synthesized with the stereo image, and a parameter representing a display position in the depth direction of the synthetic image based on the synthetic data. Separation means for separating, decoding means for decoding image data of the stereo image, transformation processing means for transforming the synthesized data so that the synthesized image is deformed, and transformation processing means for transforming the image data of the stereo image by the transformation processing means. Combining means for combining the combined data based on the parameters.
[0016]
The composite image may be a caption image.
[0017]
According to a second image processing method of the present invention, from input data, image data of a stereo image, synthetic data of a synthetic image synthesized with the stereo image, and a parameter representing a display position in the depth direction of the synthetic image based on the synthetic data are obtained. A separation step of separating, a decoding step of decoding image data of the stereo image, a deformation processing step of deforming the synthesized data so that the synthesized image is deformed, and a deformation processing step of transforming the stereo image data into image data. A combining step of combining the combined data based on the parameters.
[0018]
In the first image processing apparatus and method of the present invention, image data of a stereo image is obtained, synthetic data of a synthetic image to be synthesized with the stereo image is generated, and a display position in the depth direction of the synthetic image based on the synthetic data is determined. Generated parameters are generated and the combined data and parameters are multiplexed into the image data of the stereo image.
[0019]
In the second image processing apparatus and method according to the present invention, from input data, image data of a stereo image, synthetic data of a synthetic image synthesized with the stereo image, and a display position in the depth direction of the synthetic image based on the synthetic data are determined. The parameters to be represented are separated, the image data of the stereo image is decoded, and the synthesized data is deformed so that the synthesized image is deformed. It is synthesized based on
[0020]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0021]
FIG. 1 is a diagram illustrating the principle of stereoscopic vision when an observer 1 observes a background 2 and a foreground 3 that are objects. Here, the distance between the eyes of the observer 1 is A1, the distance from the observer 1 to the background 2 is B1, and the distance from the observer 1 to the foreground 3 is B2. As shown in FIG. 1, the parallax is different between the distance B1 from the observer 1 to the background 2 and the distance B2 from the foreground 3, and this parallax gives the observer 1 a distance, thereby providing a stereoscopic view. Is possible.
[0022]
FIG. 2 shows a configuration of an encoding system 11 to which the present invention is applied, which obtains a stereoscopic image based on such a principle and encodes it. The encoding system 11 includes a video camera 21, a video camera 22, a stereo encoding unit 23, a caption generation unit 24, and a multiplexing unit 25.
[0023]
The video cameras 21 and 22 arranged on the left and right sides at a predetermined distance in the horizontal plane simultaneously capture the foreground 3 shown in FIG. 1 which is the target object, respectively, and convert an imaging signal corresponding to the captured image into a stereo image. Output to the encoding unit 23.
[0024]
The stereo encoder 23 encodes one of the image signals input from the left and right video cameras 21 and 22 using an encoding method such as a normal MPEG (Moving Picture Experts Group) 2 or MPEG4. , Make the data less redundant. On the other hand, the other image signal is reduced as redundancy from the one image signal by, for example, differential encoding or affine encoding to be encoded data. In both of these data, the one encoded data is used as main encoded data, and the other encoded data is used as so-called enhanced data as sub-encoded data. Is supplied to the multiplexing unit 25.
[0025]
Thus, the main encoded data can be handled as normal MPEG encoded data, and when the decoding side is a conventional flat display device, the conventional MPEG decoding device can be used as it is.
[0026]
The caption generation unit 24 generates caption data D and determines how far the caption is displayed from the user on the decoding-side stereoscopic display device 45 (FIG. 4), that is, the display screen of the stereoscopic display device 45 (shown in FIG. 4). The distance parameter (parameter indicating the position in the depth direction) E indicating how far the subtitles should be displayed from the subtitles is generated and supplied to the multiplexing unit 25.
[0027]
For example, the caption generation unit 24 can generate three distance parameters E indicating that the caption is displayed such that the caption is displayed at a distance of 2 m from the user. Thereby, the caption is displayed on the stereoscopic display device 45 so as to be at a distance of 2 m from the user. Therefore, when displaying a stereo moving image of a movie in stereo, even if there is a scene change, the user can always see the subtitles at the same distance (for example, 2 m from the user) without feeling uncomfortable.
[0028]
Further, the caption generation unit 24 can also generate a special code specifying a display screen of the stereoscopic display device 45 as the distance parameter E. Thus, in the stereoscopic display device 45, the caption is displayed as if it were on the display screen, regardless of the image displayed on the display screen. That is, the caption is displayed in the same manner as the conventional flat display (display other than stereo display).
[0029]
Thus, on the decoding side, based on the distance parameter E generated by the caption generation unit 24, the display position (display position) of the caption based on the caption data D on the display screen is controlled separately from the stereo moving image. be able to.
[0030]
The multiplexing unit 25 multiplexes the encoded image data C supplied from the stereo encoding unit 23, the subtitle data D supplied from the subtitle generating unit 24, and the distance parameter E based on a predetermined format.
[0031]
The multiplexed data stream F is transmitted to the decoding system 31 (FIG. 4) via a predetermined transmission path or medium (neither is shown) in a format as shown in FIG. In the example of FIG. 3, audio data is arranged next to the MPEG data (encoded image data C), followed by subtitle data D and a distance parameter E.
[0032]
FIG. 4 shows a configuration of a decoding system 31 to which the present invention is applied, which decodes a data stream transmitted from the encoding system 11 of FIG. The decoding system 31 includes a demultiplexer 41, a decoding unit 42, a caption transformation unit 43, a display control unit 44, and a stereoscopic display device 45.
[0033]
The demultiplexer 41 separates the data stream F supplied from the encoding system 11 (FIG. 2) into encoded image data C, subtitle data D, and a distance parameter E via a transmission path or a medium. The demultiplexer 41 supplies the separated encoded image data C to the decoding unit 42, and supplies the separated subtitle data D and the distance parameter E to the subtitle transformation unit 43.
[0034]
The decoding unit 42 performs predetermined processing on the encoded image data C input from the demultiplexer 41, decodes the encoded image data C into a stereo moving image, and supplies the stereo moving image to the display control unit 44. The caption transformation unit 43 performs a predetermined image conversion process on the caption data D based on the distance parameter E supplied from the demultiplexer 41, and converts the caption data D into the right channel caption data D1 and the left channel caption data. It is transformed into D2 and supplied to the display control unit 44.
[0035]
The display control unit 44 acquires the stereo moving image data from the decoding unit 42, acquires the right channel subtitle data D1 and the left channel subtitle data D2 from the subtitle deforming unit 43, and superimposes them on the stereoscopic display device. 45. The stereoscopic display device 45 acquires a stereo moving image and superimposed data of the right channel subtitle data D1 and the left channel subtitle data D2 from the display control unit 44, and displays the corresponding image on a display screen (not shown).
[0036]
Next, an encoding process of a stereo moving image in the encoding system 11 will be described with reference to a flowchart of FIG.
[0037]
In step S1, the left and right video cameras 21 and 22 simultaneously capture, for example, as shown in FIG. 6, an object existing within an imaging range 61 indicated by a broken line in the figure. In the case of the example in FIG. 6, a background 2 and a foreground 3 are present in an object to be photographed. Also, as shown in FIG. 6, the caption 62 “the season of fresh green” (that is, an image based on the caption data D) is displayed at the bottom of the screen and in front.
[0038]
In step S2, the stereo encoding unit 23 performs predetermined processing on the imaging signal captured in the processing in step S1 by an encoding method such as MPEG to generate encoded image data C with less redundancy. , To the multiplexing unit 25.
[0039]
In step S3, the caption generation unit 24 generates caption data D and supplies the caption data D to the multiplexing unit 25. In step S4, the caption generation unit 24 sets a distance parameter E indicating how far from the display screen (not shown) of the stereoscopic display device 45 (FIG. 4) the caption based on the caption data D is displayed. Generated and supplied to the multiplexing unit 25.
[0040]
FIG. 7 shows a display distance of a caption on the stereoscopic display device 45. As shown in FIG. 7, there is almost no parallax for an object at almost infinity, such as a building or a cloud in the background 2. In contrast, the trees in the foreground 3 have parallax. Also, since the subtitles 62 are displayed on the display screen (not shown) so as to be located closer to the foreground 3, the subtitles 62 must generate a larger parallax than the foreground 3. This distance is appropriately set by the user.
[0041]
In step S5, the multiplexing unit 25 converts the encoded image data C supplied from the stereo encoding unit 23, and the subtitle data D and the distance parameter E supplied from the subtitle generating unit 24, based on a predetermined format. Multiplex. In step S6, the multiplexing unit 25 transmits the multiplexed data stream to the decoding system 31 (FIG. 4) via a predetermined transmission path or medium.
[0042]
As described above, the distance parameter E indicating how far from the display screen of the stereoscopic display device 45 the caption data D is displayed is superimposed on the data stream encoded by the encoding system 11. , The decoding side can be notified of the appropriate display position of the caption at the time of decoding. Therefore, in the decoding system 31, the caption can be stereoscopically displayed at a predetermined distance based on the distance parameter E.
[0043]
The processing of step S3 and step S4 may be performed in the reverse order or in parallel.
[0044]
Next, decoding processing of a stereo moving image in the decoding system 31 will be described with reference to the flowchart in FIG.
[0045]
In step S21, the demultiplexer 41 separates the encoded image data C, the subtitle data D, and the distance parameter E from the data stream F supplied from the encoding system 11 via a transmission path or a medium. The demultiplexer 41 supplies the separated encoded image data C to the decoding unit 42, and supplies the separated subtitle data D and the distance parameter E to the subtitle transformation unit 43. In step S22, the decoding unit 42 performs predetermined processing on the encoded image data C input from the demultiplexer 41, decodes the encoded image data C into stereo moving image data, and supplies the stereo moving image data to the display control unit 44.
[0046]
In step S23, the caption transformation unit 43 performs a predetermined image conversion process on the caption data D based on the distance parameter E supplied from the demultiplexer 41, and converts the caption data D into the right channel caption data D1. It is transformed into subtitle data D2 for the left channel. That is, the caption deforming unit 43 appropriately applies the caption data D to the user on the display screen of the stereoscopic display device 45 at a predetermined distance (the distance set by the distance parameter E). Perform a deformation process that gives a great parallax.
[0047]
Specifically, for example, image conversion processing is performed on the caption data D using a simple grid transformation.
[0048]
As shown in FIG. 9, the subtitle data D supplied from the demultiplexer 41 to the subtitle transformation unit 43 includes an original subtitle 151 that is originally free from distortion. The subtitle transformation unit 43 obliquely transforms the horizontal component of each character of the original subtitle 151 included in the subtitle data D, based on the distance parameter E supplied from the demultiplexer 41, as shown in FIG. 151, the subtitle data 152-1 for the right channel and the subtitle data 152-2 for the left channel are generated.
[0049]
As a result, the original subtitle 151 having no distortion is transformed in consideration of the parallax as if photographed by the left and right video cameras (the subtitle data for right channel 152-1 and the subtitle data for left channel 152- 2). Therefore, appropriate parallax can be given to the subtitle data D so that the subtitle data D is located at a predetermined distance to the user within the display screen of the stereoscopic display device 45.
[0050]
The subtitle transformation unit 43 supplies the right channel subtitle data D1 and the left channel subtitle data D2 to the display control unit 44.
[0051]
In step S24, the display control unit 44 acquires the stereo moving image data from the decoding unit 42, and acquires the right channel subtitle data D1 and the left channel subtitle data D2 from the subtitle deforming unit 43. The images are synthesized so as to form an image and supplied to the stereoscopic display device 45. In step S25, the stereoscopic display device 45 displays, on a display screen (not shown), an image based on the stereo moving image data in which the subtitle data D1 for the right channel and the subtitle data D2 for the left channel supplied from the display control unit 44 are combined. indicate.
[0052]
In this way, the decoding device 31 can display the caption stereoscopically at a predetermined distance on the display screen of the stereoscopic display screen 45.
[0053]
In the above description, subtitles are combined with a stereo moving image. However, the present invention can be applied to a case where a combined image other than subtitles is combined.
[0054]
The series of processes described above (the encoding process in FIG. 5 and the encoding process in FIG. 8) can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer built into dedicated hardware or installing various programs. It is installed in a possible, for example, a general-purpose personal computer from a network or a recording medium.
[0055]
FIG. 10 shows a configuration of a personal computer. In FIG. 10, a CPU 201 executes various processes according to a program stored in a ROM 202 or a program loaded from a storage unit 208 into a RAM 203. The RAM 203 also appropriately stores data necessary for the CPU 201 to execute various processes.
[0056]
The CPU 201, the ROM 202, and the RAM 203 are mutually connected via a bus 204. An input / output interface 205 is also connected to the bus 204.
[0057]
The input / output interface 205 includes an input unit 206 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), an output unit 207 including a speaker, and a storage including a hard disk. A communication unit 209 including a unit 208, a modem, a terminal adapter, and the like is connected. The communication unit 209 performs communication processing via a network including the Internet (not shown).
[0058]
A drive 210 is connected to the input / output interface 205 as necessary, and a magnetic disk 221, an optical disk 222, a magneto-optical disk 223, a semiconductor memory 224, or the like is appropriately mounted. It is installed in the storage unit 208 as needed.
[0059]
As shown in FIG. 10, a program storage medium that is installed in a computer and stores a program that can be executed by the computer includes a magnetic disk 221 (including a floppy disk), an optical disk 222 (CD-ROM (Compact Disk)). A package medium including a read only memory (DVD) (including a digital versatile disk), a magneto-optical disk 223 (including an MD (mini-disk)), or a semiconductor memory 224, or a program temporarily or permanently. And a hard disk constituting the storage unit 228. The storage of the program in the program storage medium is performed using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via an interface such as a router or a modem as necessary.
[0060]
In this specification, the steps of describing a program stored in a program storage medium may be performed in chronological order according to the described order, or may be performed in parallel, even if not necessarily performed in chronological order. Alternatively, it also includes individually executed processing.
[0061]
【The invention's effect】
As described above, according to the present invention, a stereo moving image can be encoded. In particular, it is possible to notify the decoding side of an appropriate display position of the caption at the time of decoding.
[0062]
Further, a stereo moving image can be decoded. Also, the decoding side can display the caption at an appropriate distance.
[Brief description of the drawings]
FIG. 1 is a diagram for explaining the principle of stereoscopic vision.
FIG. 2 is a block diagram illustrating a configuration of an encoding system to which the present invention has been applied.
FIG. 3 is a diagram showing a configuration of a data stream.
FIG. 4 is a block diagram illustrating a configuration of a decoding system to which the present invention has been applied.
5 is a flowchart illustrating a stereo moving image encoding process in the encoding system of FIG. 2;
FIG. 6 is a diagram for explaining a photographing range.
FIG. 7 is a diagram illustrating a display distance of a caption in a stereoscopic display device.
FIG. 8 is a flowchart illustrating a decoding process of a stereo moving image in the decoding system of FIG. 4;
FIG. 9 is a diagram for explaining a transformation by image conversion in the caption transformation unit of FIG. 4;
FIG. 10 is a block diagram illustrating a configuration of a personal computer.
[Explanation of symbols]
Reference Signs List 11 encoding system, 21 and 22 video camera, 23 stereo encoding unit, 24 caption generation unit, 25 multiplexing unit, 31 decoding system, 41 demultiplexer, 42 decoding unit, 43 caption transformation unit, 44 display control unit, 45 3D display device

Claims (6)

Acquiring means for acquiring image data of a stereo image,
First generating means for generating synthetic data of a synthetic image to be synthesized with the stereo image;
Second generating means for generating a parameter representing a display position in the depth direction of the composite image based on the composite data;
An image processing apparatus comprising: a multiplexing unit that multiplexes the combined data and the parameter into image data of the stereo image. The image processing apparatus according to claim 1, wherein the composite image is a caption image. An acquisition step of acquiring image data of a stereo image,
A first generating step of generating synthetic data of a synthetic image to be synthesized with the stereo image;
A second generation step of generating a parameter representing a display position in a depth direction of a composite image based on the composite data;
A multiplexing step of multiplexing the composite data and the parameter with image data of the stereo image. Separating means for separating, from the input data, image data of a stereo image, synthesized data of a synthesized image synthesized with the stereo image, and a parameter indicating a display position in a depth direction of the synthesized image based on the synthesized data;
Decoding means for decoding the image data of the stereo image;
Transformation processing means for transforming the composite data so that the composite image is transformed;
An image processing apparatus comprising: a synthesizing unit configured to synthesize the synthesized data subjected to the deformation processing by the deformation processing unit with the image data of the stereo image based on the parameter. The image processing apparatus according to claim 4, wherein the composite image is a caption image. A separation step of separating, from the input data, image data of a stereo image, synthesized data of a synthesized image synthesized with the stereo image, and a parameter indicating a display position in a depth direction of the synthesized image based on the synthesized data;
A decoding step of decoding image data of the stereo image,
A transformation processing step of transforming the composite data so that the composite image is transformed;
A synthesizing step of synthesizing, based on the parameters, the synthesized data deformed in the deformation processing step with the image data of the stereo image.

Images