JP2012230478A - Image processing apparatus, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus for generating a stereoscopic image easy to see even when displaying a display area such as a menu picture on a stereoscopic image in which a plurality of subjects having respectively different solidity are included.SOLUTION: The image processing apparatus for processing contents including at least images from a plurality of view points and making a viewer perceive a stereoscopic image includes: a first overlapped image generation part for generating a first overlapped image which is an overlapped image for notifying information to a user and is displayed overlappedly to an image obtained from a first view point out of the plurality of view points; a second overlapped image generation part for generating a second overlapped image to be displayed overlappedly to an image from the second view point out of the plurality of view points; and an image composition part for combining the first overlapped image with the image from the first view point and combining the image from the second view point with the second overlapped image. Parallax between the first overlapped image and the second overlapped image is changed in a longitudinal direction or a lateral direction.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  When a stereoscopic image is displayed on the stereoscopic image display device, these images are displayed so that only the left eye image can be seen from the left eye and only the right eye image can be seen from the right eye. Thereby, the viewer can view stereoscopically. Such a stereoscopic view can be performed by performing a stereoscopic display using parallax included between a plurality of images obtained by photographing the same subject from different positions. There are roughly two methods for performing stereoscopic display. One is a glasses-type stereoscopic display device that is viewed by a viewer wearing glasses. The other is an autostereoscopic display device in which a viewer can view stereoscopically with the naked eye. The glasses-type stereoscopic display device includes a time-division method in which a left-eye image and a right-eye image are alternately switched and displayed, and a polarization method in which the polarization directions are superimposed and displayed. In autostereoscopic display devices, a parallax barrier method that alternately arranges left-eye images and right-eye images on the back side of a slit-like opening called a parallax barrier, or a kamaboko-shaped lenticular lens is arranged. There is a lenticular method that spatially separates an image for the left eye and an image for the right eye.

  There has been proposed a compound-eye imaging device having a plurality of imaging units that performs imaging for performing stereoscopic display as described above. Some of these compound-eye imaging devices include a stereoscopic display device. The compound-eye imaging device can generate a stereoscopic image for stereoscopic display from images acquired by a plurality of imaging units, and stereoscopically display the generated stereoscopic image on the stereoscopic display device. When it is desired to perform some operation on the compound-eye imaging apparatus, it is necessary to display a menu screen for instructing the operation in the stereoscopic image. However, the amount of projection and retraction of the stereoscopic image, that is, the stereoscopic effect, varies depending on the distance of the subject included in the image from the imaging unit. For this reason, in a menu screen in which characters and symbols are superimposed on a stereoscopic image so as to have a preset stereoscopic effect, if the stereoscopic effect of the subject included in the image is different from the stereoscopic effect of the displayed characters and symbols, There was a problem that a sense of incongruity would occur in and symbols.

  In order to solve such a problem, a stereoscopic image display device that generates a parallax display area according to the parallax of a subject in a stereoscopic image when displaying a predetermined display area such as a menu screen for performing various operations. Has been proposed (see Patent Document 1).

JP 2009-210840 A

  If a subject close to the camera and a subject far from the camera are photographed at the same time and a plurality of subjects with different stereoscopic effects are included in the photographed image, a stereoscopic image with depth can be photographed. However, in the stereoscopic image display device of Patent Document 1, a menu is generated so as to have a parallax according to the parallax of the subject. Therefore, when shooting a stereoscopic image with depth, a subject that is close to the camera and has a large amount of parallax and a subject that is far from the camera and has a small amount of parallax are mixed, and the parallax of the menu is equal to the parallax of any subject. It will be a response. For this reason, when the parallax of the subject included in the image is various, the subject that is greatly different from the menu is displayed. At this time, if the line of sight is passed from the menu to the subject, it is necessary to greatly activate the eye convergence function, so that the stereoscopic image cannot be displayed due to the presence of the menu even though the stereoscopic image is displayed on the screen. There is a problem that it may end up.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a stereoscopic image that is easy to see even when a display area such as a menu screen is displayed on a stereoscopic image that includes a plurality of subjects having different stereoscopic effects. An object is to provide an image processing apparatus, an image processing method, and a program to be generated.

(1) The present invention has been made to solve the above-described problems, and one aspect of the present invention is an image process for processing content that includes at least images from a plurality of viewpoints and allows a viewer to perceive a stereoscopic image. 1st superimposition image generation which is an apparatus and is a superimposition image which notifies a user of information, and generates a 1st superimposition image displayed on a picture from the 1st viewpoint among these viewpoints. A second superimposed image generating unit that generates a second superimposed image to be displayed overlaid on an image from a second viewpoint among the plurality of viewpoints, and the first superimposed image And an image composition unit for compositing the second superimposed image with the superimposed image from the second viewpoint, and the first superimposed image, the second superimposed image, The parallax between is characterized by changing in the vertical or horizontal direction. To.

(2) Moreover, the other aspect of this invention is the above-mentioned image treatment apparatus, Comprising: The parallax between the said 1st superimposed image and the said 2nd superimposed image is changing continuously. It is characterized by that.

(3) Further, another aspect of the present invention is the above-described image processing device, wherein the parallax between the first superimposed image and the second superimposed image becomes 0 as the image end approaches. It is characterized by approaching.

(4) Further, another aspect of the present invention is the above-described image processing device, and restricts a parallax between the first superimposed image and the second superimposed image within a predetermined range. A parallax restriction unit, and the second superimposed image generation unit applies the parallax restricted by the parallax restriction unit to the first superimposed image to generate the second superimposed image. It is characterized by.

(5) According to another aspect of the present invention, in the above-described image processing device, the parallax limiting unit is configured to perform the predetermined range according to a size at which the image combined by the image combining unit is displayed. It is characterized by determining.

(6) In addition, another aspect of the present invention is an image processing method for processing content that includes at least images from a plurality of viewpoints, and that allows a viewer to perceive a stereoscopic image, and is a superimposed image that notifies a user of information. A first process of generating a first superimposed image displayed to be superimposed on an image from a first viewpoint among the plurality of viewpoints, and a second viewpoint among the plurality of viewpoints. A second process of generating a second superimposed image to be displayed overlaid on the image, and the first superimposed image is synthesized with an image from the first viewpoint, and the superimposed image from the second viewpoint is And a third step of synthesizing the second superimposed image, and the parallax between the first superimposed image and the second superimposed image changes in the vertical direction or the horizontal direction. It is characterized by that.

(7) According to another aspect of the present invention, a computer of an image processing apparatus that processes content that includes at least images from a plurality of viewpoints and makes a viewer perceive a stereoscopic image, and a superimposed image that notifies a user of information A first superimposed image generation unit that generates a first superimposed image that is displayed superimposed on an image from the first viewpoint among the plurality of viewpoints, and a second viewpoint among the plurality of viewpoints A second superimposed image generating unit for generating a second superimposed image to be displayed overlaid on the image from the first, the first superimposed image is synthesized with the image from the first viewpoint, and the superimposed from the second viewpoint The image is caused to function as an image synthesis unit that synthesizes the second superimposed image, and the parallax between the first superimposed image and the second superimposed image changes in the vertical direction or the horizontal direction. It is a program characterized by this.

  According to this invention, even when a display area such as a menu screen is superimposed on a stereoscopic image including a plurality of subjects having different stereoscopic effects, an easily viewable stereoscopic image can be generated.

It is a schematic block diagram which shows the structure of the stereo image imaging device 100 in one Embodiment of this invention. It is FIG. (1) for demonstrating parallax. It is FIG. (2) for demonstrating parallax. It is FIG. (3) for demonstrating parallax. It is FIG. (4) for demonstrating parallax. It is a figure which shows the example of the display position table which the memory 106 in the embodiment stores beforehand. It is a figure which shows the example of the menu in the case of displaying on the right end of the display screen in the embodiment. It is a figure which shows the example of the parallax image for arrangement | positioning "right end" in the embodiment. It is the figure which represented the parallax image 402 in the same embodiment in the three-dimensional space. It is a figure which shows the example of the menu image in the case of displaying on the upper end of the display screen in the embodiment. It is a figure which shows the example of the parallax image for arrangement | positioning "upper end" in the embodiment. It is the figure which represented the parallax image 602 in the same embodiment in the three-dimensional space. It is a figure showing having displayed the to-be-photographed object with predetermined | prescribed parallax on a large sized, medium sized, and small display screen. It is a figure which shows the example of the parallax normalization table in the embodiment. It is a figure explaining operation | movement of the menu screen production | generation part 105a for left eyes in the embodiment. It is a figure explaining operation | movement of the menu screen production | generation part 105b for right eyes in the embodiment. It is a figure which shows the example of the display screen for left eyes and the display screen for right eyes which the image synthetic | combination part 111 in the embodiment produces | generates. It is a figure which shows the position where each area | region is perceived by the viewer, when the stereoscopic image in the embodiment is stereoscopically viewed. It is a figure which shows another example of the display screen for left eyes and the display screen for right eyes which the image synthetic | combination part 111 in the embodiment produces | generates. It is a figure which shows the position where each area | region is perceived by the viewer, when another stereoscopic image in the embodiment is stereoscopically viewed. It is a flowchart explaining operation | movement of the stereo image imaging device 100 in the embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In addition, the figure which shows positional relationships, such as FIG. 2, is exaggerated for easy understanding, and is different from the actual interval and size. FIG. 1 is a schematic block diagram showing a configuration of a stereoscopic image capturing apparatus 100 that is an image processing apparatus according to an embodiment of the present invention. As shown in FIG. 1, the stereoscopic image capturing apparatus 100 according to the present embodiment includes a bus 120, an image acquisition unit 112, a menu screen parallax normalization unit 104 (parallax restriction unit), a menu screen generation unit 105, a memory 106, and a display unit 107. , An input unit 108, a stereoscopic image generation unit 110, and an image composition unit 111. The bus 120 connects the above-described units so that data can be transmitted to each other. The image acquisition unit 112 includes two imaging units 101L and 101R, two signal processing units 102L and 102R, and an imaging control unit 103. The menu screen generation unit 105 includes a left-eye menu screen generation unit 105a (first superimposed image generation unit) and a right-eye menu screen generation unit 105b (second superimposed image generation unit).

  The imaging units 101L and 101R are cameras that respectively capture a subject and acquire a left-eye image L and a right-eye image R. Each of the imaging units 101L and 101R includes an imaging element such as a photographing lens and a CCD (Charge Coupled Device). The imaging control unit 103 controls the imaging units 101L and 101R to drive the focus position, zoom magnification, shutter, and the like using the imaging lens. In addition, the imaging unit 101L and the imaging unit 101R are arranged at a predetermined interval, and the base line length that is the interval between the imaging units 101L and 101R and the convergence angle between the imaging units 101L and 101R are changed. Can do. The convergence angle between the imaging units 101L and 101R is an angle formed by the imaging direction of the imaging unit 101L, the imaging direction of the imaging unit 101R, and the subject.

  Each of the signal processing units 102L and 102R includes a left-eye image L and a right-eye image R for the left-eye image L and the right-eye image R acquired by the corresponding imaging units 101L and 101R, respectively. The process which correct | amends the brightness | luminance difference between these, the process which correct | amends the difference in sharpness, and the process which correct | amends distortion and inclination are performed. Furthermore, image processing such as white balance adjustment processing, shading correction, and color correction is performed on the corrected image.

  The input unit 108 is an input device such as a button or a touch panel for operating the stereoscopic image capturing apparatus 100. The menu screen generation unit 105 generates a menu screen including a menu based on an operation on the input unit 108 by the user. Here, the menu is an image for notifying the user of operation candidates for performing various operations on the stereoscopic image capturing apparatus 100. This menu screen is composed of a left-eye menu screen and a right-eye menu screen so that the viewer can perceive it as having a depth by stereoscopic viewing. The left-eye menu screen generation unit 105a of the menu screen generation unit 105 generates this left-eye menu screen, and the right-eye menu screen generation unit 105b generates a right-eye menu screen. As will be described later, the right-eye menu screen generation unit 105b applies the parallax according to the minimum and maximum parallax values determined by the menu screen parallax normalization unit 104 to the left-eye menu screen. The right eye menu screen is generated. The details of the left-eye menu screen generation unit 105a and the right-eye menu screen generation unit 105b will be described later.

  The menu screen parallax normalization unit 104 determines the minimum and maximum values of the parallax between the left-eye menu screen and the right-eye menu screen described above, that is, the size and resolution of the display screen of the display unit 107 described later, that is, described later. This is determined in accordance with the size and resolution at which the synthesis result by the image synthesis unit 111 is displayed. When the stereoscopic image capturing apparatus 100 is connected to a display device such as a digital television receiver with an HDMI (registered trademark) cable and the menu is displayed on the display screen of the display device, the display screen of the display device is displayed. The maximum value and the minimum value of the parallax are determined according to the size and resolution. The memory 105 associates the display position of the menu with the parallax image, the parallax image, the parallax image, the parallax image that correlates the maximum value and the minimum value (range of parallax) with the display screen size and resolution. Store the normalization table. These tables and parallax images are used when the right-eye menu screen generation unit 105b generates the right-eye menu screen. Details of the table and parallax image stored in the menu screen parallax normalization unit 104 and the memory 105 will be described later. The display unit 107 includes a display screen such as a liquid crystal panel, and displays a stereoscopic image that can be stereoscopically viewed by a parallax barrier method.

The image composition unit 111 converts the image L for the left eye acquired by the image acquisition unit 112 into a resolution corresponding to the display unit 107, and superimposes the menu screen for the left eye on the conversion result, thereby displaying the display screen for the left eye Is synthesized. Further, the image composition unit 111 converts the right-eye image R acquired by the image acquisition unit 112 into a resolution corresponding to the display unit 107, and superimposes a right-eye menu screen on the conversion result so that the right-eye image is displayed. Synthesize the display screen. Note that when there is no menu to be displayed, the image composition unit 111 does not synthesize the menu screen, and converts the left-eye image L into a resolution corresponding to the display unit 107 as the left-eye display screen. A right-eye display screen is obtained by converting the right-eye image R in the same manner.
The stereoscopic image generation unit 110 generates an image signal for stereoscopic display on the display unit 107 from the left-eye display screen and the right-eye display screen. Specifically, the stereoscopic image generation unit 110 generates an image signal for displaying a stereoscopic image by alternately arranging the pixels of the display screen for the left eye and the pixels of the display screen for the right eye in the horizontal direction. To the display unit 107. In this way, when alternately arranged in the horizontal direction, the display unit 107 uses the parallax barrier method, so that the pixels on the display screen for the left eye can be viewed only from the left eye of the viewer, and the display for the right eye The pixels on the screen can be seen only from the viewer's right eye.

  Here, parallax in the case of performing stereoscopic viewing will be described. 2 to 5 are diagrams for explaining the parallax. 2 to 5, LE is the left eye viewpoint, and RE is the right eye viewpoint. S is a screen surface on which an image is displayed. P and P1 are left eye and right eye gazing points, RP and RP1 are right eye gazing points, and LP and LP1 are left eye gazing points. First, as shown in FIG. 2, when a stereoscopic image is viewed without performing stereoscopic viewing, that is, when the left eye and the right eye both watch the point P on the screen surface S, The image displayed on P is perceived as being on the screen surface S. At this time, the distance from the viewer to the position at which the viewer perceives the image, that is, the distance to the screen surface S is defined as a virtual distance 201 that serves as a reference for the stereoscopic effect of the subject.

  In a stereoscopic image, if the positive parallax is large, it is perceived at a position close to the viewer. Here, the parallax is a difference between a position in the image of a point in the image for the left eye and a position in the image of a point in the image for the right eye corresponding to the point. Specifically, it is a value obtained by subtracting the distance from the left end of the image for the right eye of the corresponding point from the distance from the left end of the image for the left eye of a certain point. Here, the corresponding points are a point where there is a subject displayed in the image for the left eye and a point where the same subject is displayed in the image for the right eye. When the viewer gazes at a certain subject in the stereoscopic image, the right eye gaze point corresponds to the left eye gaze point.

  As shown in FIG. 3, when the viewer gazes at a certain subject in the stereoscopic image, if the right-eye gazing point RP is on the left side of the left-eye gazing point LP, the parallax is positive. At this time, the line of sight of the left eye and the line of sight of the right eye intersect before the screen surface S. The stereoscopic image of the subject that the viewer is gazing is perceived to be at a position where the line of sight of the left eye and the line of sight of the right eye intersect. Accordingly, the stereoscopic image of the subject being watched by the viewer is perceived as being positioned in front of the screen surface S. In this case, the virtual distance 202 from the both eyes to the position where the stereoscopic image is perceived is smaller than the virtual distance 201.

  On the other hand, as shown in FIG. 4, when the viewer gazes at a certain subject in the stereoscopic image and the right eye point of interest RP is on the right side of the left eye point of interest LP, negative parallax and Become. At this time, the line of sight of the left eye and the line of sight of the right eye intersect behind the screen surface S. Therefore, the stereoscopic image of the subject being watched by the viewer is perceived as being located behind the screen surface S. In this case, the virtual distance 203 from the two eyes to the position where the stereoscopic image is perceived is larger than the virtual distance 201.

Furthermore, in a stereoscopic image, the parallax may vary depending on the portion of the subject. For example, in FIG. 5, when gazing at a certain part of the subject, the right-eye gazing point RP1 is on the right side of the left-eye gazing point LP1, and when gazing at another part, the right-eye gazing point and left Both eye gaze points coincide at point P2. In this case, the virtual distance from the two eyes to the position where the stereoscopic image is perceived is a virtual distance 204 that is perceived so that a certain part is located behind the screen surface S, and the other part is on the screen surface S. The virtual distance 205 is perceived as being located, and the stereoscopic image is perceived as being arranged from the front to the back.
As described above, the subject included in the stereoscopic image can be seen as a stereoscopic image of the subject popping out from the front side of the screen surface when the stereoscopic image is stereoscopically viewed according to the parallax, or retracted to the back side from the screen surface. Or it seems to be arranged from the front to the back.

  FIG. 6 is a diagram illustrating an example of a display position table stored in advance in the memory 106. As illustrated in FIG. 6, the display position table stored in the memory 106 includes information indicating a menu arrangement such as “right end” and “top end”, and information indicating a parallax image applied to the menu screen at the time of the arrangement. Are stored in association with each other. When the menu is placed at the right edge, the menu is placed at the right edge of the display screen. When it is at the top edge, the menu is placed at the top edge of the display screen. "Indicates that the menu is arranged at the left end of the display screen, and" bottom end "indicates that the menu is arranged at the lower end of the display screen. Note that coordinates (for example, upper left and lower right coordinates) in the menu display screen may be used as information indicating the menu layout.

  The parallax image is an image having the same size as the menu and representing the parallax at each pixel by the pixel value of the pixel. In the example of FIG. 6, the reference information of the parallax image D1 is stored in association with the arrangement “right end”. Similarly, the reference information of the parallax image D2 is stored in association with the arrangement “upper end”, the reference information of the parallax image D3 is stored in association with the arrangement “left end”, and is associated with the arrangement “lower end”. Reference information of the parallax image D4 is stored. Note that the memory 106 also stores these parallax images D1 to D4 in advance.

  FIG. 7 is a diagram illustrating an example of a menu when displayed on the right end of the display screen. A menu image 401 illustrated in FIG. 7 is a left-eye menu image generated by the left-eye menu screen generation unit 105a. Here, the menu image is an image including only a menu display area, and the menu screen is an image including an area corresponding to a display image displayed on the display unit 107 including the menu image. The left-eye menu image is an image including only a menu display area on the left-eye display screen. After generating the left-eye menu image, the left-eye menu screen generating unit 105a generates a left-eye menu screen on which the left-eye menu image is arranged. The menu image 401 has a horizontal length (number of pixels in the x-axis direction) of w1 pixels and a vertical length (number of pixels in the y-axis direction) of h1 pixels. In the example shown in FIG. 7, a title indicating a menu (“menu”) and five selection items (“• item 1” to “• item 5”) are arranged vertically.

FIG. 8 is a diagram illustrating an example of a parallax image for the arrangement “right end”. The parallax image 402 shown in FIG. 8 is an example of the parallax image when the menu is displayed at the right end of the display screen, that is, the parallax image D1 in FIG. Therefore, the parallax image 402 is stored in the memory 106. The parallax image 402 has the same size (vertical and horizontal pixel numbers) as the menu image 401, the horizontal length is w1 pixels, and the vertical length is h1 pixels. In the parallax image 402, the parallax is represented by 0 to -255. When the upper left corner of the parallax image 402 is the origin, the parallax changes from −255 to 0 as the x-axis value changes from 0 to w1. As described with reference to FIGS. 2 to 5, when the parallax is 0, it is perceived as being on the display screen surface when viewed stereoscopically. Further, when the parallax is positive, it is perceived as being located in front of the display screen surface, and when the parallax is negative, it is perceived as being located behind the display screen. Accordingly, in the stereoscopic image according to the parallax image 402, the perceived position changes from the back to the display screen surface from the display screen as the x-axis value changes from 0 to w1, that is, the stereoscopic image It has changed from large to small. Here, the parallax is represented by 0 to −255, but may have a positive value such as −255 to 255.
FIG. 9 is a diagram showing the parallax image 402 in a three-dimensional space. As shown in FIG. 9, in the parallax image 402, as the x-axis value changes from 0 to w1, the parallax value changes linearly and continuously in the horizontal direction, and the parallax at the right end is 0. It has become.

  10 to 12, a menu image and its parallax image when the menu is arranged at the upper end of the display screen will be described. The examples shown in FIGS. 10 to 12 are examples in which the parallax of the menu changes linearly in the vertical direction. FIG. 10 is a diagram illustrating an example of a menu image when displayed on the upper end of the display screen. A menu image 601 illustrated in FIG. 10 is a left-eye menu image generated by the left-eye menu screen generation unit 105a. The menu image 601 has a horizontal length (number of pixels in the x-axis direction) of w2 pixels and a vertical length (number of pixels in the y-axis direction) of h2 pixels. In the example shown in FIG. 10, a title indicating a menu and three selection items arranged side by side are arranged.

  FIG. 11 is a diagram illustrating an example of a parallax image for the arrangement “upper end”. A parallax image 602 shown in FIG. 11 is an example of the parallax image when the menu is displayed at the upper end of the display screen, that is, the parallax image D2 in FIG. Therefore, the parallax image 602 is stored in the memory 106. The parallax image 602 has the same size (number of vertical and horizontal pixels) as the menu image 601, and the parallax changes from 0 to -255 as the y-axis value changes from 0 to h2. Therefore, in the stereoscopic image according to the parallax image 602, as the y-axis value changes from 0 to h2, the perceived position changes from the display screen surface to the back of the display screen. It has changed from small to large. Here, the parallax is represented by 0 to −255, but may have a positive value such as −255 to 255.

  FIG. 12 is a diagram showing the parallax image 602 in a three-dimensional space. As shown in FIG. 12, in the parallax image 602, as the y-axis value changes from 0 to h2, the parallax value changes linearly and continuously in the vertical direction, and the parallax at the upper end is 0. It has become. In these parallax images 402 and 602, the parallax value approaches 0 as it approaches the longest side among the sides in contact with the display screen edge, that is, approaches 0 as it approaches the frame of the display screen of the display unit 107. It has become. By doing this, the perceived positions of the frame of the display screen and the end of the menu match, so that they can be stereoscopically viewed without a sense of incongruity, and the stereoscopic video is easy to see.

  The menu screen parallax normalization unit 104 normalizes the parallax included in the parallax image according to the size of the display screen. Here, the relationship between the size of the display screen and the parallax will be described. FIG. 13 is a diagram illustrating that a subject having a predetermined parallax is displayed on a large, medium, and small display screen. Large display screens are liquid crystal televisions, plasma televisions, projectors, and the like. The medium display screen is a portable liquid crystal television such as a car navigation system. The small display screen is a display device of a digital camera or a mobile phone.

  FIG. 13A is a diagram showing a state where a subject with parallax A is displayed on a large display screen. A distance 800 in the figure is a distance between the left eye image and the right eye image on the screen at this time. FIG. 13B is a diagram illustrating a state in which a subject having a parallax A is displayed on a medium-sized display screen. A distance 801 in the figure is a distance between the left eye image and the right eye image on the screen at that time. FIG. 13C is a diagram illustrating a state in which an object with parallax A is displayed on a small display screen. A distance 802 in the figure is a distance between the left eye image and the right eye image on the screen at that time. The magnitude relationship between these distances 800, 801, and 802 is distance 800> distance 801> distance 802. That is, even with a subject having the same parallax A, the parallax on the screen increases as the display screen becomes larger. Therefore, it is necessary to set the parallax maximum and minimum values according to the size of the display screen.

  Therefore, in this embodiment, the memory 106 stores a parallax normalization table that associates the size (number of inches, resolution) of the display screen with the minimum and maximum values of parallax. FIG. 14 is a diagram illustrating an example of a parallax normalization table. The value of the parallax normalization table uses the comfortable parallax range defined in the 3DC safety guidelines issued by the 3D consortium (3D consortium, “3DC safety guidelines for the spread of 3D that are human-friendly”, p. 20-22, April 20, 2010).

  The maximum parallax that enables stereoscopic viewing when the viewer views an image displayed on a display screen with HD (High Definition, horizontal 1920 pixels × vertical 1080 pixels) at a position that is more than the standard viewing distance from the screen. The value is 57 pixels. The standard viewing distance is a distance that is three times the screen height of the screen surface. Under this condition, the range of the maximum parallax does not depend on the screen size but depends on the resolution. Therefore, the maximum parallax of a display screen with a resolution of HD is set to 57 pixels. About the minimum parallax, it sets so that the parallax on a screen surface may be below the distance between pupils. The interpupillary distance is the distance between the left eye and the right eye of a human, and here is 5 cm in consideration of the child. That is, the minimum parallax is the number of pixels corresponding to 5 cm on the display screen. This depends on the size and resolution of the display screen. Therefore, when the resolution is HD on a 60-inch display device, the number of pixels corresponding to the interpupillary distance of 5 cm is -72 pixels, -87 pixels when the resolution is 50 inches and HD, -87 pixels when the resolution is HD, and the resolution is HD when the resolution is 40 inches. -108 pixels.

  In this way, the parallax normalization table of FIG. 14 stores the minimum parallax “−72” and the maximum parallax “57” in association with the display screen size “60 inches, resolution 1920 × 1080”. . Similarly, the minimum parallax “−87” and the maximum parallax “57” are stored in association with the display screen size “50 inches, resolution 1920 × 1080”, and the display screen size “40 inches, resolution 1920 × The minimum parallax “−108” and the maximum parallax “57” are stored in association with “1080”.

The menu screen parallax normalization unit 104 normalizes the parallax image using the parallax normalization table. Specifically, first, the menu screen parallax normalization unit 104 reads the minimum and maximum parallax values stored in the parallax normalization table in association with the size of the display screen. If the maximum value of the parallax image selected by the right-eye menu screen generation unit 105b is 0 and the minimum value is a negative value, the maximum value remains 0 and the minimum value is the minimum value read from the parallax normalization table. Normalize so that When the minimum value of the parallax image selected by the right-eye menu screen generation unit 105b is 0 and the maximum value is positive, the minimum value remains 0 and the maximum value is the maximum value read from the parallax normalization table. Normalize so that When the minimum value of the parallax image selected by the right-eye menu screen generation unit 105b is a negative value and the maximum value is a positive value, normalization is performed so that the minimum value read from the parallax normalization table is the maximum value. Apply. By doing so, a menu screen having a parallax value of 0, that is, a menu screen stereoscopically viewed on the display surface, can be displayed with the same stereoscopic effect even if it is normalized, Become. In the present embodiment, the parallax image selected by the right-eye menu screen generation unit 105b has a negative minimum value (−255) and a maximum value of 0. Therefore, the menu screen parallax normalization unit 104 performs the normalized parallax. Is calculated using the following equation (1).
Pn = P × Mn / M (1)
Here, P is the parallax value of the parallax image, Mn is the minimum parallax value read from the parallax normalization table, and M is the minimum parallax value in the parallax image.

  FIG. 15 is a diagram for explaining the operation of the left-eye menu screen generation unit 105a. The example shown in FIG. 15 is an example when the menu arrangement is “right end”. As shown in FIG. 15, the left-eye menu screen generation unit 105a superimposes the generated menu image 401 on the right end of the left-eye menu screen M11 as it is. The menu image 401 is the menu image 401 shown in FIG. In the left-eye menu screen Ml1, the pixel value of the area where the menu image 401 is not arranged is a value indicating transparency (for example, 0xFFFFFF).

  FIG. 16 is a diagram for explaining the operation of the right-eye menu screen generation unit 105b. The example shown in FIG. 16 is an example when the menu arrangement is “right end”, as in FIG. 15. The right-eye menu screen generation unit 105b generates the right-eye menu image 401r by applying the normalized parallax image 402n to the menu image 401 generated by the left-eye menu screen generation unit 105a. . Furthermore, the right-eye menu screen generation unit 105b superimposes this menu image 401r on the right end of the right-eye menu screen Mr1. Here, since the parallax is 0 at the right end of the normalized parallax image 402n, the menu image 401r is arranged at the right end of the right-eye menu screen Mr1. When there is a parallax at the right end of the normalized parallax image 402n, the menu image 401r is arranged at a position shifted in the horizontal direction by the parallax value. Further, in the right-eye menu screen Mr1, the pixel value in the area where the menu image 401r is not arranged is a value indicating transparency, as in the right-eye menu screen Mr1. In this way, the menu screens are shifted and superimposed according to the corresponding parallax images. By doing so, parallax that changes in the horizontal direction can be given to the menu screen itself, and a stereoscopic effect can be given to the menu screen.

A specific example of the operation of the menu screen generation unit 105 will be described. First, a method will be described in which the left-eye menu screen generation unit 105a generates a left-eye menu screen. The menu position on the left-eye menu screen is determined so that the start position (upper left coordinates) of the menu image is determined from the menu arrangement such as “upper end” and “right end” described above, and the upper left corner of the menu image is the start position. Superimpose images. The coordinates (Xl, Yl) when the pixel at the coordinates (x, y) in the menu image for the left eye is arranged in the menu screen for the left eye are calculated using the following equations (2) and (3). To do. This completes the generation of the left-eye menu screen.
Xl = x + X10 (2)
Yl = y + Yl0 (3)
Here, X10 and Y10 are start positions (upper left coordinates) of the menu image. For example, if the resolution of the display screen is 1920 × 1080, the horizontal width of the menu image for the left eye is 400, and the arrangement is at the right end, Xl0 = 1920−400 = 1520 and Yl0 = 0.

Next, a method in which the right-eye menu screen generation unit 105b generates a right-eye menu image will be described. The right-eye menu image is arranged so as to be shifted from the left-eye menu image by the amount of parallax of the parallax image. The coordinates (Xr, Yr) when the pixel at the coordinates (x, y) in the left-eye menu image is converted into the right-eye menu image using the parallax image and arranged in the right-eye menu screen are as follows: It calculates using the following formula | equation (4) and (5).
Xr = x−D (x, y) + X10 (4)
Yr = y + Y10 (5)
Here, D (x, y) represents a parallax value at coordinates (x, y) in the normalized parallax image.

  As a result of arranging the pixels in the left-eye menu image on the right-eye menu screen using the above-described equations (4) and (5), priority is given to a pixel with a large parallax in a portion where the pixels overlap. If a gap is created as a result of the arrangement, interpolation is performed using surrounding pixels. This completes the generation of the right-eye menu screen.

  FIG. 17 is a diagram illustrating an example of a left-eye display screen and a right-eye display screen generated by the image composition unit 111. In this example, the left-eye menu image 401 is arranged at the right end of the left-eye display screen Dl1, and the right-eye menu image 401r is arranged at the right end of the right-eye display screen Dr1. In this example, the left-eye display screen and the right-eye display screen are stereoscopic images on which menu screens with parallax changing in the horizontal direction are superimposed.

  FIG. 18 is a diagram illustrating positions where each area is perceived by the viewer when the stereoscopic image including the left-eye display screen D11 and the right-eye display screen Dr1 in FIG. 17 is stereoscopically viewed. In this way, the menu 401p is perceived as if the depth has changed in the horizontal direction from the back to the display screen SF surface rather than the display screen SF. Other subjects O1 to O4 in the image acquired by the other image acquisition unit 112 are perceived as being located in front of the display screen SF like the subject O1, or displayed like the subjects O2 to O4. Some are perceived as being located behind the screen SF.

  FIG. 19 is a diagram illustrating another example of the left-eye display screen and the right-eye display screen generated by the image composition unit 111. In this example, a left-eye menu image 601 is arranged at the upper end of the left-eye display screen Dl2, and a right-eye menu image 601r is arranged at the upper end of the right-eye display screen Dr2. In the example of FIG. 19, the left-eye display screen and the right-eye display screen are stereoscopic images on which menu screens with parallax changing in the vertical direction are superimposed.

  FIG. 20 is a diagram illustrating positions where each region is perceived by the viewer when the stereoscopic image including the left-eye display screen D12 and the right-eye display screen Dr2 in FIG. 19 is stereoscopically viewed. In this way, the menu 401p is perceived as if the depth has changed in the horizontal direction from the back to the display screen SF surface rather than the display screen SF. The other subjects O1 to O4 in the image acquired by the other image acquisition unit 112 are the same as those in FIG.

  FIG. 21 is a flowchart for explaining the operation of the stereoscopic image capturing apparatus 100. First, the input unit 108 acquires an operation input by a user (S101). Next, the left-eye menu screen generation unit 105a determines a menu display item based on the operation input acquired in step S101 (S102). As a result of the determination in step S102, it is determined whether or not a menu is displayed (S103). As a result of this determination, when the image is not displayed (S103-No), the image acquisition unit 112 acquires the left eye image L (S104) and the right eye image R (S105). These are used as the left-eye display image and the right-eye display image as they are. Then, the process proceeds to step S115 described later.

  On the other hand, when the menu is displayed as a result of the determination in step S103, the process proceeds to step S106, and the left-eye menu screen generation unit 105a determines the menu arrangement. This arrangement may be determined based on the display item of the menu determined in step S102, or may be determined based on the operation input acquired in step S101. Next, the left-eye menu screen generation unit 105a generates a left-eye menu screen (S107). Next, the imaging unit 101L and the signal processing unit 102L of the image acquisition unit 111 acquire the left-eye image L (S108). Then, the image synthesis unit 111 synthesizes the left-eye display image from the left-eye menu screen and the left-eye image L (S109).

  Next, the menu screen parallax normalization unit 104 determines the minimum and maximum parallax values based on the size of the display screen (S110). Next, the menu screen parallax normalization unit 105 normalizes the parallax image selected by the right-eye menu screen generation unit 105b based on the menu arrangement based on the minimum and maximum parallax values determined in step S110. (S111). Next, the right-eye menu screen generation unit 105b generates a right-eye menu screen from the left-eye menu image generated by the left-eye menu screen generation unit 105a and the normalized parallax image. Next, the imaging unit 101R and the signal processing unit 102R of the image acquisition unit 111 acquire the right-eye image R (S113). Then, the image synthesis unit 111 synthesizes the right-eye display image from the right-eye menu screen and the right-eye image R (S114).

  Next, the stereoscopic image generation unit 110 generates an image signal of a stereoscopic image from the display image for the left eye and the display image for the right eye (S115). Then, the display unit 107 displays a stereoscopic image based on this image signal (S116), and returns to step S101.

  As described above, according to the present embodiment, when an instruction to display an image for notifying the user such as a menu is displayed when a stereoscopic image is displayed, an image having a depth is generated on the image itself, and the image is superimposed. The displayed stereoscopic image is displayed. For this reason, when the displayed stereoscopic image is stereoscopically viewed, the image itself such as the menu has a depth such as retraction or popping. Accordingly, the viewer can stereoscopically view images such as menus as well as stereoscopic images without a sense of incongruity.

Also, if the image itself, such as a menu, has a depth, if the subject in the stereoscopic video is widely distributed from the front to the back, a portion of the same stereoscopic effect as the stereoscopic effect on which the subject is displayed is included in the image. Exists. Therefore, the viewer can stereoscopically view the image without greatly changing the convergence angle of the eyes, and can generate a stereoscopic image that is easy to see.
In this embodiment, the case of a menu, which is an example of an image for notifying the user of information, has been described. However, the present invention may be used for a digital broadcast information screen, a program guide, a gadget, a promotion screen, subtitles, and the like.

  In the above embodiment, the image acquisition includes the imaging units 101L and 101R, the signal processing units 102L and 102R, and the imaging control unit 103 as content that includes images from a plurality of viewpoints and makes the viewer perceive a stereoscopic image. Although the example in which the stereoscopic image captured by the unit 112 is applied has been shown, content acquired by a broadcast video receiving unit, a recorded video playback unit, or the like included in a television receiver, a hard disk recorder, a Blu-ray disc player, a mobile phone, or the like You may apply to. That is, a television receiver, a hard disk recorder, a Blu-ray disc player, and a mobile phone may be applied as the image processing device.

  In the stereoscopic image generation unit 110 of the above embodiment, the image signal of the stereoscopic image is generated by alternately arranging the pixel of the display image for the left eye and the pixel of the display image for the right eye. When displaying, an image signal corresponding to a connection interface with the display device or the like is generated. For example, the left-eye display image and the right-eye display image are arranged on the left and right, the top-and-bottom arrangement, the left-eye image on the odd lines and the right-eye image on the even lines, There are an eye image arranged in an odd number column and a right eye image arranged in an even number column, and a left eye display image and a right eye display image arranged alternately in a time direction.

  In the above embodiment, the left-eye menu image generation unit 105a superimposes the menu image on the left-eye menu screen without using the parallax image. However, the parallax as in the right-eye menu image generation unit 105b occurs. The menu image may be shifted by the amount of parallax using the image and superimposed on the left-eye menu screen.

  In the above embodiment, the right-eye menu image is generated using the left-eye menu image and the left-eye parallax image, but the left-eye menu image and the right-eye menu image are generated. You may make it produce | generate the menu image for right eyes using a parallax image. The right-eye parallax image is a parallax image indicating the parallax with the left-eye menu image in each pixel of the right-eye menu image.

In the above embodiment, the left eye is the first viewpoint and the right eye is the second viewpoint, and the right eye menu image is generated from the left eye menu image and the left eye parallax image. However, it may be reversed.
In the above embodiment, the parallax of the parallax image is changed in the vertical direction or the horizontal direction. However, the parallax may be changed in the vertical direction and the horizontal direction, that is, in the oblique direction.
In the above embodiment, the parallax of the parallax image is linearly changed in the vertical direction or the horizontal direction, but may be changed nonlinearly.

  1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed, thereby realizing the functions of the respective units. May be. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100 ... Stereoscopic imaging device 101L, 101R ... Imaging part 102L, 102R ... Signal processing part 103 ... Shooting control part 104 ... Menu screen parallax normalization part 105 ... Menu screen generation part 105a ... Menu screen generation part 105b for left eyes ... Right Eye menu screen generation unit 106 ... Memory 107 ... Display unit 108 ... Input unit 110 ... Stereoscopic image generation unit 111 ... Image composition unit 112 ... Image acquisition unit 120 ... Bus

Claims (7)

An image processing device that processes content that includes at least images from a plurality of viewpoints and allows a viewer to perceive a stereoscopic image,
A superimposed image for notifying a user of information, a first superimposed image generating unit that generates a first superimposed image that is displayed superimposed on an image from a first viewpoint among the plurality of viewpoints;
A second superimposed image generating unit that generates a second superimposed image to be displayed superimposed on an image from a second viewpoint among the plurality of viewpoints;
An image synthesizing unit that synthesizes the first superimposed image with the image from the first viewpoint, and synthesizes the superimposed image from the second viewpoint with the second superimposed image;
The parallax between the first superimposed image and the second superimposed image changes in the vertical direction or the horizontal direction.   The image processing apparatus according to claim 1, wherein a parallax between the first superimposed image and the second superimposed image continuously changes.   The image processing apparatus according to claim 2, wherein a parallax between the first superimposed image and the second superimposed image approaches 0 as the image end approaches. A parallax limiting unit that limits parallax between the first superimposed image and the second superimposed image within a predetermined range;
The second superimposed image generation unit generates the second superimposed image by applying the parallax restricted by the parallax restriction unit to the first superimposed image. The image processing device according to claim 3.   The image processing apparatus according to claim 4, wherein the parallax restriction unit determines the predetermined range according to a size at which an image synthesized by the image synthesis unit is displayed. An image processing method for processing content that includes at least images from a plurality of viewpoints and allows a viewer to perceive a stereoscopic image,
A first process of generating a first superimposed image for notifying a user of information, wherein the first superimposed image is displayed to be superimposed on an image from a first viewpoint among the plurality of viewpoints;
A second step of generating a second superimposed image to be displayed overlaid on an image from a second viewpoint among the plurality of viewpoints;
A third step of synthesizing the first superimposed image with the image from the first viewpoint, and synthesizing the superimposed image from the second viewpoint with the second superimposed image;
The parallax between the said 1st superimposed image and the said 2nd superimposed image is changing to the vertical direction or the horizontal direction, The image processing method characterized by the above-mentioned. A computer of an image processing apparatus that processes content that includes at least images from a plurality of viewpoints and that allows a viewer to perceive a stereoscopic image;
A superimposed image for notifying a user of information, a first superimposed image generating unit that generates a first superimposed image that is displayed superimposed on an image from a first viewpoint among the plurality of viewpoints;
A second superimposed image generating unit that generates a second superimposed image to be displayed overlaid on an image from a second viewpoint among the plurality of viewpoints;
Combining the first superimposed image with the image from the first viewpoint, and causing the superimposed image from the second viewpoint to function as an image combining unit that combines the second superimposed image;
A parallax between the first superimposed image and the second superimposed image changes in a vertical direction or a horizontal direction.