JP2003284094A - Stereoscopic image processing method and apparatus therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that optimum programming is difficult in stereoscopically displaying with various types of display devices because of a difference in suitable parallax and this causes a heavy drag on wider use of a stereoscopic image. <P>SOLUTION: A stereoscopic sense adjusting unit 112 displays a stereoscopic image for a user. When a displayed object comes within a limit parallax, the user responds to the unit 112. A parallax control unit 114 generates a parallax image so that the suitable parallax can be attained in the succeeding stereoscopic display, according to the obtained suitable parallax information. The control of parallax is realized by optimally setting a camera parameter retroactive to three-dimensional data. A function for realizing the suitable parallax is provided in a library format. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image processing technique, and more particularly to a method and apparatus for generating or displaying a stereoscopic image based on a parallax image.

[0002]

2. Description of the Related Art In recent years, the inadequacy of network infrastructure has been regarded as a problem, but at the time of transition to broadband, the number of types and the number of contents that effectively utilize a wide band are becoming conspicuous. Video has always been the most important means of expression, but most of the efforts so far have involved improving display quality and data compression ratios, and compared to these, technical possibilities that expand the possibilities of expression. There is a feeling that such efforts are falling behind.

Under such circumstances, stereoscopic image display (hereinafter, simply referred to as stereoscopic display) has been variously studied, and has been put to practical use in a limited market which uses theaters and special display devices. In the future, R & D in this direction will accelerate in order to provide more realistic content, and it is expected that individual users will enjoy stereoscopic display at home.

[0004]

In such a flow, some problems have been pointed out in stereoscopic display from before. For example, it is difficult to optimize the parallax that causes a stereoscopic effect. Originally, it does not really project a three-dimensional object, but it projects the image by shifting it to the left and right eyes in pixel units, so it is not easy to give that artificial three-dimensional effect a natural feeling. Absent.

Further, too much parallax may be a problem, and an observer of a stereoscopic image (hereinafter, also simply referred to as a user).
Some people may complain of mild discomfort. Of course, this is not limited to stereoscopic display, and there are various factors such as the situation in which the displayed scene does not match the situation or sense of one's surroundings. However, as a rule of thumb, such a problem is apt to be observed when the parallax is too large, in other words, when the stereoscopic effect is too strong.

Although the above is the physiological story of human beings, there are other technical factors that prevent the spread of stereoscopic video contents and applications. Although stereoscopic vision is realized by parallax, even if parallax is represented by the amount of pixel shift between the left and right images, there are cases where the same stereoscopic image can be appropriately stereoscopically viewed due to differences in the hardware of the display device. . When the parallax that expresses the distance exceeds the inter-eye distance,
Theoretically, stereoscopic vision is not possible. As the resolutions and screen sizes of display devices are diversifying such as PCs (personal computers), television receivers, and portable devices like today, various hardware is considered to create optimal content for stereoscopic display. Is more difficult, or more accurate, if no methodology is given for it.

Further, even if the methodology is given, it would be difficult for a general programmer to understand it and expect it to be used for creating contents and applications.

The present invention has been made in view of such a background, and an object thereof is to generate or display a stereoscopic image which is easily adapted to human physiology. Another object is to generate or display a stereoscopic image suitable for the user even if the display target image or the display device changes. Still another object is to adjust the stereoscopic effect by a simple operation when the stereoscopic display is performed. Yet another object is to reduce the burden on the programmer when creating a content or an application that can display appropriately in three dimensions. Still another object is to provide a technology for realizing an appropriate stereoscopic display as a business model.

[0009]

DISCLOSURE OF THE INVENTION The knowledge of the inventor, which is the basis of the present invention, is that the proper parallax is determined by once determining factors such as the hardware of the display device and the distance between the user and the display device (hereinafter, these are collectively referred to as “hardware”). Wear it ”). That is, the expression of the proper parallax is generalized by the camera interval and the optical axis crossing position, which will be described later, and then described in a general-purpose form that does not depend on hardware. "Independent of hardware" means that it is basically unnecessary to read out the hardware information peculiar to the display device. If this general-purpose description is made, the rest of the parallax image will be based on the proper parallax. Is generated or adjusted, a desired stereoscopic display is realized.

By providing a control for obtaining the proper parallax and realizing the proper parallax at the time of stereoscopic display of an image with a library, a general programmer can call the library and be aware of complicated stereoscopic principles and programming. Proper stereoscopic display is realized without doing.

Of the various aspects of the present invention, the first group is based on the technique of acquiring the proper parallax based on the response of the user. This technique can be used for the “initial setting” of the parallax by the user, and once the proper parallax is acquired in the device, the proper parallax is realized even when another image is displayed thereafter. However, this technology is not limited to the initial settings,
It is also used for "manual adjustment" in which the user appropriately adjusts the parallax of the image being displayed. The first group will be described below.

The present invention relates to a stereoscopic image processing apparatus, and an instruction acquisition unit for acquiring a user's response to a stereoscopic image displayed based on a plurality of viewpoint images corresponding to different parallaxes.
A parallax specifying unit that specifies an appropriate parallax for the user based on the acquired response.

The instruction acquisition unit is provided as, for example, a GUI (graphical user interface, the same applies below), and first displays while changing the parallax between viewpoint images. When the user obtains the stereoscopic effect he / she likes, he / she inputs it by operating the buttons.

A "stereoscopic image" is an image displayed with a stereoscopic effect, and the substance of the data is a "parallax image" in which a plurality of images have parallax. A parallax image is generally a set of two-dimensional images. Each image forming the parallax image is a “viewpoint image” having a corresponding viewpoint. That is, a parallax image is composed of a plurality of viewpoint images, and when it is displayed, it is displayed as a stereoscopic image. Displaying a stereoscopic image is also simply referred to as “stereoscopic display”.

The "parallax" is a parameter for producing a stereoscopic effect and can be defined in various ways, but as an example, it can be expressed by the shift amount of pixels representing the same point between viewpoint images.
Hereinafter, in the present specification, the definition will be used unless otherwise specified.

The proper parallax may be specified in a range. In that case, both ends of the range will be called “limit parallax”.
The “identification of the appropriate parallax” may be performed with the maximum value that is allowable as the parallax of the near object described later.

The stereoscopic image processing apparatus of the present invention may further include a parallax control unit for performing processing so that the specified proper parallax is realized even when another image is displayed. When another image is a stereoscopic image generated by using three-dimensional data as a starting point, the parallax control unit may determine a plurality of viewpoints for generating the stereoscopic image according to the appropriate parallax. More specifically, the distance between a plurality of viewpoints and the intersection position of the optical axes that see the object from those viewpoints may be determined. An example of these processes is performed by the camera placement determination unit described later. If these processes are performed in real time, optimal stereoscopic display is always realized.

The parallax control unit may perform control so that proper parallax is realized in a predetermined basic three-dimensional space to be displayed. An example of this processing is performed by the projection processing unit described later.

The parallax control unit may perform control so that the proper parallax is realized with respect to the coordinates of the object located closest in the three-dimensional space and the coordinates of the object located farthest away. An example of this processing is performed by the projection processing unit described later. The object may be static.

"Close-up" means a line of sight of a camera placed at each of a plurality of viewpoints, that is, a plane at an optical axis intersection position (hereinafter referred to as "optical axis intersection position") (hereinafter referred to as "optical axis intersection plane"). ) Refers to a state where the parallax is stereoscopically seen before. On the contrary, “distant position” refers to a state in which parallax is provided so that a stereoscopic view can be seen behind the optical axis intersecting plane.
The closer the parallax of the near object is, the closer the user is perceived, and the farther the parallax of the far object is, the farther the user is from the user. That is, unless otherwise specified, the parallax is defined as a non-negative value in which positive and negative are not inverted in the near position and the far position, and the near position parallax at the optical axis crossing plane,
The distance parallax is zero.

In a portion having no parallax in the displayed object or space, the optical axis intersecting surface coincides with the screen surface of the display device. This is because, in pixels with no parallax, the lines of sight seen from both the left and right eyes have just reached the same position on the screen surface, that is, intersected there.

When the other image is a plurality of two-dimensional images to which parallax has already been given, the parallax control unit may determine the horizontal shift amount of the plurality of two-dimensional images according to the proper parallax. . In this aspect, the input for stereoscopic display is not generated with a high degree of freedom starting from the three-dimensional data, but is a generated parallax image, and the parallax is fixed. In this case, it is not possible to return to the original three-dimensional space or the real space actually photographed and change the camera position to redraw or re-photograph. Therefore, the parallax is adjusted by horizontally shifting the viewpoint images forming the parallax images or the pixels included in the viewpoint images.

In the case where the other image is a plane image to which depth information is given (hereinafter also referred to as “image with depth information”), the parallax control unit may adjust the depth according to the proper parallax. . An example of this processing is the third described below.
The two-dimensional image generation unit of the three-dimensional image processing device.

The stereoscopic image processing apparatus further includes a parallax holding section for recording the proper parallax, and the parallax control section has a predetermined timing, for example, when the apparatus is activated, or the stereoscopic image processing function of the apparatus or one of the functions. The proper parallax may be read when the unit is activated, and the value may be used as an initial value for processing. That is, "start" may have either a hardware meaning or a software meaning. According to this aspect, once the user determines the proper parallax, the automatic process for adjusting the stereoscopic effect is realized thereafter. This is a function called “initial setting of appropriate parallax”.

Another aspect of the present invention relates to a stereoscopic image processing method, which is a step of displaying a plurality of stereoscopic images with different parallax to a user, and an appropriate parallax for the user based on the user's response to the displayed stereoscopic image. And a step of identifying

Still another aspect of the present invention also relates to a stereoscopic image processing method, which comprises a step of acquiring a proper parallax depending on a user, and a step of processing an image before display so that the obtained proper parallax is realized. Including. Here, the “acquisition” may be a process of positively specifying or a process of reading from the parallax holding unit or the like.

If each of these steps is implemented as a function of a stereoscopic display library and the functions of this library can be called as a function from a plurality of programs, the programmer considers the hardware of the stereoscopic display device and executes the program. It is effective because there is no need to describe it.

The second group of the present invention is based on a technique of adjusting parallax based on a user's instruction. This technology
It can be used for “manual adjustment” of parallax by the user, and the user can appropriately change the stereoscopic effect of the image being displayed. However, this technique is not limited to manual adjustment, and can also be used when stereoscopically displaying a certain image, reading the appropriate parallax described above, and automatically adjusting the parallax of the image. The difference from the automatic adjustment of the first group is that the automatic adjustment of the second group acts on a two-dimensional parallax image or an image with depth information, and when changing the parallax retrospectively to three-dimensional data, Use one group of technologies. Hereinafter, the second group will be described.

An aspect of the present invention relates to a stereoscopic image processing apparatus, which includes an instruction acquisition unit for acquiring a user's instruction for a stereoscopic image displayed from a plurality of viewpoint images, and an inter-view image between the plurality of viewpoint images according to the acquired instruction. And a parallax control unit that changes the parallax amount. An example of this processing is shown in FIG. 45, which will be described later, and is a typical example of “manual adjustment”. It is convenient if the user's instruction is provided by a simple GUI such as button operation.

Another aspect of the present invention also relates to a stereoscopic image processing apparatus, wherein a parallax amount detecting section for detecting a first parallax amount generated when displaying a stereoscopic image from a plurality of viewpoint images, and a first parallax amount are And a parallax control unit that changes the parallax amount between the plurality of viewpoint images so as to fall within the range of the second parallax amount that is the allowable parallax amount of the user. This is a typical example of “automatic adjustment”, and the above-mentioned appropriate parallax can be used as the second parallax amount.
An example of this processing is shown in FIG. 46 described later.

The parallax amount detection unit detects the maximum value of the first parallax amount, and the parallax control unit determines the parallax amount between a plurality of viewpoint images so that the maximum value does not exceed the maximum value of the second parallax amount. You may change it. In order to avoid excessive stereoscopic effect due to too much parallax, the maximum value of the parallax amount, that is, the limit parallax is intended to be protected. The maximum value here may be considered as the maximum value on the near side.

The parallax amount detector detects the first parallax amount by calculating corresponding point matching between a plurality of viewpoint images, or the first parallax amount recorded in the header of one of the plurality of viewpoint images in advance. The amount of parallax may be detected. An example of these processes is shown in FIG. 47 described later.

The parallax control unit may change the parallax amount between the plurality of viewpoint images by shifting the composite position of the plurality of viewpoint images. This is common to FIGS. The shift of the combining position is a shift in the horizontal or vertical direction in units of pixels or the entire image. When the input is an image with depth information, the parallax control unit may adjust the depth information to change the parallax amount.

Another aspect of the present invention relates to a stereoscopic image processing method, which comprises a step of acquiring a user's instruction for a stereoscopic image displayed based on a plurality of viewpoint images, and, according to the instruction, a method of interposing the plurality of viewpoint images. And changing the amount of parallax.

Still another aspect of the present invention also relates to a stereoscopic image processing method, wherein the step of detecting a first parallax amount occurring when a stereoscopic image is displayed from a plurality of viewpoint images, and the first parallax amount being the user's Changing the amount of parallax between the plurality of viewpoint images so that the amount of parallax falls within the range of the second amount of parallax that is the allowable amount of parallax.

Each of these steps may be implemented as a function of the stereoscopic display library so that the functions of this library can be called as a function from a plurality of programs.

The third group of the present invention is based on the technique of correcting parallax based on the position in the image. This "automatic correction" acts to reduce the user's discomfort or rejection of stereoscopic display, and can be used in combination with the techniques of the first and second groups. Generally, in stereoscopic display, technical or physiological problems are pointed out, such that a plurality of viewpoint images are observed to be displaced as they are closer to the edge of the image, and a sense of discomfort is easily generated. In the third group, this problem is mitigated by reducing the parallax near the image edge or adjusting the parallax so that the object moves from the near side to the far side. The third group will be described below.

One embodiment of the present invention relates to a stereoscopic image processing apparatus, and a parallax control unit for correcting parallax between a plurality of viewpoint images for displaying a stereoscopic image, and a correction to be referred to by the parallax control unit in its processing. The correction map is described so that the parallax is corrected based on the position in the viewpoint image. In the correction map,
There are a parallax correction map and a sense of distance correction map.

The parallax control unit reduces the parallax in the peripheral portion of a plurality of viewpoint images, for example, or changes the parallax so that the object is sensed farther from the user. The parallax control unit may change the parallax by selectively processing any of the plurality of viewpoint images.

When the plurality of viewpoint images are generated from three-dimensional data, that is, when the viewpoint images can be generated by returning to the three-dimensional space, the parallax control unit controls the camera parameters when generating the plurality of viewpoint images. Then, the parallax may be changed. The camera parameters include the distance between the left and right cameras, the angle at which the object is viewed from the camera, or the optical axis intersection position.

Similarly, when a plurality of viewpoint images are generated from three-dimensional data, the parallax control unit distorts the three-dimensional space itself in, for example, the world coordinate system to change the parallax when generating the plurality of viewpoint images. May be. On the other hand, when a plurality of viewpoint images are generated from the image with depth information, the parallax control unit may change the parallax by operating the depth information.

Another aspect of the present invention relates to a stereoscopic image processing method, which comprises a step of acquiring a plurality of viewpoint images for displaying a stereoscopic image and a parallax between the acquired viewpoint images in the viewpoint images. Changing based on position. These steps may be implemented as the functions of the stereoscopic display library, and the functions of this library may be called as functions from a plurality of programs.

The fourth group of the present invention relates to a technique for providing the first to third groups and their related functions as a software library to reduce the burden on programmers and users and to promote the spread of stereoscopic image display applications. The fourth group will be described below.

An aspect of the present invention relates to a stereoscopic image processing method, which holds information related to stereoscopic image display in a memory,
The held information is shared among a plurality of different programs, and when any of these programs displays a stereoscopic image, the held information is referred to determine the state of the image to be output. An example of the state of the image is how much parallax is given to the parallax image and its degree.

The "held information" may include information on any one of the format of images input to the stereoscopic image display device, the display order of viewpoint images, and the amount of parallax between viewpoint images. Further, in addition to sharing the held information, a process unique to stereoscopic image display may be shared by a plurality of programs. An example of the “processing unique to stereoscopic image display” is processing for determining retained information. Another example is a process related to a graphical user interface for determining a proper parallax, a process for displaying a parallax adjustment screen that supports realization of a proper parallax state, a process for detecting and tracking a user's head position, and a stereoscopic display device. And a process of displaying an image for adjusting.

Another aspect of the present invention relates to a stereoscopic image processing apparatus, which includes a stereoscopic effect adjusting unit for providing a user with a graphical user interface for adjusting the stereoscopic effect of a stereoscopic display image, and a stereoscopic effect adjustment by the user. And a parallax control unit that generates a parallax image in a manner that protects the marginal parallax that is found as a result.

This device further converts the format of the parallax image generated by the parallax control unit according to the information detection unit that acquires information to be referred to in order to optimize the stereoscopic image display and the acquired information. The conversion unit may be included.

The parallax control unit may control the camera parameter based on the three-dimensional data to generate the parallax image while keeping the limit parallax, or control the depth of the image with depth information to generate the parallax image. Alternatively, the parallax image may be generated after determining the horizontal shift amount of the plurality of two-dimensional images having parallax.

The fifth group of the present invention relates to one application or business model using the above stereoscopic image processing technique or its related technique. A fourth group of software libraries is available. Less than,
Regarding the fifth group.

An aspect of the present invention relates to a stereoscopic image processing method, wherein the appropriate parallax for stereoscopically displaying a parallax image is once converted into an expression format that does not depend on the hardware of the display device, and the appropriate parallax according to this expression format is displayed differently. Distributed between devices.

Another aspect of the present invention also relates to a stereoscopic image processing method, which comprises the step of reading the proper parallax of the user acquired by the first display device into the second display device, and the step of reading the proper parallax by the second display device. The method includes adjusting the parallax between the parallax images according to the parallax, and outputting the adjusted parallax image from the second display device. For example, the first display device is a device that the user normally uses, and the second display device is a device that is provided at another location.

Another aspect of the present invention relates to a stereoscopic image processing apparatus, which includes a first display device, a second display device, and a server connected via a network, and the first display device corresponds to the device. When the proper parallax information of the user acquired by the above is transmitted to the server, the server receives the proper parallax information and records it in association with the user, and when the user requests output of image data on the second display device. The device reads out the appropriate parallax information of the user from the server, adjusts the parallax, and then outputs the parallax image.

It is to be noted that any combination of the above constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a recording medium, a computer program and the like are also effective as an aspect of the present invention.

[0054]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT FIG. 1 shows a user 10 and a screen 1.
2. Shows the positional relationship of the three-dimensionally displayed reproduction object 14. The eye distance of the user 10 is E, the distance between the user 10 and the screen 12 is D, and the width of the reproduction object 14 when displayed is W. Since the reproduction object 14 is stereoscopically displayed, the pixels that are sensed closer to the screen 12, that is, the pixels that are closer to the screen, and the screen 12 are displayed.
Pixels that are sensed further away, i.e., pixels that are farther away. Pixels without parallax are screen 1
It is perceived on the screen 12 as it looks exactly the same on both eyes from both eyes.

FIG. 2 shows a photographing system for generating the ideal display of FIG. Set the distance between the two cameras 22 and 24 to E
And the distance from them to the optical axis crossing position when the real object 20 is viewed (this is called the optical axis crossing distance)
Is D, and a parallax image can be obtained from the two cameras if the object 20 having the same width as the screen 12 and the actual width W is photographed. When this is displayed on the screen 12 of FIG. 1, the ideal state of FIG. 1 is realized.

FIGS. 3 and 4 show a state in which the positional relationship of FIG. 2 is multiplied by A (A <1) and B (B> 1). The ideal state of FIG. 1 is also realized with the parallax image obtained by these positional relationships. That is, the basics of an ideal stereoscopic display are
It begins with making W: D: E constant. This relationship is also the basis of how to add parallax.

FIG. 5 to FIG. 10 show the outline of processing until stereoscopic display based on the three-dimensional data of the object 20 in the embodiment. FIG. 5 shows a model coordinate system, that is, a coordinate space that each three-dimensional object 20 has. Coordinates when the object 20 is modeled are given in this space. Usually object 20
Bring the origin to the center of.

FIG. 6 shows the world coordinate system. The world space is a large space in which a scene is formed by arranging objects 20, floors, and walls. The modeling of FIG. 5 and the determination of the world coordinate system of FIG. 6 can be recognized as “construction of three-dimensional data”.

FIG. 7 shows the camera coordinate system. Camera 2 at any angle of view in any direction from any position in the world coordinate system
By setting 2, the conversion to the camera coordinate system is performed. The camera position, direction, and angle of view are camera parameters. In the case of stereoscopic display, parameters are set for two cameras, so the camera interval and optical axis crossing position are also set. Also, in order to make the middle point of the two cameras the origin,
The origin is also moved.

8 and 9 show a perspective coordinate system. First, FIG.
As described above, the space to be displayed is clipped by the front projection plane 30 and the rear projection plane 32. As will be described later, one feature of the embodiment is that the surface with the maximum disparity point in the near position is the front projection surface 30, and the surface with the maximum disparity point in the distance is the rear projection surface 3.
There is to be 2. After clipping, this view volume is converted into a rectangular parallelepiped as shown in FIG. 8 and 9
This process is also called a projection process.

FIG. 10 shows the screen coordinate system. In the case of stereoscopic display, images from each of a plurality of cameras are converted into coordinate systems having screens, and a plurality of two-dimensional images, that is, parallax images are generated.

FIG. 11, FIG. 12 and FIG. 13 show the configuration of a stereoscopic image processing apparatus 100, which is partially different. Less than,
For the sake of convenience, those stereoscopic image processing apparatuses 100 are also referred to as first, second, and third stereoscopic image processing apparatuses 100, respectively. These three-dimensional image processing apparatus 100 can be integrated into the apparatus, but here, the complexity of the figure is avoided and
It is divided into two. The first stereoscopic image processing apparatus 100 is effective when the object and space to be drawn can be obtained from the stage of three-dimensional data, and therefore the main input is three-dimensional data. The second stereoscopic image processing apparatus 100 is effective for parallax adjustment of a plurality of two-dimensional images to which parallax has already been given, that is, existing parallax images, and thus inputs two-dimensional parallax images. The third stereoscopic image processing apparatus 100 operates the depth information of the image with depth information to realize proper parallax, and therefore the input is mainly the image with depth information. These three types of inputs are collectively referred to as “original data”.

When the first to third stereoscopic image processing devices 100 are integrally mounted, an "image format determination unit" is provided as a preprocessing unit for determining the three-dimensional data, parallax image, and image with depth information. After that, the most suitable one of the first to third stereoscopic image processing apparatuses 100 may be activated.

The first stereoscopic image processing apparatus 100 has the functions of "initial setting" and "automatic adjustment" in setting the stereoscopic effect for stereoscopic display. When the user specifies the appropriate parallax range for the stereoscopically displayed image, this is acquired by the system, and thereafter, when another stereoscopic image is displayed,
A conversion process is performed in advance so that this proper parallax is realized and the result is displayed. Therefore, the first stereoscopic image processing apparatus 100
Thus, as a general rule, the user can enjoy the stereoscopic display suitable for himself after the setting procedure is performed only once.

The first stereoscopic image processing apparatus 100 further includes
It has a sub-function called "parallax correction" that artificially relaxes the parallax in the peripheral area of the image. As described above, the shifts of the plurality of viewpoint images are more likely to be recognized as “double images” as the image ends are approached. This is mainly due to mechanical errors such as the parallax barrier and the warp of the screen of the display device. Therefore, in the peripheral part of the image, 1) reduce both the near parallax and the far parallax, 2) reduce the near parallax and leave the far parallax unchanged, and 3) the entire parallax regardless of the near parallax and the far parallax. Various methods are implemented, such as shifting to the far parallax. Note that this “parallax correction” function also exists in the third stereoscopic image processing apparatus 100, but due to the difference in input data,
Processing is different.

The first stereoscopic image processing apparatus 100 adjusts the stereoscopic effect based on the user's response to the stereoscopically displayed image to adjust the stereoscopic effect, and the appropriate parallax specified by the stereoscopic effect adjusting unit 112. Parallax information holding unit 120 to be saved
Then, the proper parallax is read from the parallax information holding unit 120, the parallax control unit 114 that generates the parallax image having the proper parallax from the original data, the hardware information of the display device, and the stereoscopic display method are obtained. An information acquisition unit 118 having a function and a format conversion unit 116 for changing the format of the parallax image generated by the parallax control unit 114 based on the information acquired by the information acquisition unit 118 are included. Original data is simply called three-dimensional data, but strictly speaking, it corresponds to object and space data described in the world coordinate system.

As an example of the information acquired by the information acquisition unit 118, the number of viewpoints for stereoscopic display, the system of the stereoscopic display device such as space division or time division, whether or not the shutter glasses are used, and the case of the multiview system It includes the way of arranging the viewpoint images, whether or not the viewpoint images in which the parallax is reversed exists in the parallax image, and the result of head tracking. Note that only the result of head tracking is exceptionally directly input to the camera placement determination unit 132 via a route (not shown) and processed there.

The above configuration can be realized in terms of hardware by a CPU, memory, or other LSI of an arbitrary computer, and can be realized in terms of software by a program having a GUI function, a parallax control function, or other functions. However, here, the functional blocks realized by those collaborations are drawn. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and the same applies to the subsequent configurations.

The stereoscopic effect adjustment unit 112 has an instruction acquisition unit 122 and a parallax identification unit 124. The instruction acquisition unit 122 acquires the appropriate parallax range when the user specifies the range of the appropriate parallax with respect to the stereoscopically displayed image. The parallax identifying unit 124 identifies the appropriate parallax when the user uses this display device based on the range. The proper parallax is represented in a representation format that does not depend on the hardware of the display device. Realizing the proper parallax enables stereoscopic viewing that is suitable for the user's physiology.

The parallax control unit 114 firstly determines the camera parameters by temporarily setting the camera temporary placement unit 130 that temporarily sets the camera parameters, the camera placement determination unit 132 that corrects the camera parameters that are temporarily set according to the appropriate parallax, and the camera parameters. An origin moving unit 134 that performs an origin moving process to set the middle point of a plurality of cameras as an origin, a projection processing unit 138 that performs the above-described projection process, and a parallax image by performing a conversion process to a screen coordinate system after the projection process. 2D image generation unit 14 for generating
Including 2 and. In addition, spatial distortion conversion (hereinafter also simply referred to as distortion conversion) in order to reduce parallax in the peripheral area of the image as necessary.
A distortion processing unit 136 for performing the above is provided between the temporary camera placement unit 130 and the camera placement determination unit 132. Distortion processing unit 13
Reference numeral 6 reads out and uses a correction map described later from the correction map holding unit 140.

If it is necessary to adjust the display device for stereoscopic display, a GUI (not shown) for that purpose may be added. With this GUI, the entire displayed parallax image may be finely shifted vertically and horizontally to determine the optimum display position.

Second stereoscopic image processing apparatus 100 in FIG.
Takes a plurality of parallax images as input. This is also called an input image. The second three-dimensional image processing apparatus 100 reads the proper parallax previously acquired by the first three-dimensional image processing apparatus 100, adjusts the parallax of the input image to fit within the proper parallax range, and outputs the parallax. In that sense, the second stereoscopic image processing apparatus 1
00 has a function of “automatic adjustment” of parallax. However, in addition to that, when the user wants to change the stereoscopic effect while the stereoscopic display is actually performed, the GUI function is provided, and the “manual adjustment” function for changing the parallax according to the user's instruction is also provided.

The parallax of the parallax image that has already been generated cannot normally be changed, but the second stereoscopic image processing apparatus 10
According to 0, the stereoscopic effect can be changed at a level that can be sufficiently put to practical use by shifting the combining position of the viewpoint images forming the parallax image. The second stereoscopic image processing apparatus 100 exerts a good stereoscopic effect adjusting function even in a situation where the input data cannot be traced back to the three-dimensional data. Hereinafter, differences from the first stereoscopic image processing apparatus 100 will be mainly described.

The stereoscopic effect adjusting unit 112 is used for manual adjustment. The instruction acquisition unit 122 displays “+ n” on the screen, for example.
Numerical value input such as "-n" is realized, and the value is specified by the parallax specifying unit 124 as the parallax change amount. There are several possible relationships between the numerical values and the instructed three-dimensional effect. For example, “+ n” may be an instruction to strengthen the stereoscopic effect, and “−n” may be an instruction to weaken the stereoscopic effect. Further, “+ n” may be an instruction to move the object in the near direction to the whole, and “−n” may be an instruction to move the object in the far direction to the whole. Alternatively, the value of n may not be specified, and only the "+" and "-" buttons may be displayed, and the parallax may be changed each time the button is clicked.

The second stereoscopic image processing apparatus 100 has a parallax amount detector 150 and a parallax controller 152. When the input image is a plurality of parallax images, the parallax amount detection unit 150 inspects the header areas of those parallax images, and the parallax amount described in the form of the number of pixels, in particular, the maximum parallax pixel and the maximum parallax distance. If there is a pixel count, this is acquired. If the amount of parallax is not described, the matching unit 158 identifies the amount of parallax by detecting corresponding points between parallax images using a known method such as block matching. The matching unit 158 may perform processing only on an important area such as the central portion of the image, or may detect only the most important close-up maximum parallax pixel number.
The detected amount of parallax is sent to the parallax control unit 152 in the form of the number of pixels.

Position shift section 160 of parallax control section 152
Shifts the combined position of the viewpoint images forming the parallax images in the horizontal direction so that the parallax amount between the viewpoint images becomes an appropriate parallax. The shift may be performed on any of the viewpoint images. The position shift unit 160 also has another operation mode, and when the user gives an instruction to increase or decrease the parallax via the stereoscopic effect adjustment unit 112, the image shift position is simply changed according to this instruction. That is, the position shift unit 160 has two functions, an automatic adjustment function for proper parallax and a manual adjustment function by the user.

The parallax writing unit 164 uses the parallax amount as the number of pixels in the header area of any of a plurality of viewpoint images forming the parallax image for the above-described parallax amount detecting unit 150 or for another purpose. Write. The image edge adjustment unit 168 fills in the pixel loss that occurs at the image edge due to the shift by the position shift unit 160.

Third stereoscopic image processing apparatus 100 of FIG.
Takes an image with depth information as input. The third stereoscopic image processing apparatus 100 adjusts the depth so that the proper parallax is realized. It also has the above-mentioned “parallax correction” function. The distortion processing unit 174 of the parallax control unit 170 includes a correction map holding unit 176.
According to the correction map stored in, the distortion conversion is performed as described later. The depth information and the image after the distortion conversion are input to the two-dimensional image generation unit 178, where the parallax image is generated. The two-dimensional image generation unit 178 is different from the two-dimensional image generation unit 142 of the first stereoscopic image processing device 100, and the proper parallax is considered here. Since the image with depth information is also two-dimensional as an image, the two-dimensional image generation unit 178
Although not shown, it has a function similar to the position shift unit 160 of the second stereoscopic image processing apparatus 100, and shifts the pixels in the image in the horizontal direction according to the depth information to generate a stereoscopic effect. At this time, proper parallax is realized by the processing described below.

Each stereoscopic image processing apparatus 1 having the above configuration
The processing operation of each unit of 00 and its principle are as follows. FIGS. 14A and 14B show the left-eye image 20 displayed in the process of identifying the proper parallax by the stereoscopic effect adjusting unit 112 of the first stereoscopic image processing apparatus 100.
0, the right eye image 202 is shown. Five black circles are displayed in each image, and the closer to the top and the greater the parallax, the lower toward the far the greater parallax.

FIG. 15 schematically shows the sense of distance perceived by the user 10 when these five black circles are displayed. The user 10 responds that the range of these five senses of distance is “appropriate”, and the instruction acquisition unit 122 acquires this response. In the figure, five black circles with different parallaxes are displayed simultaneously or in sequence, and the user 10 inputs whether or not the parallax is acceptable. On the other hand, in FIG. 16, the display itself is performed by one black circle, but the parallax is continuously changed, and a response is given when the user 10 reaches the limit allowed in each of the far position and the near position. For the response, a known key operation such as normal key operation, mouse operation, or voice input may be used.

In both cases of FIG. 15 and FIG. 16, the instruction acquisition unit 122 can acquire the appropriate parallax as a range, and the limit parallax on the near side and the far side is determined. The maximum parallax is the parallax corresponding to the closeness allowed to the point that is closest to the user, and the maximum far parallax is the parallax corresponding to the distance allowed to the point that is the farthest from the user. However, in general, it is often necessary to care for the maximal parallax due to the physiological problem of the user, and hereinafter, only the maximal parallax may be referred to as the marginal parallax.

FIG. 17 shows the principle of actually adjusting the parallax between two viewpoints when a stereoscopically displayed image is extracted from three-dimensional data. First, the marginal parallax determined by the user is converted into the view angle of the temporarily arranged camera. The angle expression provides a general-purpose expression independent of the display device hardware. As shown in the figure, the limit parallax between the near position and the far position can be represented by M and N in terms of the number of pixels, and the angle of view θ of the camera corresponds to the number of horizontal pixels L of the display screen. The angles, the near maximum prospective angle φ and the farthest maximum prospective angle ψ, which are angles, are represented by θ, M, N, and L.

Tan (φ / 2) = Mtan (θ / 2) / L tan (ψ / 2) = Ntan (θ / 2) / L Next, this information is used to extract a two-viewpoint image in a three-dimensional space. Apply. As shown in FIG. 18, first, a basic expression space T (its depth is also expressed as T) is determined. Here, it is assumed that the basic expression space T is determined based on the restrictions on the arrangement of objects. Let S be the distance from the front projection plane 30, which is the front surface of the basic expression space T, to the camera arrangement plane, that is, the viewpoint plane 208. The user can specify T and S. There are two viewpoints, and the distance of the optical axis intersection plane 210 from the viewpoint plane 208 is D
And A is the distance between the optical axis intersecting surface 210 and the front projection surface 30.

Next, assuming that the parallaxes of the near position and the far position in the basic expression space T are P and Q, respectively, E: S = P: A E: S + T = Q: TA E is the distance between viewpoints. Now, the point G, which is a pixel with no parallax, is the optical axis K2 from both cameras.
Are intersecting with each other on the optical axis intersecting surface 210, and the optical axis intersecting surface 210 is the position of the screen surface. Maximum parallax P
The light ray K1 that produces the light beam intersects on the front projection plane 30, and the light ray K3 that produces the maximum distant parallax Q intersects on the rear projection plane 32.

P and Q are represented by P = 2 (S + A) tan (φ / 2) Q = 2 (S + A) tan (ψ / 2) using φ and ψ as shown in FIG. , E = 2 (S + A) tan (θ / 2) · (SM + SN + T
N) / (LT) A = STM / (SM + SN + TN) is obtained. Now that S and T are known, A and E are automatically determined in this way, and therefore the optical axis crossing distance D
And the inter-camera distance E is automatically determined, and the camera parameters are fixed. If the camera placement determination unit 132 determines the placement of the cameras according to these parameters, the projection processing unit 138 and the two-dimensional image generation unit 142 perform the processing independently for the images from the respective cameras, so that the proper processing is performed. A parallax image having parallax can be generated and output. As described above, E and A do not include hardware information, and a hardware-independent expression format is realized.

After that, if a camera is arranged so as to protect A or D and E even when another image is stereoscopically displayed, proper parallax can be automatically realized. The process from specifying the proper parallax to the ideal 3D display can be automated, so if this function is provided as a software library, programmers who create contents and applications do not need to be aware of programming for 3D display. .
Further, when L, M, and N are represented by the number of pixels, L indicates a display range, and therefore L can indicate whether the display is a full screen or a part of the screen. L is also a parameter that does not depend on hardware.

FIG. 20 shows four cameras 22, 24, 26,
28 shows a four-lens camera arrangement by 28. To be exact, the first
The above-mentioned A and E should be determined so that the parallax between adjacent cameras such as between the second camera 22 and the second camera 24 is close to the center.
Even if A and E determined between the camera 24 and the third camera 26 are used for other cameras, almost the same effect can be obtained.

Although T is a limitation on the arrangement of objects, it may be determined by a program as the size of the basic three-dimensional space. In this case, it is possible to arrange the object only in this basic expression space T throughout the program, or for the purpose of effective display, sometimes the object is given a parallax so as to intentionally jump out of this space.

As another example, T may be determined with respect to the coordinates of the object located closest and the object located farthest in the three-dimensional space. Objects can be placed in the expression space T. As an exception to always putting an object in the basic expression space T, a short-time exception can also be made by operating under the relaxation condition that "the average of positions for a certain period of time needs to be in the basic expression space T". Furthermore, the basic expression space T
The object that defines s may be limited to a static object, and in this case, a dynamic object can give an exceptional operation that extends out of the basic expression space T. As still another example, the space in which the objects are already arranged may be reduced to the size of the width T of the basic expression space, or may be combined with the above-described operation.

If the stereoscopic effect adjusting unit 112 of the first stereoscopic image processing apparatus 100 displays a double image as an image to be displayed to the user, the marginal parallax is set to a small value and another image is displayed. It is possible to reduce the frequency of appearance of double images when doing. It is known that an image in which the double image is likely to appear has contrasting colors and brightness between an object and a background, and such an image may be used at the stage of specifying the parallax, that is, at the time of initial setting. .

21 to 36 show processing by the distortion processing unit 136 of the first stereoscopic image processing apparatus 100 and its principle. FIG. 21 conceptually shows an example of the correction map stored in the correction map holding unit 140 of the first stereoscopic image processing apparatus 100. This map directly corrects parallax,
The whole corresponds to the parallax image as it is, and the parallax becomes smaller toward the periphery. FIG. 22 shows a change in parallax as a result of the camera arrangement determined by the distortion processing unit 136 according to this correction map and the camera arrangement determination unit 132 receiving the camera arrangement having received the operation. "Normal parallax" is attached when looking in the front direction from the left and right viewpoint positions of the two cameras, while "small parallax" is attached when looking in a direction largely deviated from the front. In reality, the camera arrangement determination unit 132 makes the camera interval closer as it goes to the periphery.

FIG. 23 shows another example in which the camera placement determining unit 132 changes the placement of cameras according to an instruction from the distortion processing unit 136 to change parallax. Here, of the two cameras, while moving only the left camera, the parallax changes as “normal parallax” → “medium parallax” → “small parallax” toward the periphery of the image. This method has a lower calculation cost than that of FIG.

FIG. 24 shows another example of the correction map. This map also changes the parallax, and the normal parallax is not touched near the center of the image, and the parallax is gradually reduced in other parallax correction areas. FIG. 25 shows the camera placement determining unit 13.
2 indicates the camera position which is changed according to this map. When the direction of the camera deviates significantly from the front, the position of the left camera shifts to the right camera for the first time, and "small parallax" is added.

FIG. 26 conceptually shows another example of the correction map. This map corrects the sense of distance from the viewpoint to the object, and in order to realize it, the camera placement determination unit 132 adjusts the optical axis intersection distance of the two cameras. If the optical axis crossing distance is made smaller toward the periphery of the image, the object appears to be relatively deep in the distant direction, so that the object is achieved especially in the sense of reducing the near parallax. In order to reduce the optical axis crossing distance, the camera placement determining unit 132 may change the optical axis direction of the camera, and may change the direction of either one of the cameras. 27 is the same as FIG.
6 shows changes in the optical axis crossing position or the optical axis crossing surface 210 when a two-dimensional image is generated by the map of FIG. The optical axis crossing plane 210 is closer to the camera in the vicinity of the image.

FIG. 28 shows another correction map relating to the sense of distance.
FIG. 29 shows how the camera placement determination unit 132 changes the optical axis intersection plane 210 according to the instruction from the distortion processing unit 136 according to the map of FIG. In this example, the object is arranged at the normal position without correction in the central area of the image, and the position of the object is corrected in the peripheral area of the image. For that purpose, in FIG. 29, the optical axis intersection plane 210 does not change near the center of the image, and the optical axis intersection plane 210 approaches the camera after passing a certain point. In FIG. 29, only the left camera is changed in direction.

FIGS. 30A to 30F show another distortion conversion by the distortion processing unit 136. Unlike the previous examples, instead of changing the camera position, the 3D space itself is distorted directly in the camera coordinate system. 30A to 30F, the rectangular area shows a top view of the original space, and the hatched area shows a top view of the space after conversion. For example, the point U in the original space of FIG. 30A moves to the point V after conversion. This means that this point has been moved in the distant direction. In FIG. 30 (a), the space is crushed toward the peripheral portion in the direction of the arrow in the depth direction, and in both the near position and the far position, the distance is close to a certain sense of distance, like the point W in the figure. It gives a feeling. As a result, the sense of distance is uniform in the peripheral area of the image, and no specially placed object is eliminated. This solves the problem of double images and provides an expression that is easily adapted to the user's physiology.

30 (b), 30 (c), 30
(D) and FIG. 30 (e) each show a modified example of conversion that brings the sense of distance close to a constant value in the image peripheral portion, and FIG. 30 (f) shows an example of conversion of all points in the distant direction. There is.

FIG. 31 shows the principle for realizing the conversion of FIG. 30 (a). The rectangular parallelepiped space 228 includes a space in which the projection processing of the first camera 22 and the second camera 24 is performed. The view volume of the first camera 22 is determined by the angle of view of the camera and the front projection plane 230 and the rear projection plane 232, and that of the second camera 24 is the view angle of the camera and the front projection plane 234 and the rear projection plane 232. Determined by 236.
The distortion processing unit 136 applies distortion conversion to the rectangular parallelepiped space 228. The origin is the center of the rectangular parallelepiped space 228. In the case of the multi-view type, the conversion principle is the same except that the number of cameras is increased.

FIG. 32 shows an example of distortion conversion, which employs reduction conversion in the Z direction. Actually, the processing is performed for each object in the space. FIG. 33 represents this conversion by comparing it to a parallax correction map. The normal parallax is on the Y axis, and the parallax becomes smaller as the absolute value of X increases, and X =
± A means no parallax. Here, the conversion formula is as follows, since the conversion is the reduction conversion only in the Z direction.

[0100]

[Equation 1] The conversion will be described with reference to FIG. First, consider the range of X ≧ 0 and Z ≧ 0. When the point (X0, Y0, Z0) is moved to the point (X0, Y0, Z1) by the reduction processing, the reduction ratio Sz
Is Sz = Z1 / Z0 = CE / CD. The coordinates of C are (X0, Y0, 0) and the coordinates of D are (X0, Y0, B).

E is the intersection of a straight line and a plane, and its coordinates are (X
0, Y0, Z2), Z2 can be obtained as below.

Z = B−X × B / A (plane) X = X0, Y = Y0 (straight line) Z2 = B−X0 × B / A Therefore, Sz = CE / CD = (B−X0 × B / A) ) / B = 1-X0 / A For X, generally, Sz = 1-X / A. When the same calculation is performed for other ranges of X and Z, the following results are obtained and the conversion can be verified.

When X ≧ 0, Sz = 1−X / A When X <0, Sz = 1 + X / A FIG. 35 shows another example of distortion conversion. Strictly speaking, in consideration of the radial photographing from the camera, the reduction processing in the X-axis and Y-axis directions is also combined. Here, the conversion is performed with the center of the two cameras as a representative of the camera positions. The conversion formula is as follows.

[0104]

[Equation 2] FIG. 36 verifies this conversion. Again, X ≧ 0 and Z
Consider the range of ≧ 0. When the point (X0, Y0, Z0) is moved to the point (X1, Y1, Z1) by the reduction process, the reduction ratios Sx, Sy, Sz are Sx = (X1-X2) / (X0-X2) = (X4 -X2) / (X3-X2) Sy = (Y1-Y2) / (Y0-Y2) = (Y4-Y2) / (Y3-Y2) Sz = (Z1-Z2) / (Z0-Z2) = (Z4 -Z2) / (Z3-Z2). Since E is the intersection of a plane and a straight line, S is the same as above.
x, Sy, and Sz can be calculated.

When the space after conversion is represented by a set of planes as described above, the processing changes at the boundary between the tangents of the planes, which may cause discomfort. In that case, the surfaces may be connected by curved surfaces, or the space may be formed only by curved surfaces. The calculation is only changed to obtain the intersection E of the curved surface and the straight line.

In the above example, the reduction ratio is the same straight line C.
Although it is the same on D, weighting may be performed. For example, a weighting function G (L) for the distance L from the camera may be applied to Sx, Sy, and Sz.

37 to 40 show processing by the distortion processing unit 174 of the third stereoscopic image processing apparatus 100 and its principle. FIG. 37 shows a depth map of the image with depth information input to the third stereoscopic image processing apparatus 100, and here it is assumed that the depth range has values K1 to K2. Here, the depth in the near position is expressed as positive, and the depth in the far position is expressed as negative.

FIG. 38 shows the relationship between the original depth range 240 and the converted depth range 242. The depth approaches a certain value as it goes to the periphery of the image. The distortion processing unit 174 converts the depth map according to this correction. The same applies to the case of providing parallax in the vertical direction. Since this conversion is also only reduction in the Z direction, it can be expressed by the following formula.

[0109]

[Equation 3] Note that Sz is divided into cases according to the value of X, and when X ≧ 0, Sz = 1−2X / L, and when X <0, Sz = 1 + 2X / L. By the above conversion, a new depth map having new elements shown in FIG. 39 is generated.

FIG. 40 shows another principle of distortion conversion for the depth map. More precisely, the space is observed radially by the user 10, so the reduction processing in the X-axis and Y-axis directions is also combined. Here, the center between the eyes is the observation position. The specific processing is the same as in the case of FIG. Although the original depth map has only Z values, X values and Y values are also held when this calculation is performed.
The Z value is converted into the pixel shift amount in the X direction or the Y direction, and the X value and the Y value may be held as offset values for them.

In any case, the depth map and the original image converted by the distortion processing unit 174 are the two-dimensional image generation unit 17
8 is input, and the synthesizing process is performed here by shifting in the horizontal direction so as to obtain an appropriate parallax. The details will be described later.

41 to 51 show the processing of the position shift section 160 of the second stereoscopic image processing apparatus 100 and the two-dimensional image generation section 178 of the third stereoscopic image processing apparatus 100 that can be grasped as an extension thereof. FIG. 41 shows the principle of shifting the combined position of two parallax images by the position shift unit 160.
As shown in the figure, the positions of the right-eye image R and the left-eye image L match in the initial state. However, when the left-eye image L is relatively shifted to the right as in the upper part of the figure, the parallax at the near position increases and the parallax at the far position decreases. On the contrary, when the left-eye image L is relatively shifted to the left as shown in the lower part of the figure, the parallax at the near point decreases and the parallax at the far point increases.

The essence of parallax adjustment by shifting the parallax image has been described above. The image may be shifted in one direction, or both may be shifted in the opposite directions. Further, from this principle, it is understood that the stereoscopic display method can be applied to all methods that use parallax, regardless of the glasses method or the method without glasses. Similar processing can be performed on multi-view images and vertical parallax.

FIG. 42 shows the shift processing at the pixel level.
A first quadrangle 250 and a second quadrangle 252 are both shown in the left-eye image 200 and the right-eye image 202. The first quadrangle 250 has a near parallax, and when the parallax amount is represented by a positive number, it is “6 pixels”. On the other hand, the second quadrangle 252 has a distant parallax, and when the parallax amount is represented by a negative number, it is “−6 pixels”. Here, the parallax amounts are F2 and F1, respectively.

On the other hand, it is assumed that the proper parallax of the display device owned by the user is found to be J1 to J2. The position shift unit 160 sets the combining start positions of both images to each other (J2-F
2) Pixel shift. FIG. 43 shows the state after the end of the shift, where F1 = -6 and F2 = 6, and J
If 1 = −5 and J2 = 4, the combining start positions are shifted by −2 pixels from each other, that is, in the direction in which the whole is shifted in the distance direction. As shown in FIG. 43, the final parallax amounts are E1 = -8 and E2 = 4, and are within the parallax at least in the near direction. It is generally said that the double image in the near direction is more uncomfortable than the far direction,
In addition, since the subject is often photographed in a state of being arranged in the close-up direction, it is basically desirable to keep the parallax in the close-up direction within the limit. A processing example is shown below.

1. When the near position is outside the limit parallax and the far position is within the limit parallax, the near position is shifted to the limit parallax point. However, if the parallax of the distant point reaches the interocular distance, the processing is stopped. 2. When the near position is outside the limit parallax and the far position is outside the limit parallax, the near position is shifted to the limit parallax point. However, if the parallax of the distant point reaches the interocular distance, the processing is stopped. 3. If both the near point and the far point are within the limit parallax, they are not processed. 4. When the near point is within the limit parallax and the far point is outside the limit parallax, the far point is shifted to the limit parallax point, but if the near point reaches the limit parallax point during the processing, the processing is stopped. .

FIG. 44 shows the loss of the image edge due to the shift of the combining position. Here, the left eye image 200 and the right eye image 202
Has a shift amount of 1 pixel, and a missing portion 26 having a width of 1 pixel at each of the right end of the left-eye image 200 and the left end of the right-eye image 202
0 occurs. At this time, the image edge adjusting unit 168 duplicates the pixel row at the image edge as shown in FIG. 44 to compensate for the number of horizontal pixels.

As another method, the missing portion 260 may be displayed in a specific color such as black or white, or may be hidden. Further, clipping or adding processing may be performed so that the size becomes the same as the size of the initial image. Further, the size of the initial image may be made larger than the actual display size in advance so that the missing portion 260 does not affect the display.

FIG. 45 shows the flow of manual adjustment of parallax by the second stereoscopic image processing apparatus 100. As shown in the figure, first, the left and right images are manually created as parallax images (S10),
This is distributed through the network and other routes (S1
2). The second three-dimensional image processing apparatus 100 receives this (S14), and in the example of this figure, first, the images are combined and displayed in the normal state without shift (S16).
That is, here, it is considered that the proper parallax has not yet been acquired or the position shift unit 160 has not been operated. Subsequently, the user instructs parallax adjustment on the stereoscopically displayed parallax image via the stereoscopic effect adjusting unit 112, and the position shift unit 160 receives the instruction in the “manual adjustment mode” to adjust the image combining position. Is displayed (S18). The steps S10 and S12 are steps 270 and S of the image creator.
From 14 onward, the procedure 272 of the second stereoscopic image processing apparatus 100.
Is. Further, although not shown, if this shift amount is recorded in the header and is referred to from the next time and synthesized, the labor of readjustment can be saved.

FIG. 46 shows the flow of automatic adjustment by the second stereoscopic image processing apparatus 100. Image Creator Procedure 27
Left and right image generation (S30) and image distribution (S3)
2) is the same as FIG. 45. Further, in the procedure 272 of the second stereoscopic image processing apparatus 100, the image reception (S34) is also the same. Next, the matching unit 158 of the parallax amount detection unit 150 detects the parallax attached in advance between parallax images, particularly the maximum parallax (S36), while obtaining the proper parallax, particularly the parallax from the parallax information holding unit 120. Yes (S38). After that, the position shift unit 160 shifts the composite position of the images so as to satisfy the limit parallax by the above-described processing (S40), and the parallax writing unit 164 and the image edge adjusting unit 16 are performed.
8. The stereoscopic display is performed through the processing by the format conversion unit 116 (S42).

FIG. 47 shows a second stereoscopic image processing apparatus 100.
3 shows another flow of automatic adjustment by. After the left and right images are generated (S50) in the procedure 270 of the image creator, the maximum parallax is detected at this point (S52) and recorded in the header of any viewpoint image of the parallax images (S54). This detection may be performed by corresponding point matching, but when the creator manually generates the parallax image, it is naturally known in the editing process, and therefore it may be recorded. Then, the image is distributed (S56).

On the other hand, in the procedure 272 of the second stereoscopic image processing apparatus 100, image reception (S58) is the same as that in FIG. Next, the header inspection unit 15 of the parallax amount detection unit 150.
The maximum parallax described above is read from the header according to S6 (S6).
0). On the other hand, the limit parallax is acquired from the parallax information holding unit 120 (S62), and the following processes S64 and S66 are the same as the processes S40 and S42 of FIG. 46, respectively. According to this method, it is not necessary to calculate the maximum parallax. Further, it is possible to realize an appropriate stereoscopic effect on the entire image. Furthermore, since the shift amount can be recorded in the header, there is no risk of damaging the original image itself. Although not shown, if the maximum parallax detected in FIG. 46 is also recorded in the header, the process can be performed in accordance with the procedure of FIG. 47.

The same processing can be performed in the multi-view system, and the same processing may be performed for the parallax amount between adjacent viewpoint images. However, in actuality, the maximum parallax among the parallaxes between the plurality of viewpoint images may be regarded as the “maximum parallax” between all viewpoint images, and the shift amount of the combined position may be determined.

The header information is required to be present in at least one of the multi-view images, but if the multi-view images are combined into one image, the header of that image may be used.

Further, there is a case where an image which has already been combined is distributed, but in that case, the image is separated by the inverse transform processing once and the combined position shift amount is calculated and recombined, or the same result is obtained. The pixels may be rearranged so that

48 to 51 show a process of shifting the composite position for an image with depth information. This is performed by the two-dimensional image generation unit 178 of the third stereoscopic image processing device 100. FIG. 48 and FIG. 49 are a plane image 204 and a depth map that form an image with depth information, respectively. Here, the near depth is expressed as positive and the far depth is expressed as negative. First quadrilateral 250 and second quadrilateral 25 as objects
2 and a third quadrangle 254 exist, the first quadrangle 250 is depth “4”, the second quadrangle 252 is “2”, the third quadrangle 2
54 is "-4". The first square 250 is the nearest point,
The second quadrangle 252 is at the intermediate proximity point, and the third quadrangle 254 is at the farthest point.

The two-dimensional image generation unit 178 first performs a process of shifting each pixel by the value of the depth map based on the original plane image 204, and generates the other viewpoint image. If the reference is the left-eye image, the original planar image 20
4 becomes the left-eye image as it is. The first quadrangle 250 has 4 pixels on the left, the second quadrangle 252 has 2 pixels on the left, and the third quadrangle 2
54 is shifted to the right by 4 pixels, and the right-eye image 202 is created as shown in FIG. The image edge adjusting unit 168 fills the missing portion 260 of the pixel information due to the movement of the object with the adjacent pixels which have the parallax of “0” and which are determined to be the background.

Subsequently, the two-dimensional image generator 178 calculates the depth satisfying the proper parallax. Depth range K1
When K2 is set and the depth value of each pixel is set to Gxy, the depth map has a form in which Hxy is changed to Gxy in FIG. In addition, the proper parallax of the display device owned by the user is J
Suppose it is found to be 1 to J2. In this case, in the depth map, the depth value G of each pixel is converted as follows, and a new depth value Fxy is obtained.

Fxy = J1 + (Gxy-K1) × (J2
-J1) / (K2-K1) In the above example, K1 = -4, K2 = 4, and J
Assuming that 1 = −3 and J2 = 2, the depth map of FIG. 49 is converted to the depth map of FIG. 51 by this conversion formula. That is, "4" is converted into "2", "2" is converted into "1", and "-4" is converted into "-3".
The intermediate value between K1 and K2 is transformed between J1 and J2. For example, the second quadrangle 252 has Gxy = 2 and Fxy =
It becomes 0.75. If Fxy is not an integer, it may be rounded off or processed to reduce the parallax.

Although the above-mentioned conversion formula is an example of linear conversion, a weighting function F (Gxy) for Gxy may be further applied and various other non-linear conversions may be considered.
It is also possible to shift the objects in the opposite directions from the original plane image 204 to newly generate left and right images. In the case of the multi-view system, the same process may be performed multiple times to generate a multi-view image.

The above is the configuration and operation of the stereoscopic image processing apparatus 100 according to the embodiment. Stereoscopic image processing apparatus 10
Although 0 is described as a device, this may be a combination of hardware and software, or can be configured by software only. In that case, it is convenient if an arbitrary part of the stereoscopic image processing apparatus 100 is made into a library so that it can be called from various programs. The programmer can skip programming that requires knowledge of stereoscopic display. For users, regardless of software or content,
Operations related to stereoscopic display, that is, GUI becomes common,
Since the set information can be shared with other software, the trouble of re-setting can be saved.

Note that it is also useful to share information among a plurality of programs rather than the processing relating to stereoscopic display. Various programs can determine the state of the image by referring to the information. An example of the shared information is the information acquired by the information acquisition unit 118 of the stereoscopic image processing apparatus 100 described above. This information may be held in a recording unit (not shown), the correction map holding unit 140, or the like.

52 to 54 show an example in which the stereoscopic image processing apparatus 100 described above is used as a library. Figure 5
Reference numeral 2 indicates the usage of the stereoscopic display library 300. The stereoscopic display library 300 is referred to by calling a function from a plurality of programs A302, programs B304, programs C306, and the like. In addition to the above-mentioned information, the parameter file 318 stores the proper parallax of the user. The stereoscopic display library 300 is used by a plurality of devices A312, B314, C316, etc. via an API (application program interface) 310.

As an example of the program A302 and the like, a game, a so-called Web3D three-dimensional application, a three-dimensional desktop screen, a three-dimensional map, a viewer of a parallax image which is a two-dimensional image, a viewer with depth information, and the like can be considered. Of course, some games use different coordinates, but the stereoscopic display library 300
Can handle that.

On the other hand, an example of the device A312 or the like is an arbitrary stereoscopic display device utilizing parallax, such as a twin-lens or multi-lens parallax barrier system, a shutter glasses system, and a polarization glasses system.

FIG. 53 shows an example in which the stereoscopic display library 300 is incorporated in the three-dimensional data software 402. The three-dimensional data software 402 is a program body 404, and a stereoscopic display library 300 that realizes proper parallax therefor,
A shooting instruction processing unit 406 is provided. Program body 404
Contacts the user via the user interface 410. The shooting instruction processing unit 406 virtually shoots a predetermined scene during operation of the program body 404 according to a user instruction. The captured image is the image recording device 4
12 is recorded. Further, it is output to the stereoscopic display device 408.

For example, assume that the three-dimensional data software 402 is game software. In that case, the user can play the game while experiencing an appropriate stereoscopic effect by the stereoscopic display library 300 during the game. During the game, if the user wants to keep a record, for example, when he or she completes a victory in a battle-type battle game, an instruction is given to the shooting instruction processing unit 406 via the user interface 410 and the scene is recorded. At this time, the stereoscopic display library 300 is used to generate a parallax image so as to have an appropriate parallax when reproduced on the stereoscopic display device 408 later, and this is recorded in an electronic album or the like of the image recording device 412. Note that the program body 40 can be recorded by recording a two-dimensional image called a parallax image.
The three-dimensional data itself possessed by 4 does not leak, and it is possible to consider copyright protection.

FIG. 54 shows the three-dimensional data software 4 of FIG.
An example in which 02 is incorporated into a network utilization type system 430 is shown. The game machine 432 is connected to the server 436 and the user terminal 434 via a network (not shown). The game machine 432 is for so-called arcade games, and includes a communication unit 442, three-dimensional data software 402, and a stereoscopic display device 440 that locally displays the game. The three-dimensional data software 402 is shown in FIG.
The parallax image displayed on the stereoscopic display device 440 from the three-dimensional data software 402 is optimally set in advance for the stereoscopic display device 440. The adjustment of parallax by the three-dimensional data software 402 is used when transmitting an image to the user via the communication unit 442 as described later. The display device used here only needs to have a function of adjusting the parallax to generate a stereoscopic image, and is not necessarily a device capable of stereoscopic display.

The user terminal 434 includes a communication unit 454, a viewer program 452 for viewing a stereoscopic image, and a stereoscopic display device 450 of any size and type for locally displaying the stereoscopic image. The stereo image processing apparatus 100 is installed in the viewer program 452.

The server 436 stores information such as the communication unit 460, the image holding unit 462 for recording images virtually taken by the user in relation to the game, the proper parallax information of the user, the user's mail address and other personal information. And a user information holding unit 464 for recording in association with The server 436 functions as, for example, an official website of the game, and records a scene that the user liked during the game execution, a moving image or a still image of a famous game. Stereoscopic display is possible with both moving images and still images.

An example of image capturing with the above configuration is performed as follows. The user performs stereoscopic display on the stereoscopic display device 450 of the user terminal 434 in advance, acquires the proper parallax based on the function of the stereoscopic image processing device 100, and notifies the server 436 via the communication unit 454 of the user information. The data is stored in the holding unit 464. This proper parallax is a general-purpose description irrelevant to the hardware of the stereoscopic display device 450 owned by the user.

The user can use the game machine 43 at any timing.
Play a game with 2. Meanwhile, the stereoscopic display device 440
The stereoscopic display is performed using the parallax initially set or the parallax manually adjusted by the user. When the user desires to record an image during the game play or replay, the stereoscopic display library 300 built in the three-dimensional data software 402 of the game machine 432 causes the two communication units 4 to operate.
42 and 460, the proper parallax of the user is acquired from the user information holding unit 464 of the server 436, a parallax image is generated in accordance with the proper parallax, and the two communication units 442 and 4 are again provided.
The parallax image regarding the virtually captured image is stored in the image holding unit 462 via 60. After returning to home, the user can download this parallax image to the user terminal 434 to perform stereoscopic display with a desired stereoscopic effect. In that case,
Stereo image processing apparatus 100 included in viewer program 452
This allows manual adjustment of parallax.

As described above, according to this application example, the programming relating to the stereoscopic effect, which should originally be set for each hardware of the display device and for each user, is concentrated in the stereoscopic image processing device 100 and the stereoscopic display library 300. The game software programmer does not have to worry about the complicated requirements for stereoscopic display. This applies not only to game software but also to any software that uses stereoscopic display, and eliminates restrictions on the development of contents and applications that use stereoscopic display. Therefore, the spread of these can be dramatically promoted.

In particular, in the case of games and other applications in which three-dimensional CG data originally exists, it has been difficult to code an accurate three-dimensional display in the past, which is a major cause. , I often did not use it for stereoscopic display.
The stereoscopic image processing apparatus 100 or the stereoscopic display library 300 according to the embodiment can eliminate such an adverse effect and contribute to the enhancement of the stereoscopic display application.

Although the user's proper parallax is registered in the server 436 in FIG. 54, the user records the information I
You may use the game machine 432 by bringing a C card or the like. You may record scores and favorite images related to this game on this card.

The present invention has been described above based on the embodiments. It should be understood by those skilled in the art that this embodiment is merely an example, and that various modifications can be made to the combinations of the respective constituent elements and the respective processing processes, and that such modifications are also within the scope of the present invention. is there. Below, such an example is given.

The first stereoscopic image processing apparatus 100 can process with high accuracy by inputting three-dimensional data. However, once the three-dimensional data is dropped into the image with depth information, the third stereoscopic image processing device 10
Parallax images may be generated using 0. In some cases
The calculation cost may be lower. Similarly, when inputting multiple viewpoint images, it is also possible to create a depth map by using high-precision corresponding point matching. A parallax image may be generated using the image processing device 100.

In the first stereoscopic image processing apparatus 100,
Although the temporary camera placement unit 130 has the configuration of the stereoscopic image processing apparatus 100, this may be pre-processing of the stereoscopic image processing apparatus 100. This is because the temporary placement of the cameras can be processed regardless of the proper parallax. Similarly, any processing unit constituting the first, second, and third stereoscopic image processing apparatus 100 can be output to the outside of the stereoscopic image processing apparatus 100, and the degree of freedom of the configuration of the stereoscopic image processing apparatus 100 can be increased. The height of will be understood by those skilled in the art.

In the embodiment, the case where the parallax control is performed in the horizontal direction has been described, but the same processing can be performed in the vertical direction.

A unit for enlarging character data may be provided during operation of the stereoscopic display library 300 and the stereoscopic image processing apparatus 100. For example, in the case of a parallax image from two horizontal viewpoints, the horizontal resolution of the image visible to the user is 1 /
It becomes 2. As a result, the readability of the characters may decrease,
It is effective to double the characters in the horizontal direction.
If there is parallax in the vertical direction as well, it is also useful to stretch the characters in the vertical direction.

When the stereoscopic display library 300 or the stereoscopic image processing apparatus 100 is in operation, an “in-operation display section” may be provided to put characters or marks such as “3D” on the displayed image. In that case, the user can know whether the image has parallax adjustable.

A stereoscopic display / normal display switching unit may be provided. This unit includes a GUI, and when the user clicks a predetermined button, the display is switched from a stereoscopic display to a normal two-dimensional display, and vice versa.

The information acquisition unit 118 does not necessarily acquire information by user input, but may have information that can be automatically acquired by a function such as plug and play.

In the embodiment, the method of deriving E and A is adopted, but a method of fixing these and deriving other parameters may be used, and the variables can be freely specified.

[0155]

The present invention has the following effects. 1. It is possible to generate or display a stereoscopic image that is suitable for human physiology. 2. Even if the display target image changes, a stereoscopic image suitable for the user can be generated or displayed. 3. You can adjust the stereoscopic effect of stereoscopic display with a simple operation. 4. The burden on the programmer can be reduced when creating content or applications that can display appropriately in three dimensions. 5. The user's labor to optimize the stereoscopic display is reduced. 6. Normally, you can easily realize stereoscopic adjustment and head tracking information that are not subject to the plug and play function.
The same applies to a device that cannot be plug and play in principle, such as a post-parallax barrier.

[Brief description of drawings]

FIG. 1 is a diagram showing a positional relationship among a user, a screen, and a reproduction object 14 who are able to achieve an ideal stereoscopic view.

FIG. 2 is a diagram showing an example of an imaging system that realizes the state of FIG.

FIG. 3 is a diagram showing another example of an imaging system that realizes the state of FIG.

FIG. 4 is a diagram showing another example of an imaging system that realizes the state of FIG.

FIG. 5 is a diagram showing a model coordinate system used in the first stereoscopic image processing apparatus.

FIG. 6 is a diagram showing a world coordinate system used in the first stereoscopic image processing apparatus.

FIG. 7 is a diagram showing a camera coordinate system used in the first stereoscopic image processing apparatus.

FIG. 8 is a diagram showing a view volume used in the first stereoscopic image processing apparatus.

9 is a diagram showing a coordinate system after perspective transformation of the volume of FIG.

FIG. 10 is a diagram showing a screen coordinate system used in the first stereoscopic image processing apparatus.

FIG. 11 is a configuration diagram of a first stereoscopic image processing apparatus.

FIG. 12 is a configuration diagram of a second stereoscopic image processing apparatus.

FIG. 13 is a configuration diagram of a third stereoscopic image processing apparatus.

14 (a) and 14 (b) are diagrams showing a left-eye image and a right-eye image displayed by a stereoscopic effect adjusting unit of the first stereoscopic image processing apparatus, respectively.

FIG. 15 is a diagram showing a plurality of objects having different parallaxes displayed by the stereoscopic effect adjusting unit of the first stereoscopic image processing apparatus.

[Fig. 16] Fig. 16 is a diagram illustrating an object in which parallax changes, which is displayed by the stereoscopic effect adjusting unit of the first stereoscopic image processing device.

[Fig. 17] Fig. 17 is a diagram illustrating a relationship among a camera view angle, an image size, and parallax when proper parallax is realized.

FIG. 18 is a diagram showing the positional relationship of the imaging system that realizes the state of FIG.

FIG. 19 is a diagram showing the positional relationship of the imaging system that realizes the state of FIG.

[Fig. 20] Fig. 20 is a diagram illustrating a camera arrangement when a multi-view image is generated with proper parallax.

FIG. 21 is a diagram illustrating a parallax correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.

22 is a diagram illustrating a camera viewpoint when generating a parallax image according to the parallax correction map of FIG. 21. FIG.

23 is a diagram showing another camera viewpoint when generating a parallax image according to the parallax correction map of FIG. 21. FIG.

[Fig. 24] Fig. 24 is a diagram illustrating a parallax correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.

25 is a diagram illustrating a camera viewpoint when generating a parallax image according to the parallax correction map in FIG. 24.

FIG. 26 is a diagram showing a sense of distance correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.

27 is a diagram showing a camera viewpoint when generating a parallax image according to the distance correction map of FIG. 26. FIG.

FIG. 28 is a diagram showing another distance feeling correction map used by the distortion processing unit of the first stereoscopic image processing apparatus.

29 is a diagram illustrating a camera viewpoint when generating a parallax image according to the distance perception correction map of FIG. 28.

FIG. 30 (a), FIG. 30 (b), FIG.
(C), FIG. 30 (d), FIG. 30 (e), FIG. 30 (f)
3A is a top view of a parallax distribution obtained as a result of the distortion processing unit of the first stereoscopic image processing apparatus performing processing on a three-dimensional space.

FIG. 31 is a diagram illustrating a principle of processing by a distortion processing unit of the first stereoscopic image processing apparatus.

FIG. 32 is a diagram specifically showing the process of FIG. 31.

FIG. 33 is a diagram specifically showing the process of FIG. 31.

FIG. 34 is a diagram specifically showing the process of FIG. 31.

[Fig. 35] Fig. 35 is a diagram illustrating another example of processing by the distortion processing unit of the first stereoscopic image processing apparatus.

FIG. 36 is a diagram specifically showing the process of FIG. 35.

FIG. 37 is a diagram showing a depth map.

[Fig. 38] Fig. 38 is a diagram illustrating an example of processing performed by a distortion processing unit of the third stereoscopic image processing apparatus.

[Fig. 39] Fig. 39 is a diagram illustrating a depth map generated by the processing by the distortion processing unit of the third stereoscopic image processing apparatus.

[Fig. 40] Fig. 40 is a diagram illustrating another example of processing by the distortion processing unit of the third stereoscopic image processing apparatus.

FIG. 41 is a diagram illustrating an example of processing performed by a two-dimensional image generation unit of the second stereoscopic image processing device.

[Fig. 42] Fig. 42 is a diagram illustrating an example of a parallax image.

[Fig. 43] Fig. 43 is a diagram illustrating a parallax image of which the synthesis position has been shifted by the two-dimensional image generation unit of the second stereoscopic image processing device.

[Fig. 44] Fig. 44 is a diagram illustrating processing of the image edge adjustment unit of the second stereoscopic image processing apparatus.

[Fig. 45] Fig. 45 is a diagram illustrating processing of the second stereoscopic image processing apparatus.

FIG. 46 is a diagram illustrating another process of the second stereoscopic image processing device.

FIG. 47 is a diagram showing another process of the second stereoscopic image processing device.

[Fig. 48] Fig. 48 is a diagram illustrating a planar image to which a depth map is added.

FIG. 49 is a diagram showing a depth map.

[Fig. 50] Fig. 50 is a diagram illustrating a manner in which a two-dimensional image generation unit of the second stereoscopic image processing apparatus generates a parallax image based on a depth map.

[Fig. 51] Fig. 51 is a diagram illustrating a depth map corrected by a two-dimensional image generation unit of the second stereoscopic image processing device.

FIG. 52 is a diagram showing how a stereoscopic image processing apparatus according to an embodiment is used as a library.

[Fig. 53] Fig. 53 is a configuration diagram in which a stereoscopic display library is incorporated in three-dimensional data software.

FIG. 54 is a diagram showing how a stereoscopic display library is used in a network-based system.

[Explanation of Codes] 10 User, 12 Screen, 14 Playback Object, 20 Real Object, 22, 2
4, 26, 28 camera, 30 front projection plane, 32
Rear projection plane, 100 stereoscopic image processing device, 112
Stereoscopic effect adjustment unit, 114, 152, 170 parallax control unit,
116 format conversion unit, 118 information acquisition unit, 122 instruction acquisition unit, 124 parallax identification unit,
132 camera placement determination unit, 136, 174 distortion processing unit, 140, 176 correction map holding unit, 142
Two-dimensional image generation unit, 150 Parallax amount detection unit, 156
Header inspection unit, 158 matching unit, 160 position shift unit, 164 parallax writing unit, 168 image edge adjustment unit, 178 two-dimensional image generation unit, 210 optical axis crossing plane, 300 stereoscopic display library, 402 three-dimensional data software, 406 Shooting instruction processing unit, 430
Network-based system, 432 game consoles,
434 user terminal, 436 server, 452
Viewer program.

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B050 BA08 BA09 BA11 CA07 EA13                       EA19 EA24 FA02 FA06                 5C061 AB02 AB04 AB08 AB12 AB14

Claims (9)

[Claims] 1. A parallax control unit that corrects parallax between a plurality of viewpoint images for displaying a stereoscopic image, and a map holding unit that holds a correction map to be referred to by the parallax control unit during the processing. The stereoscopic image processing apparatus according to claim 1, wherein the correction map is described such that the parallax is corrected based on a position in a viewpoint image. 2. The parallax control unit reduces the parallax in a peripheral portion of the plurality of viewpoint images, or changes the parallax so that an object is sensed farther from a user. 1. The device according to 1. 3. The apparatus according to claim 1, wherein the parallax control unit changes the parallax by selectively performing processing on any of the plurality of viewpoint images. 4. The plurality of viewpoint images are generated from three-dimensional data as a starting point, and the parallax control unit controls a camera parameter to change the parallax when generating the plurality of viewpoint images. The device according to any one of claims 1 to 3, characterized in that: 5. The plurality of viewpoint images are generated from three-dimensional data as a starting point, and the parallax control unit distorts the parallax by distorting the three-dimensional space itself when generating the plurality of viewpoint images. Device according to claim 1 or 2, characterized in that it is varied. 6. The plurality of viewpoint images are generated from a plane image to which depth information is given, and the parallax control unit changes the parallax by operating the depth information. Claim 1 or 2
The device according to. 7. A step of acquiring a plurality of viewpoint images for displaying a stereoscopic image, and a step of changing a parallax between the acquired viewpoint images based on a position in the viewpoint images. A stereoscopic image processing method characterized by the above. 8. The method according to claim 7, wherein each of the steps is implemented as a function of a stereoscopic display library, and the function of the library can be called as a function from a plurality of programs. 9. A computer comprising: a step of acquiring a plurality of viewpoint images for displaying a stereoscopic image; and a step of changing a parallax between the acquired viewpoint images based on a position within the viewpoint images. A computer program characterized by being executed by.