JP2007133843A - Three-dimensional image processing program and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a background image processing program and a device, in which a background image seen from the visual point of an operating object freely movable in a space range can be made to a real-scene image, in a game in which the object can freely move within an optionally set existing three-dimensional space range. <P>SOLUTION: An image most appropriate for visual point conversion is selected based on the position and direction of the visual point of the operating object from real images obtained by photographing all directions in a plurality of predetermined points within a three-dimensional space of a predetermined range. The background image visual point-converted from a part of the selected image is continuously displayed according to the movement of the visual point, whereby a real-scene background image which freely moves longitudinally, laterally and vertically with improved sense of reality or sense of presence is constituted. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing technique for displaying a live-action image of an actual place from a viewpoint that can move freely on a moving image background image in an image processing program executed by a computer.

  In a conventional game that can be operated by an operator and freely move within a predetermined three-dimensional space range, a method of displaying a virtual three-dimensional space image using a polygon model as the background image is often used.

  Here, the virtual three-dimensional space image means that a viewpoint (virtual viewpoint) is set at an arbitrary position in the three-dimensional space having information specified by coordinates in the three-dimensional coordinate space, and the image display element starts from the viewpoint. By making the scene overlooking the interior and continuously moving its viewpoint according to the movement of the operation object operated by the operator, it feels as if the operator is moving in the three-dimensional space. It is an image and is generally called 3D computer graphics.

  However, according to the above technique, since it is a virtual three-dimensional space image, there is a difference compared with a live-action background video of an actual place. As a result, the reality and presence are poor. Further, only by changing the playback speed of a video obtained by shooting a live-action image as a background image (JP-A-9-220308), there is only a degree of freedom in the shooting direction. In other words, the degree of freedom is one-dimensional. Furthermore, in the method (Japanese Patent No. 3726239) for displaying the reproduced image after changing the viewpoint, the reproduction image can move freely only in the photographing direction and in a limited range around it. In other words, the degree of freedom is a combination of a one-dimensional one and a two-dimensional one with a limited range.

  Therefore, the present invention provides a live-action image of a background image viewed from the viewpoint of an operation target that can freely move in the space range in a game or the like that can freely move in an actually set arbitrary three-dimensional space range. It is an object of the present invention to provide a background image processing program and apparatus.

  In order to solve the above-described problems, the first invention at least outputs a manipulation position and a manipulation direction based on manipulation information of a movable manipulation target in an arbitrarily set range in a three-dimensional space; A process of selecting an appropriate image based on the operation position from a plurality of photographed images taken in a desired direction within the range, and a viewpoint conversion background image from the viewpoint based on the operation position and the operation direction. An image processing program for causing a computer to execute

  According to the first aspect of the present invention, a live-action image in which a desired direction is photographed at a plurality of points within an arbitrarily set range in a three-dimensional space is prepared, and photographed at a point closest to the manipulation position of the manipulation target. By displaying an image obtained by converting the viewpoint from the viewpoint based on the operation position and operation direction of the operation target object, a live-action image from the position and direction operated by the operator within an arbitrarily set range is displayed. Can do. This desired direction is a direction arbitrarily determined according to preference or necessity, and all directions are preferable for outputting a realistic image. For example, if the operation direction is two-dimensional only in the horizontal direction, the captured image in the vertical direction is taken. Is optional because it can be omitted.

  The second invention is the image processing program according to claim 1, which further causes a computer to execute a process of creating a moving image by continuously displaying the viewpoint conversion background image.

  According to the second aspect of the present invention, as the moving image display means for giving an operational feeling that the operator is moving freely in a predetermined three-dimensional space by continuously displaying the images converted from the viewpoint of the first aspect. Make the computer work. The “program for causing a computer to execute a process” is a process executed by a CPU of a computer apparatus or a program for giving an instruction for the CPU to perform processing on hardware connected to the CPU.

  According to a third aspect of the present invention, there is provided operation information output means for outputting an operation position and an operation direction based on operation information of a movable operation object in an arbitrarily set range in a three-dimensional space, and a desired direction within the predetermined range. An image selection unit that selects an appropriate image based on the operation position and the operation direction from a plurality of photographed images, and a viewpoint conversion background image from the viewpoint based on the operation position and the operation direction. A viewpoint conversion background image creation means for creating, and the fourth invention includes a moving image creation means for continuously displaying the viewpoint conversion background image and creating a moving image. The image processing apparatus according to claim 3, wherein the image processing apparatus is an image processing apparatus.

  According to the third and fourth inventions of the third and fourth aspects, a real image obtained by photographing all directions at a plurality of points within a predetermined range in a three-dimensional space is prepared, and from the operation position of the operation object By continuously displaying images obtained by converting viewpoints from viewpoints based on the operation position and operation direction of images captured at the closest point, the operator feels that the operator is moving freely in a predetermined three-dimensional space. An image display processing device can be provided.

  According to the first and second aspects of the invention, it is possible to provide a computer program that displays a continuous live-action background moving image from a viewpoint that freely moves in an arbitrarily set three-dimensional space. According to the third and fourth aspects of the invention, it is possible to provide an image processing apparatus that displays a continuous live-action background moving image from a viewpoint that is moving freely in an arbitrarily set three-dimensional space.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. This embodiment has a structure as shown in FIG. 1 when viewed from above with the floor as a reference, and as viewed in FIG. 5 when viewed from diagonally above. It is assumed that the program displays a video from a viewpoint moved at a constant speed for 20 seconds at a speed of 30 frames per second. Also, the front, back, left, right, up and down directions are photographed at the points P1 to P9 shown in FIG. It is assumed that the operation information for which the viewpoint moves is created in advance. At this time, it is assumed that the heights of the points S, E, and P1 to P9 from the floor of the room are all the same.

  Here, 301 in FIG. 2 will be described. A square portion labeled “upper” 301 is taken from the shooting point P7 in the upward direction (Pv2 direction in FIG. 5) with the floor as a reference, and a square portion written “right” is taken from the shooting point P7. The right direction (Pv3 direction in FIG. 5) is taken. Similarly, “down”, “left”, “front” (Pv1 direction in FIG. 5), and “rear” are photographed. In this embodiment, it is assumed that an image taken at each shooting point is recorded on the recording medium 2 in a compressed state by a method compliant with JPEG (Joint Picture Experts Group).

  FIG. 3 is a block diagram showing the configuration of an embodiment of a computer apparatus to which the image processing apparatus of the present invention is applied. This apparatus includes a main body 1, a recording medium 2 storing program data and an image file, an input device 3 for inputting operation information of an operator, a speaker 4 for outputting sound during a game, and a computer display for displaying a game screen. It consists of five.

  The main body 1 has a CPU (Central Processing Unit) 100 and a bus line 101 including an address bus, a data bus, and a control bus connected to the CPU 100. A data RAM (main memory) 102, an input interface 103, a ROM 104, a recording medium decoder 105, a drawing processor 108, a decompression circuit 110, a sound processor 111, and an amplifier 113 are connected to the bus line 101.

  The form of this computer device varies depending on the application. For example, when the computer device is an arcade game device, all the components shown in FIG. 1 are housed in a single case, and when the home game device includes a CPU, it is connected to a TV monitor instead of the display 5 and the speaker 4. However, the input device becomes a game controller or the like.

  Next, each component will be described. First, the data RAM 102 stores a program body, image data, other necessary data, a work area, and the like. The input interface 103 is a process for extracting the operation information of the operator from the input device and making it into a form that can be processed by the CPU 100.

  The ROM 104 corresponds to a basic input output system (BIOS) in a personal computer. The ROM 104 is a computer device startup process, a program for controlling reading and execution of a program executed first on the recording medium 2, and a driver for a basic input / output device. Stores programs.

  The recording medium driver 107 reads data from, for example, a recording medium such as a CD-ROM, a DVD-ROM, or a hard disk drive, and performs error correction correction processing by ECC (Error Correction Code) in the recording medium decoder 105. In accordance with the instruction from the CPU, the read data is sent to the data RAM 102 or the like.

  The drawing processor 108 generates a signal that allows the monitor to display the contents of the drawing buffer 109 as an image based on an instruction from the CPU 100.

  The decompression circuit 110 performs a process of decompressing a compressed image based on MPEG (Moving Picture Experts Group) for moving images and JPEG for still images. In this embodiment, the image data is decompressed for use as a background.

  The sound processor 111 stores the sound data recorded on the recording medium directly or temporarily in the buffer 112 based on an instruction from the CPU 100 and processes the sound data as necessary (with the sound effect applied). Is generated. The generated sound is amplified by the amplifier 113 and output from the connected speaker 4.

  FIG. 4 is a program flowchart in this embodiment. First, when the program processing is started, the CPU 100 reads necessary initial data such as the image data previously created and recorded from the recording medium 2 into a part of the data RAM 102 (step S1). CPU100 which completed reading starts the process of a program main body (step S2).

  When the process is started, the viewpoint position and direction are calculated from the viewpoint operation information (step S3). That is, the process of step S3 performed by the CPU 100 is a process of outputting the operation position and the operation direction based on the operation information of the movable operation object in the arbitrarily set range of the three-dimensional space.

  Then, the CPU 100 obtains a photographing point having the closest distance from the calculated viewpoint position, and selects a photographed image from the obtained photographing point (step S4). For example, when the viewpoint position is the point S in FIG. 1, the closest shooting point obtained by the CPU 100 is P7, the CPU 100 selects an image shot from the shooting point P7, and the expansion circuit 110 selects the image based on the instruction of the CPU 100. Expand the recorded image. That is, the process of step S4 performed by the CPU 100 is a process of selecting an appropriate image based on the operation position from a plurality of photographed images taken in a desired direction within an arbitrarily set range.

  Then, the CPU 100 cuts out an area necessary for the viewpoint conversion process performed in step S6 from the selected photographed image (step S5). CPU100 performs the process which deform | transforms the shape of the part cut out as needed, after cutting out. The CPU 100 creates an image obtained by subjecting the cut image to a viewpoint conversion process based on the position and direction of the viewpoint (step S6). That is, the process of step S6 performed by the CPU 100 is a process of creating a viewpoint conversion background image from the viewpoint based on the operation position and the operation direction of the selected image.

  Here, the viewpoint conversion technique of “Japan Patent Office Standard Technology Collection, Technology Classification A-10 Combining with Live Action, Synthesizing Method of 2D Video and 3D CG with Moveable Viewpoint” is used. This "Japanese Patent Office Standard Technology Collection, Technology Classification A-10 Combining with Live Action, Synthesizing Method of 2D Video and 3D CG that Can Move the Viewpoint" transforms one image as seen from another camera angle. However, this is an image composition method that allows viewpoint movement. In order to obtain a natural composite image of CG and live-action, it is necessary to match the position, direction, and angle of view of the camera for live-action shooting with the camera angle for CG video production. Therefore, it is a technique that can be easily combined if the real camera image fixed to the camera is deformed in accordance with the camera angle of the CG instead of matching the camera angle of the real camera with the CG. The system automatically estimates the three-dimensional shape model of the original image by dividing the image into regions such as left and right, front wall, ceiling, and floor, and obtaining respective three-dimensional coordinates. This is a technique for naturally synthesizing a live-action picture and a CG by transforming a two-dimensional image so as to match the camera angle of the CG based on the three-dimensional shape. (Source: “Interactive image re-synthesis that expresses arbitrary viewpoint movement”, “Information Processing Society of Japan Research Report Graphics and CAD 81-11” (August 23, 1996), Yoichi Horii, Kiyoshi Arai, Information Published by Processing Society of Japan)

  Returning to the description of FIG. The CPU 100 transfers the viewpoint-converted image data subjected to the viewpoint conversion process in step S6 to the drawing buffer 109. The drawing processor 108 displays the contents of the drawing buffer on the monitor (step S7). Next, the CPU 100 determines whether or not the display time has reached 20 seconds (in this case, the number of displayed sheets is 30 sheets × 20 seconds = 600 sheets), and when the display is performed for 20 seconds, the process ends (step S8). . By repeating the above steps S3 to S8 in 1/30 seconds by the CPU 100 and the like, it is possible to create a moving image from the viewpoint moving from the S point to the E point. That is, the process of repeating steps S3 to S8 performed by the CPU 100 or the like is a process of creating a moving image by continuously displaying the viewpoint conversion background image.

  Here, a specific example is given and demonstrated about the process of step S5 and step S6. For example, the viewpoint position is the point S in FIG. 1, the viewpoint direction is the upward direction in FIG. 1 (the arrow direction from the point S in FIG. 1), and the selected image is taken at the photographing point P7 301 in FIG. In such a case, the region cut out from 301 in FIG. 2 for the viewpoint conversion processing is a hatched portion 304 in FIG. Next, an image (302 in FIG. 2) in which a part of the left side of the cut image is shape-converted is created. Since an image obtained by partially converting the shape is an image taken from the photographing point P7, a viewpoint background image from the point S as indicated by 303 in FIG.

"Effect of the embodiment"
According to this embodiment, it is possible to provide a live-action image from a viewpoint that freely moves within a predetermined range of the three-dimensional space, so that it is possible to improve realism and realism. This can provide a live-action background moving image with a sense of reality and presence in a real room.

"Other embodiments"
In the above embodiment, the image prepared in advance has a data structure in which the dice are expanded. However, there are several elongated lens shapes such as a shape in which the cylinder is expanded (FIG. 6) and a map attached to the globe. Or they may be arranged side by side. In the above embodiment, the operation information is given in advance. However, the present invention is also applicable to a game that can be operated in real time regardless of genre, such as a game that can freely move in a three-dimensional space. It is possible to provide a live-action moving image that freely moves in an actual space. In addition, the viewpoint conversion processing of the above-described embodiment was performed using the method of “Japan Patent Office Standard Technology Collection, Technology Classification A-10 Combining with Live Shooting, Synthesizing Method of 2D Video and 3D CG That Can Move the Viewpoint”. If the distance information from the photographing point of all the elements of the image (all the dots of the image) is given, the viewpoint conversion process with the same result can be performed.

  Further, in the above embodiment, the range of the three-dimensional space in which only the interior of the room that is supposed to exist is arbitrarily set, but this set range is not limited to a narrow place and may be all corners of the city as shown in FIG. By preparing images in which all directions are taken in advance at a plurality of points outside the building in the corner of the city and inside the entire building, it is possible to provide a live-action background video when moving freely within this range. Furthermore, it can also be used to create live-action moving images such as images created when trying a better moving image while moving the camera (viewpoint) variously in one scene of a movie or TV program, or a camera angle image that cannot be moved normally. Can be applied.

Top view of a room that is assumed to exist in the embodiment Image taken from P7 point in the embodiment, cut out, viewpoint conversion processing image diagram 1 is a configuration block diagram of an embodiment of a computer apparatus according to the present invention. Program flowchart in the embodiment The perspective view seen from the diagonally upper direction of the room which the Example has existed. In order to facilitate understanding, the walls separating the spaces are transparent, and only the pattern drawn on each wall is displayed with a solid line. The pattern is the same on both sides of the wall. Shooting data in the form of a cylinder developed in another embodiment (taken from P5 in FIG. 1) A corner of a real city

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Computer apparatus main body 2 Recording medium 3 Input device 4 Speaker 5 Display monitor 100 CPU
101 Bus line 102 Data RAM
103 Input interface 104 ROM
DESCRIPTION OF SYMBOLS 105 Recording medium decoder 106 Recording medium buffer 107 Recording medium driver 108 Drawing processor 109 Drawing buffer 110 Decompression circuit 111 Sound processor 112 Sound buffer 113 Amplifier 200 Top view of the actual room of the embodiment 201 Floor of the actual room of the embodiment 301 Image 302 in front / back / left / right / up / down direction taken from point P7 in FIG. 1 Image 303 cut out from 301 based on operation position and operation direction Image 302 obtained by converting viewpoint from viewpoint based on operation position and operation direction 304 301 302 part cut out from the line
400 A perspective view 401 seen from diagonally above a room assumed to exist in the embodiment 401 Development view of a room assumed to exist in the embodiment

Claims (4)

  At least processing for outputting an operation position and an operation direction based on operation information of a movable operation object in an arbitrarily set range in a three-dimensional space, and a plurality of live-action images obtained by capturing a desired direction within the range An image for causing a computer to execute a process of selecting an appropriate image based on the operation position from the image and a process of creating a viewpoint conversion background image from the viewpoint based on the operation position and the operation direction of the selected image. Processing program.   Furthermore, the image processing program of Claim 1 for making a computer perform the process which produces the moving image by continuously displaying the said viewpoint conversion background image.   Operation information output means for outputting an operation position and an operation direction based on operation information of a movable operation object in an arbitrarily set range in a three-dimensional space, and a plurality of images obtained by photographing a desired direction within the predetermined range Image selection means for selecting an appropriate image based on the operation position and the operation direction from a photographed image, and a viewpoint conversion background for generating a viewpoint conversion background image from the viewpoint based on the operation position and the operation direction for the selected image And an image creating means.   The image processing apparatus according to claim 3, further comprising a moving image creating unit that creates a moving image by continuously displaying the viewpoint conversion background image.

Images