JP5652037B2 - Simulated video generation device, method, and program - Google Patents

Description

  The present invention relates to a simulation technique for displaying a video in a virtual space in response to a user operation input.

  In recent years, real-time video simulation technology has been developed that allows a user to walk through, drive through, or fly through a virtual world created by computer graphics in real time.

  In such real-time video simulation, there is a need to express complex scenes created in more detail in order to enhance the reality and realism of video. For example, in a 3D game, it is desired to express a more precisely modeled world. However, a drawing computer for drawing a virtual world by computer graphics has a limit in performance. If you try to draw a scene with a complexity that exceeds that limit, the time taken to draw one frame of the video will increase, and the frame rate (number of frames drawn per unit time) will decrease and the video will move. The smoothness is lost. At the same time, the video is delayed with respect to the operation input, which causes a great sense of discomfort for the user moving in the virtual world.

  Conventionally, in order to prevent this uncomfortable feeling, the scene has been simplified by performing data reduction work so as not to exceed the limit at which the drawing computer can maintain a sufficient frame rate. In other words, it was not possible to draw a complex scene that exceeded the performance limit of a single drawing computer. In the case of a 3D game, the limit of the drawing performance of a single game machine has determined the limit of expression in the virtual world. Therefore, in order to improve the time delay in drawing, a method of correcting a positional error caused by the time delay by translating the image has been proposed (see, for example, Patent Document 1).

JP-A-5-324799

  However, when the image is translated by the conventional method, an undrawn area may occur after the movement. The larger the drawing area error due to time delay, the larger the undrawn area. This may cause a new feeling of strangeness.

  The present invention has been made in view of the above problems, and an object thereof is to reduce a sense of incongruity due to a delay of a simulation video.

The simulated video generation device disclosed in the present application updates data representing a virtual world according to an operation input from a user or the passage of time, and calculates a visual field to be displayed in the virtual world, and a first time, A drawing field calculation unit that calculates a field of view to be drawn which is a field of view wider than the first field of view calculated by the simulation unit; and generation of a first video corresponding to the field of view to be drawn A video generation unit that starts immediately after the first time based on data representing the world, and a second display calculated by the simulation unit at a second time after the start of generation of the first video An image correction unit that corrects the first image to a second image corresponding to the second field of view to be displayed immediately after the second time using a field of view; Before At the first time, the second visual field to be displayed at the second time is predicted and calculated by a predetermined method, and the visual field to be drawn is the first visual field to be displayed and the predicted first visual field. as the inclusive vision field of vision to be displayed of 2, the calculation of the visual field to be the drawing.

  According to the disclosure of the present specification, it is possible to reduce a sense of incongruity due to a delay of a simulation video.

It is a functional block diagram which shows an example of a structure of the simulation apparatus (simulated image generation apparatus) concerning 1st Embodiment. It is a functional block diagram which shows the modification of a structure of the simulation apparatus (simulated image generation apparatus) concerning 1st Embodiment. It is a functional block diagram which shows the structural example of the simulation system in 2nd Embodiment. FIG. 4A is a diagram illustrating an example of the position and field of view of a car in the virtual world at time t = T _N and time t = T _Nn . FIG. 4B is a diagram showing an example of a visual field image to be displayed at time t = T _N and a video image corrected at time t = T _Nn . FIG. 5A is a diagram illustrating an example of the visual line direction vector and the relationship between the visual field and the video at time T _N and the video. FIG. 5B is a diagram illustrating an example of the relationship between the line-of-sight direction vector and the translated image at time T _Nn and an example of the translated image. FIG. 6A is a diagram illustrating an example of the visual line direction vector, the relationship between the visual field and the video, and the video when the video is captured and processed as a projection onto a spherical surface. FIG. 6B is a diagram illustrating an example of the relationship between the line-of-sight direction vector and the visual field and the image after movement, and the image after movement when the image is captured and processed as a projection onto a spherical surface. FIG. 7A is a diagram illustrating a video example before and after correction when the visual field at the time of video generation is drawn and corrected as it is. FIG. 7B is a diagram illustrating a video example before and after correction when a visual field to be drawn wider than the visual field at the time of video generation is calculated and corrected. FIG. 8 is a diagram illustrating an example of an image that is synthesized after a plurality of objects in the virtual world are drawn separately. FIG. 4 is a process flow diagram illustrating an operation example of the simulation system illustrated in FIG. 3.

  Embodiments of the present invention will be specifically described below with reference to the drawings.

(First embodiment)
[Example configuration of simulation device]
FIG. 1 is a functional block diagram illustrating an example of a configuration of a simulation apparatus (simulated video generation apparatus) according to the first embodiment. A simulation apparatus 10 shown in FIG. 1 is an apparatus that updates data representing a virtual world (virtual space) in response to an operation input from a user or the passage of time, and generates and displays a video of the virtual world by computer graphics. is there. The simulation apparatus 10 should be displayed by a computer using data representing a virtual world such as the position and shape of an object in the virtual world and data representing a field of view such as the position, orientation, and angle of view of the user's viewpoint (camera). It has a function to automatically generate a video of the virtual world. Note that the simulation target and application are not limited to specific ones.

  The simulation apparatus 10 includes a drawing visual field calculation unit 1, a video generation unit 2, a video correction unit 3, a display unit 4, a simulation unit 5, and an input unit 6. The simulation unit 5 updates the data representing the virtual world according to the operation input from the user received through the input unit 6 or the passage of time, and calculates the visual field to be displayed in the virtual world.

  Here, the visual field to be displayed is data for specifying the visual field of the user or the camera in the virtual world, and a video of the virtual world included in the visual field to be displayed is output by the simulation apparatus 10. The visual field to be displayed includes, for example, the position, orientation, angle of view, projection parameter, and the like of the viewpoint. The projection parameter includes, for example, a parameter that defines a surface on which an object or background in the virtual space is projected.

  The data representing the virtual world can be, for example, a set of shape data (for example, polygon data) of an object such as an object or a plane in the virtual world and attribute data such as position, orientation, color, and texture.

  For example, when an operation input is received via an input unit 6 from an input device including a handle and a pedal that can be operated by the user, and the user realizes a simulation of driving a vehicle in the virtual world, the simulation unit 5 controls the vehicle. A simulation for updating the system and drive system data according to the operation is performed, and the field of view of the virtual driver is calculated as the field of view to be displayed.

  Alternatively, if the measurement result of the sensor that measures the position and orientation of the user's head is received via the input unit 6 and the swing in the virtual world is realized, the simulation unit 5 The measurement result of the position and orientation of the head can be calculated to correspond to the movement of the head of the person in the virtual world, and the field of view of the virtual person can be obtained as the field of view to be displayed.

  In addition, the simulation part 5 can also update the data of a virtual world automatically with a predetermined period, and can perform a simulation, without receiving the operation input from a user. Or you may receive the calculation result of another simulation apparatus via the input part 6. FIG.

  The drawing visual field calculation unit 1 calculates a visual field to be drawn, which is a wider visual field than the visual field to be displayed. The visual field to be drawn is data representing the visual field of the video generated by the video generator 2. That is, the drawing visual field calculation unit 1 calculates the visual field of the video generated by the video generation unit 2 based on the visual field to be displayed updated by the simulation unit 5. The field of view to be drawn can include, for example, the position of the viewpoint, the orientation, the angle of view, the projection parameters, and the like, similarly to the field of view to be displayed.

  The visual field to be drawn calculated here is a visual field in a range in which the visual field to be displayed that is updated at the time after the video generation of the visual field to be drawn is started in the video generation unit 2 can be included. Is preferred. Therefore, the drawing visual field calculation unit 1 can calculate, for example, a visual field in a wider range than the visual field to be displayed at the time of video generation as the visual field to be drawn. In this case, the drawing visual field calculation unit 1, for example, in which direction and how much the visual field to be drawn is expanded with respect to the visual field to be displayed, history data of the visual field to be displayed calculated in the past, This can be calculated based on the time required for video generation. In this way, the drawing field calculation unit 1 calculates the field of view to be drawn, which is a field of view wider than the field of view to be displayed, and generates the image of the field of view to be drawn. It becomes possible to reduce the uncomfortable feeling due to the delay of the video.

  For example, the drawing visual field calculation unit 1 calculates a range that can be displayed after the start of image generation based on the history of the visual field to be displayed calculated by the simulation unit 5, and the visual field including the range is calculated as the visual field including the range. It can be a field of view to be drawn. Thus, based on the history of the visual field to be displayed, the visual field to be displayed after the start of video generation can be accurately predicted, and an appropriate visual field to be drawn can be obtained.

  In addition to or instead of this configuration, the drawing visual field calculation unit 1 uses the predicted value of the time taken from the start of video generation to the start of video correction by the video generation unit 2 as a history of the time taken to generate the video. And the range of the field of view to be drawn can be adjusted according to the predicted value. Accordingly, it is possible to more accurately predict the visual field to be displayed after the start of video generation and to obtain a more appropriate visual field to be drawn.

  The video generation unit 2 generates a video of the visual field to be drawn calculated by the drawing visual field calculation unit 1 based on data representing a virtual world. In the present embodiment, when the video generation unit 2 generates a virtual world video using the field of view to be drawn, the field of view to be displayed and the field of view to be drawn at the time when the generation of the image is started are respectively the display field of view. The holding unit 8 and the drawing visual field holding unit 7 hold the image in association with the generated video. The generated video is sent to the video correction unit 3. At this time, for example, as will be described later, the video may be sent between the video generation unit 2 and the video correction unit 3 via compression / decompression by a codec or via a network.

  The video correction unit 3 corrects the video to a video corresponding to the new visual field to be displayed, using the new visual field to be displayed calculated after the start of the video generation. The display unit 4 presents the video corrected by the video correction unit 3 to the user.

  As an example, the video correction unit 3 receives the video generated from the video generation unit 2, and receives from the simulation unit 5 a field of view to be displayed at the input timing (referred to as a first field of view). This first visual field is an example of a new visual field to be displayed calculated by the simulation unit 5 after the video generation unit 2 starts generating the video. The video correction unit 3 further displays a visual field to be displayed (second visual field) and a visual field to be drawn (third visual field) associated with the input video with the display visual field holding unit 8 and the drawing visual field, respectively. Read from the holding unit 7. Then, the input video is corrected using these three fields of view.

  The first field of view is the field of view to be displayed. Since the second and third visual fields are visual fields at the start of video generation, the visual fields are old for the time required for video generation and other necessary processing (for example, communication, codec, etc.). The image correcting unit 3 corrects the images of the old fields of view (second and third fields of view) to generate an approximate image of the field of view (first field of view) to be displayed. This correction can be performed by, for example, two-dimensional translation, enlargement / reduction, rotation, and the like of the video, but is not limited thereto.

  The video correction unit 3 is, for example, a region corresponding to a new first field of view (new field of view to be displayed) calculated after the start of generation of this image among the images of the third field of view (field of view to be drawn). Can be corrected to become a display image. Thereby, even if the image generation is delayed with respect to the change in the visual field, the video can be corrected so as to correspond to the change in the visual field. Further, since the third field of view is wider than the second field of view, the possibility that the range visible from the first field of view deviates from the range of the second field of view is reduced, and the occurrence of further discomfort due to correction can be suppressed.

  More specifically, the video correction unit 3 can also perform correction for moving the display position of the generated video in accordance with the amount and direction of change from the second visual field to the first visual field. By correcting the display position in this way, it is possible to correct at high speed with a simple process.

  Note that when the visual field changes from the second visual field to the first visual field, the range visible from the first visual field deviates from the range of the second visual field. As described above, when the range visible from the first visual field deviates from the range of the second visual field, an undrawn portion is generated when an image in which the second visual field is drawn is corrected. Therefore, in the present embodiment, the drawing field calculation unit 1 obtains a third field of view (field to be drawn) in which the second field of view is adjusted to include the range of the first field of view, and the image generation unit 2 performs the first field of view. It is configured to generate images with three fields of view.

  Note that since the first field of view is the future when viewed from the timing of obtaining the third field of view, it is difficult to determine the third field of view so that the first field of view is surely included. Further, from the viewpoint of suppressing the load of image generation, it is desirable to make the third field of view as narrow as possible. Therefore, based on the tendency of the simulation system to which the simulation 10 is applied, a method for calculating the third visual field based on the second visual field can be determined in advance. For example, in a system such as a driving simulator where the visual field rarely moves in the vertical direction, the field of view of the second visual field can be mainly expanded in the horizontal direction to determine the third visual field. . Further, by estimating the maximum value of the horizontal movement of the visual field, the necessary amount of expansion of the horizontal angle of view can be determined. In addition, it can also be set as the structure which can update the expansion direction and expansion amount with respect to a 2nd visual field of a 3rd visual field based on the input from a user.

  Further, when the relationship between the second visual field and the third visual field is fixed, there is a high possibility that the drawing is wasted except in the direction in which the visual field actually moves. Therefore, the drawing visual field calculation unit 1 predicts the visual field to be displayed (first visual field) at the time of video correction from the history of the visual field to be displayed (first visual field or second visual field), and determines the predicted visual field. By obtaining the third visual field so as to include it, it is possible to suppress the waste of drawing. For example, the maximum value of the lateral movement of the visual field can be estimated based on the history of the first visual field and / or the history of the video generation time by the video generation unit 2.

[Modification of simulation apparatus]
FIG. 2 is a functional block diagram illustrating a modified example of the configuration of the simulation apparatus (simulated video generation apparatus) according to the first embodiment. In FIG. 2, the same functional blocks as those in FIG. The simulation apparatus 10a shown in FIG. 2 further includes a second video generation unit 11 and a synthesis unit 9 in addition to the configuration of the simulation apparatus 10 of FIG. Based on the new field of view to be displayed calculated by the simulation unit 5 after starting the generation of the image by the first image generating unit 2, the image generating unit 11 is a part of the virtual world in the new field of view to be displayed. Generate video. The synthesizing unit 9 synthesizes the video corrected by the video correction 3 and the video generated by the second video generating unit 11. The display unit 4 outputs the video synthesized by the synthesis unit 9.

  According to this configuration, it is possible to draw a virtual world image by dividing a portion to be corrected and a portion not to be corrected, and then combining after correcting one. For this reason, it is possible to correct only a portion having a high degree of change with respect to an image including a portion having a different change degree with respect to the motion of the user's field of view (camera) in the virtual world, thereby reducing a sense of incongruity due to time delay.

  For example, the image of the vehicle body of the vehicle in the driving simulator moves differently from the image of the outside world with respect to the camera (driver view). When there is a portion that moves differently with respect to the field of view, the sense of incongruity may increase if the image of the vehicle body is moved together with the outside world by correction. Therefore, in such a case, the video of the external part is generated by the first video generation unit 2, and the generated video corresponds to the visual field to be displayed after the video correction unit 3 generates the video. Can be corrected as follows. Then, the video of the vehicle body part can be generated by the second video generation unit 11 so as to correspond to the visual field to be displayed after the video generation.

  Specifically, the simulation unit 5 updates the virtual world data in accordance with the operation input from the user received by the input unit 6 as in the case of FIG. As a result, information on the visual field to be displayed in the virtual world (for example, position, orientation, angle of view, projection parameters, etc.) is also updated. The drawing visual field calculation unit 1 calculates the visual field to be drawn by the first video generation unit 2 based on the visual field to be displayed as in the case of FIG.

  The first video generation unit 2 generates a video of a part of a pre-selected target in the virtual world using the input visual field to be drawn. For example, the first video generation unit 2 can select an object in a target virtual world for generating a video by referring to data indicating a target object recorded in advance in a memory or the like. The generated video is sent to the video correction unit 3.

  The visual field to be displayed and the visual field to be drawn when the first video generation unit 2 starts generating the video are held in association with the generated video by the display visual field holding unit 8 and the drawing visual field holding unit 7, respectively. The

  The video correction unit 3 inputs the video generated by the first video generation unit 2, inputs the visual field to be displayed at the timing from the simulation unit 5, and draws the visual field to be displayed associated with the input video. The visual field to be read is read from the display visual field holding unit 8 and the drawing visual field holding unit 7, respectively. Then, the input video is corrected using these three fields of view, and the corrected video is sent to the combining unit 9. The correction can be performed in the same manner as in FIG.

  The second video generation unit 11 generates a video of a part of objects selected in advance in the virtual world (those not to be corrected) according to the field of view to be displayed, and sends the video to the synthesis unit 9. Data for specifying what is not to be corrected in this virtual world can also be recorded in advance in a memory or the like. The second video generation unit 11 can specify a target for generating a video in the virtual world with reference to data recorded in advance.

  The combining unit 9 combines the images from the image correction unit 3 and the second image generation unit 11 and sends them to the display unit 4. The composition can be performed, for example, by superimposing two images in a predetermined order. In each image, the portion of the field of view where the object is not shown may be transparent.

  The simulation apparatuses 10 and 10a are configured to correct an old image by a delay time and obtain an approximate image for the current visual field using information about the current visual field and the visual field at the start of image generation. Thereby, the responsiveness of the video with respect to the operation input is improved in a pseudo manner, and the uncomfortable feeling caused by the delay can be reduced. Furthermore, since the drawing visual field wider than the visual field to be displayed at the time of generating the video is calculated, it is possible to further suppress a sense of incongruity in the corrected approximate video.

  The simulation apparatuses 10 and 10a can be realized by installing a predetermined program in a general-purpose computer such as a personal computer or a server machine, for example. In addition, the simulation apparatuses 10 and 10a are formed by a computer incorporated in an electronic device such as a portable game machine terminal, an in-vehicle information terminal, a mobile phone, a PDA (Personal Digital Assistant), and a home appliance, for example. May be.

  In addition, by distributing the functions of the simulation apparatuses 10 and 10a to a plurality of computers, the simulation apparatuses 10 and 10a can be configured by a plurality of computers. For example, the functions of the simulation apparatuses 10 and 10a may be distributed to a plurality of general-purpose computers connected via a network. Alternatively, the simulation apparatuses 10 and 10a can be configured by a CPU and a DSP (Digital Signal Processor) connected by a bus.

  The drawing visual field calculation unit 1, the video generation unit 2, the video correction unit 3, the display unit 4, the simulation unit 5, the first video generation unit 2, the second video generation unit 11, the synthesis unit 9, and the input unit 6. Each function is realized by a processor such as a CPU or a DSP (Digital Signal Processor) executing a predetermined program. Therefore, a program for realizing the above functions by a computer or a recording medium on which the program is recorded is also an embodiment of the present invention. This recording medium is a non-transitory recording medium and does not include a temporary medium like a signal. The drawing visual field holding unit 7, the display visual field holding unit 8, and other means for recording necessary data are realized by a built-in storage device of a computer (for example, a memory such as a ROM or a RAM) or a storage device accessible from the computer. It becomes. The same applies to each functional unit in the second embodiment described below.

[Second Embodiment]
FIG. 3 is a functional block diagram illustrating a configuration example of the simulation system according to the second embodiment. The simulation system shown in FIG. 3 includes a simulator server 12 (hereinafter simply referred to as server 12) that is an example of a server device and a simulator client 13 (hereinafter simply referred to as client 13) that is an example of a client device. The server 12 and the client 13 can communicate with each other. This communication is not limited to wired communication but may include wireless communication.

  The server 12 includes a simulation unit 5, a drawing visual field calculation unit 1, and a video generation unit 2. The simulation unit 5 updates the data representing the virtual world according to the operation input from the user or the passage of time, and calculates the visual field to be displayed in the virtual world. The visual field calculation unit 1 calculates a visual field to be drawn, which is a wider visual field than the visual field to be displayed. The video generation unit 2 generates a visual field image to be drawn based on data representing a virtual world. The functions of the simulation unit 5, the drawing visual field calculation unit 1, and the video generation unit 2 can be the same as those in the first embodiment.

  The server 12 further includes a video generated by the video generation unit 2, a visual field to be displayed based on a visual field on which the video is to be drawn, and a new visual field to be displayed calculated after the generation of the video is started. As a transmission unit that transmits to the client 13, a drawing result transmission unit 18 and a visual field transmission unit 16 are provided.

  The client 13 uses the video sent from the transmission unit of the server 12 and the new field of view to be displayed calculated after the start of the generation of the video, and this video corresponds to the new field of view to be displayed. A video correction unit 3 for correcting the video is provided.

  As described above, the simulation unit 5, the video generation unit 2, and the transmission unit (the drawing result transmission unit 18 and the visual field transmission unit 16) are provided in the server 12, and the video correction unit 3 is provided in the client 13. In addition, it is possible to reduce the uncomfortable feeling of the video due to the delay due to communication. As a result, it is possible to use the client 13 via the network for drawing complicated scenes by the server 12 and for applications that require a quick response.

  The simulation system having the configuration shown in FIG. 3 can be used in, for example, a business simulation apparatus that receives inputs from a plurality of clients and returns simulation results to the client, or a network game. The use and object of the simulation are not particularly limited.

[Example of simulation]
In the example illustrated in FIG. 3, the client 13 includes an input unit 6 that receives an operation input from a user, and an operation input transmission unit 24 that transmits the operation input to the server 12.

  As an example, the client 13 includes an input unit 6 connected to a handle or a pedal, and a display unit 4 such as a display. The user can drive in the virtual world by operating an input device such as a handle while watching the video displayed on the display.

  The input information input by the client 13 is transmitted through a network, for example, received by the operation input receiving unit 14 of the server 12, and sent to the simulation unit 5. Since the input information has a small amount of data, it can be sent with a negligible delay.

  As an example, the simulation unit 5 holds data of objects such as roads, vehicles, and cityscapes that make up the virtual world as simulation data. The simulation data is recorded, for example, in the server 12 built-in memory or accessible memory. The simulation unit 5 performs a simulation on the behavior of each object based on the input information, and updates the simulation data representing the virtual world in real time. As a result of the update, for example, the position / posture and attributes (color, texture, etc.) of the object in the virtual world are changed. The simulation is usually lighter processing than video generation, and is executed at a high frequency of, for example, a period of several milliseconds to several tens of milliseconds.

  As a result of the simulation, information on the visual field (for example, position, orientation, angle of view, projection method and parameters thereof) used to display the virtual world is determined. For example, in the case of a driving simulator, the field of view is usually fixed to the viewpoint of the virtual driver in the driver's seat of the virtual car, so when the position and posture of the car are updated, the field of view to be displayed accordingly Updated. The viewpoint (camera) can be placed at any place such as another place of the car, another car, or an object or building on the road as required.

  Thus, when the visual field to be displayed by the simulation unit 5 is updated, the drawing visual field calculation unit 1 calculates the visual field to be drawn by the video generation unit 2 based on the visual field to be displayed. It is preferable that the visual field transmission unit 16 transmits the visual field to be displayed updated in real time to the client 13 every time the visual field to be displayed by the simulation unit 5 is updated. As a result, the video correction unit 3 of the client 13 can correct the old video corresponding to the updated field of view to be displayed.

[Example of parallel drawing]
As shown in FIG. 3, the video generation unit 2 may include a plurality of drawing units that draw a plurality of frames in parallel. In the present embodiment, as an example, the video generation unit 2 draws a plurality of drawing units 22 a, 22 b, and 22 c that can execute drawing (video generation) processing in parallel and the drawing visual field calculation unit 1 calculates. The visual field of view should be transferred to the plurality of drawing units 22a, 22b, and 22c, and the parallel drawing control unit 21 that executes processing for generating the video of the visual field to be drawn, and the drawing units 22a, 22b, and 22c, respectively. And a parallel drawing integration unit 23 for receiving the generated video together with the visual field to be displayed and the visual field to be drawn. With this configuration, for example, frames can be sequentially assigned to the drawing units 22a, 22b, and 22c connected in parallel, and drawing can be performed in parallel with a time difference. Drawing can be performed by a so-called alternate frame drawing method.

  Specifically, the video generation unit 2 (also referred to as a parallel drawing unit) of the server 12 generates a virtual world video according to the visual field to be drawn obtained by the drawing visual field calculation unit 1 as described below. For example, one of the drawing units 22a, 22b, and 22c can be assigned to each video frame to perform drawing in parallel. Thereby, a video is generated at high speed. Each of the drawing units 22a, 22b, and 22c uses memory drawing data (for example, model data and texture data of an object constituting the virtual world) that is added with virtual world update information sent from the simulation unit 5. It is possible to generate a video of the virtual world. In this way, when assigning drawing units in units of frames, the number of frames that can be generated per unit time is several times as many as the drawing units (three times in the example shown in FIG. 3) compared to drawing in a single drawing unit. ) Can be improved. That is, the frame rate can be increased to the number of prepared computers (drawing units).

  However, the time taken to draw one frame is the same as that for a single frame. Therefore, even if the frame rate is improved, the delay from the input of the visual field information to be displayed to the video generation unit 2 until the video is output is the same as in the case of a single drawing unit. The influence of this delay can be reduced by a function including the drawing visual field calculation unit 1 and the video correction unit 3 of the client 13.

  The parallel drawing control unit 21 monitors the execution state of each drawing unit 22a, 22b, 22c. If there is a drawing unit that has finished drawing the previous frame, the virtual unit receives update information of the virtual world and the visual field to be displayed from the simulation unit 5, receives information on the visual field to be drawn from the drawing visual field calculation unit 1, and receives the drawing unit. To generate the next video. As a result, all the drawing units are always in a state of generating images in parallel. Each drawing unit stores the received field of view to be displayed and field of view to be drawn during image generation.

  The parallel drawing integration unit 23 monitors the execution state of each drawing unit, and if there is a drawing unit that has finished drawing the frame, the drawing unit displays the video of the drawn frame, the field of view to be displayed, and the field of view to be drawn. Receive. The drawn frame video is passed to the video encoder 17. As a result, the images of the frames drawn in parallel by the drawing units 22a, 22b, and 22c with a time difference are sequentially sent to the video encoder 17.

  The video generation unit 2 has three drawing units as drawing computers, but the number of drawing computers can be changed in accordance with the required peak performance. However, when the number of drawing computers is matched to the peak performance, the entire apparatus including the drawing computers tends to be large-scale, and waste when peak performance is unnecessary tends to increase. Therefore, it is possible to make a configuration in which a plurality of drawing computers can be shared and used via a network. For example, as in the present embodiment, it is possible to transmit a video processed in parallel by a plurality of rendering units to the outside and perform correction at the transmission destination, thereby rendering a complex scene by a rendering device having a parallel configuration. Necessary application network can be used. The configuration of the present embodiment can also cope with a problem that a delay in a codec or communication itself is added when a rendering result image is transmitted to a client via a network.

[Server-client communication]
The video drawn by the video generation unit 2 is sent to the video encoder 17 and compressed to become a compressed video. The compressed video is transmitted to the client 13 through the network after being associated with the corresponding visual field information (the visual field to be displayed and the visual field to be drawn) in the drawing result transmitting unit 18.

  The transmitted compressed video and visual field information are received by the drawing result receiving unit 28 of the client 13. The video is sent to the video decoder 27 and decompressed. The visual field information of the visual field to be displayed and the visual field to be drawn is sent to the video correction unit 3. Video compression / decompression and network transmission require, for example, several tens to several hundreds of milliseconds.

  When receiving the video, the video correction unit 3 also receives visual field information corresponding to the video, that is, a visual field to be displayed and a visual field to be drawn at the start of video generation. Further, a new field of view to be displayed at the timing of receiving the video is received from the field receiver 26. The video correction unit 3 corrects the video based on the new field of view to be displayed. The timing, that is, the visual field to be displayed immediately before the video correction is acquired from the simulation unit 5 via the visual field transmission unit 16 and the visual field reception unit 26 via the network. Since the visual field information indicating the visual field to be displayed has a small amount of data, it can be sent with a negligible delay.

  As described above, in the present embodiment, the server 12 has a mechanism for independently transmitting the generated video and the visual field to be displayed that may be updated after the video generation to the client 13 independently in real time. Prepare. Thereby, the client 13 can correct the video by using the visual field to be displayed which is updated by the simulation unit 5 after the video is generated. Further, the server 12 calculates a field of view to be drawn wider than the field of view to be displayed and generates an image of the field of view to be drawn, so that it is possible to suppress a new sense of incongruity from being corrected by the client 13. Next, a specific example of correction processing of the video correction unit 3 in the client 13 and calculation of the field of view to be drawn in the server 12 will be described.

[Specific Example of Correction Processing by Video Correction Unit 3]
For example, if the user quickly moves his / her line of sight to the right and the field of view to be displayed quickly shifts to the right in the virtual world, the displayed image must immediately flow to the left. However, when there is a delay due to the generation and transmission of the video, the video flows to the left with a slight delay with respect to the movement of the visual field. This delay makes the user feel uncomfortable with the video, and causes VR (virtual reality) sickness.

  Therefore, the video correction unit 3 can execute video correction processing as follows, for example. The video correction unit 3 can perform correction such that the video is moved in a two-dimensional manner in accordance with the motion of the field of view that is currently known with respect to a slightly old video that has been drawn.

Here, an example of video correction will be described with reference to FIGS. 4A and 4B. FIG. 4A is a diagram illustrating an example of the position and field of view of a car in the virtual world at time t = T _N and time t = T _Nn . FIG. 4B is a diagram showing an example of a visual field image to be displayed at time t = T _N and a video image corrected at time t = T _Nn . In the example illustrated in FIG. 4A, a road is between the lines D1 and D2, and a building C, which is an example of an object, is located on the side of the road. The car K is proceeding diagonally to the right with respect to the building. The field of view of the car K is indicated by a dotted line, and this field of view is the field of view to be displayed. Here, the video production _| generation part 2 starts the production _| generation of an image _| video based on the visual field which should be displayed in time _TNn , and produces _| generates the video shown in the upper stage of FIG. 4B. Thereafter, the image correction unit 3 at a time T _N, and the video, received a field to be displayed at the time T _N, executes the correction. In this example, the image correction unit 3 translates the image in the left direction. As described above, the video correction unit 3 can move the video display area in accordance with the amount and direction of the change in the visual field from the visual field to be displayed at time T _Nn to the visual field to be displayed at time T _N. it can. A calculation example of the moving direction and moving distance of the video at this time will be described later.

  Due to the parallel movement of the image, the delay of the image is apparently concealed with respect to the operation input for moving the line of sight in the virtual world to the right, and the image immediately responds and flows to the left. Therefore, the uncomfortable feeling related to responsiveness is greatly reduced. Note that in the case of correction only by translating the video, the content of the video is slightly different from the video that should be originally. However, when the field of view is large, the sense of incongruity due to the displacement of the entire image is more dominant than the difference in the content of the image. A reduction effect can be obtained.

  The image correction method is not limited to the parallel movement of the image according to the change in the viewing direction. Corresponding to various viewpoint movements, translation, enlargement / reduction, rotation, affine transformation, etc. can be performed. Note that it is meaningless if a new large delay is caused by the video correction, so it is desirable that the video correction be as simple as possible. Hereinafter, a specific example of the image correction according to the change of the visual field will be described.

(Image correction corresponding to visual field rotation)
Here, the time corresponding to the visual field (the old field) to be displayed is sent from the rendering result receiving unit 28 (in this case, the time at the image generation start as an example) to be displayed is sent to T _Nn, from view receiver 26 _Let T _N be the time corresponding to the field of view (new field of view). The time T _N is a time after the start of video generation, and can be almost the current time, for example.

Focusing on the rotational movement of the field of view, the difference between times T _Nn and time T field in _{the N} line of sight vector (vector directed from the point of view to the center of the field of view), the pan relative to the viewpoint position in the T _Nn (about the vertical axis Left and right rotation), tilt (up and down rotation around the left and right axis (horizontal axis)), and roll (rotation around the line-of-sight direction vector). Respective rotation angles are θ _pan , θ _tilt , and θ _roll .

Here, let us consider a case where the field angle in the horizontal direction (horizontal direction) of the field of view is φ _fovH , the video width is W pixels, and the line-of-sight direction vector is rotated by θ _{pan in the} horizontal direction. In this case, the object near the center of the visual field moves in the horizontal direction by the number of pixels D _H represented by the following formula (1) in the video, for example.

図５Ａは、時刻Ｔ_Nにおける視線方向ベクトルおよび視野と映像の関係、および映像の一例を示す図である。図５Ａの上側は、映像の投影面に垂直な面における視線方向と視野の一例を表す図である。図５Ａの下側は、時刻Ｔ_Nにおける映像の例を示す。図５Ｂは、時刻Ｔ_N-nにおける視線方向ベクトルと平行移動された映像の関係、および平行移動された映像の一例を示す図である。図５Ａに示す映像は、視線ベクトルの水平方向の回転によって、図５Ｂに示すように、視線ベクトルの移動方向と逆方向へ、視線ベクトルの移動量に対応する映像上の長さ（Ｄ_Hピクセル）だけ平行移動される。すなわち、視線ベクトルが垂直軸を中心に時計周りにθ_panだけ回転すると、映像の投影面において表示するべき範囲は、線Ｓ１で挟まれた範囲から点線Ｓ２で挟まれた範囲にずれる。映像補正部３は、このずれにあわせて、映像を平行移動させることができる。

FIG. 5A is a diagram illustrating an example of the visual line direction vector and the relationship between the visual field and the video at time T _N and the video. The upper side of FIG. 5A is a diagram illustrating an example of the line-of-sight direction and the visual field in a plane perpendicular to the projection plane of the video. The lower side of FIG. 5A shows an example of an image at time _TN . FIG. 5B is a diagram illustrating an example of the relationship between the line-of-sight direction vector and the translated image at time T _Nn and an example of the translated image. As shown in FIG. 5B, the video shown in FIG. 5A has a length (D _H pixel) on the video corresponding to the amount of movement of the line-of-sight vector in the direction opposite to the direction of movement of the line-of-sight vector, as shown in FIG. ) Is translated. That is, when the line-of-sight vector rotates clockwise around the vertical axis by θ _pan , the range to be displayed on the image projection plane shifts from the range sandwiched by the line S1 to the range sandwiched by the dotted line S2. The video correction unit 3 can translate the video in accordance with this shift.

Similarly, let us consider a case where the field angle in the vertical direction (vertical direction) of the field of view is φ _fovV , the height of the video is H pixels, and the line-of-sight direction vector is rotated by θ _{tilt in the} vertical direction. In this case, the object in the vicinity field center is in the video, moves in the vertical direction by pixel number D _v represented by the following formula (2).

従って、Ｔ_N-n時点の映像を、視野中心の移動と逆の方向に、横にD_H、縦にD_Vだけ平行移動すれば、視野中心のオブジェクトの画面内での位置をＴ_N時点のものと一致させることができる。さらに、θ_rollが0でなければ、Ｔ_N-n時点の映像を映像中心周りにθ_rollだけ回転させることにより、視野中心のオブジェクトの画面内の向きもＴ_N時点のものと一致させることができる。これらにより、視野中央付近のオブジェクトが、操作入力に正しく対応して画面内を動くことになる。

Thus, the image of the T _Nn time, the movement opposite to the direction of the field center, next to D _H, vertical only D _V if translation, the position in the screen object field center of T _N times as Can be matched. Furthermore, if θ _roll is not 0, the orientation of the object at the center of the visual field in the screen can be made to coincide with that at the time of T _N by rotating the video at the time of T _Nn by θ _roll around the video center. As a result, the object near the center of the field of view moves in the screen in response to the operation input correctly.

  In the above correction, the corrected image is strictly correct only in the vicinity of the center of the visual field, and is approximated in other portions. This approximation can also reduce the sense of incongruity, but in order to increase the accuracy of approximation, there is a method of executing processing by capturing an image as a projection on a cylinder or a sphere instead of on a plane. The video correction unit 3 receives, for example, data representing an image pasted on a projection surface in a three-dimensional space such as a spherical surface or a cylinder as a video, corrects it according to a change in the field of view, and then converts the spherical image into a plane. The image projected on can be output.

6A and 6B are diagrams illustrating an example of a visual line direction vector at the time T _N and the time T _N , the relationship between the visual field and the video, and the video when the video is processed as a projection onto a spherical surface. 6A and 6B, the upper side shows an example of the line-of-sight direction vector and the visual field in a plane perpendicular to the projection plane, and the lower side shows an example of an image.

In the example illustrated in FIGS. 6A and 6B, the image correction unit 3 moves the image on the cylindrical surface or the spherical surface, which is the projection surface, instead of the parallel movement on the plane. An image of the visual field at the time point T _Nn (the range sandwiched between the two lines S1) pasted on a three-dimensional projection surface such as a cylinder or a spherical surface shown in FIG. 6A is displayed at the time point T _N as shown in FIG. 6B. The surface is moved on the spherical surface or the cylindrical surface so as to meet the field of view (a range between two dotted lines S2). The amount of movement can be determined by the angle between the line-of-sight vector at time T _{Nn and} the line-of-sight vector at time T _N. Alternatively, the video correction unit 3 extracts and corrects the video in the range of the new visual field to be displayed (for example, visual field at the time of _TN ) updated after the video generation from the video pasted on the spherical surface or the cylindrical surface. It may be output as a later video.

The processing for converting the image on the spherical or cylindrical surface into the image on the plane may be executed by the image correction unit 3 or a dedicated processor provided outside the image correction unit 3. Thus, an approximate image at the time point T _N can be generated by projecting the image after moving on the spherical surface or the cylindrical surface onto the plane again.

  If the field of view is only pan, the projection surface can be a cylinder, but if there is a tilt or roll, it is preferable to perform processing with a spherical surface. Further, the projection surface is not limited to a spherical surface or a cylindrical surface, and may be, for example, a cube, a rectangular parallelepiped, or a surface in another three-dimensional space. As described above, when the image transformation process is performed in three dimensions, the calculation load increases. For example, the load can be reduced by utilizing the texture mapping capability in recent standard graphics hardware.

(Correction of visual field translation)
When the field of view translates (especially when moving forward and backward), it is difficult for the user to notice that the image is delayed with respect to the operation input compared to when the field of view rotates. Therefore, the effect of reducing the uncomfortable feeling of the video can be obtained only by correcting the rotation of the visual field.

In the case where the image is corrected for movement in the same direction as the line-of-sight direction of the viewpoint position, that is, back-and-forth movement, the image correcting unit 3 can perform correction by enlarging or reducing the image. For example, when the viewpoint position has moved forward in the line-of-sight direction from the time T _{Nn at the} start of video generation at the time T _{N when} the correction after the video generation is started, the video correction unit 3 displays the video according to the moving distance. Can be enlarged.

  The enlargement ratio at this time depends on which distance in the depth direction (line-of-sight direction) is focused. That is, it is preferable to increase the enlargement ratio when focusing on nearby objects, and conversely, it is preferable to decrease the expansion ratio when focusing on distant objects. Thus, there is no correct enlargement factor for all distance objects. For this reason, it is possible to determine a distance of interest (reference distance) in advance and record it in a memory or the like so as to minimize the sense of discomfort according to the characteristics of the application. For example, if the simulation device is a driving simulator, it is conceivable that the building at the closest distance in front of the visual field of the virtual world is used as a reference (distance is, for example, several tens of meters). Assuming that the reference distance is D and the moving distance to the front of the field of view is d, the enlargement ratio M can be calculated using the following equation (3), for example.

上下や左右移動など、視線方向に垂直な方向の視野移動について補正を行う場合には、映像の上下または左右への平行移動により補正を行うことができる。この場合も、映像補正部３は、補正量は着目する基準距離を決めてから、平行移動の量を決定することができる。例えば、基準とする距離をD、視野の右方向への移動距離をd、視点から映像の投影面までの距離をhとすると、映像を投影面上で左方向に平行移動する量sは、例えば下記式（４）により計算することができる。左への移動や上下の移動についても同様に補正量を計算することができる。この場合、映像補正部３は、視野の移動方向と逆の方向にｓだけ映像を平行移動させる。

When correction is made for visual field movement in a direction perpendicular to the line-of-sight direction, such as up / down or left / right movement, the correction can be performed by parallel movement of the image up / down or left / right. Also in this case, the video correction unit 3 can determine the amount of translation after determining the reference distance to be focused on as the correction amount. For example, if the reference distance is D, the moving distance of the visual field in the right direction is d, and the distance from the viewpoint to the projection plane of the video is h, the amount s of moving the video in the left direction on the projection plane is For example, it can be calculated by the following equation (4). The correction amount can be calculated in the same manner for the leftward movement and the vertical movement. In this case, the image correction unit 3 translates the image by s in the direction opposite to the moving direction of the visual field.

以上、補正処理の具体例を説明したが、映像補正部３による補正処理は、上記例に限定されない。例えば、映像補正部３は、バッファメモリに記録された描画すべき視野の映像のうち、映像生成後に変化した視野に対応する領域のアドレスを特定し、この領域の映像を取り出して、表示する映像として出力するといった補正処理を実行することもできる。

The specific example of the correction process has been described above, but the correction process by the video correction unit 3 is not limited to the above example. For example, the video correction unit 3 specifies the address of the area corresponding to the visual field that has changed after the video generation from the video of the visual field to be drawn recorded in the buffer memory, extracts the video in this area, and displays the video It is also possible to execute a correction process such as outputting as

[Specific example of drawing field of view calculation]
As described above, when the approximate video at the time point T _N is generated by video correction, an undrawn area may occur after the video is moved. For example, when the range of the visual field at the time point T _N exceeds the visual field range to be drawn determined based on the visual field at the time point T _Nn , video data is not generated for the exceeding region, and thus the undrawn region It becomes. When the line-of-sight movement is large, this undrawn area also becomes large, which may cause another sense of incongruity. On the other hand, the drawing that is not displayed by the correction is wasted.

For example, FIG. 7A is a diagram illustrating a video example before and after correction when the visual field to be displayed at the time T _Nn of video generation is drawn as it is, and correction is performed based on the visual field to be displayed at the time T _N. In the example shown in FIG. 7A, the corrected image (lower image) is translated to the left in accordance with the change in the field of view, so the right side is the undrawn region P. Therefore, in the first and second embodiments, the drawing field calculation unit 1 is provided to make the range of the field of view to be drawn variable. For example, the drawing visual field calculation unit 1 calculates the visual field to be drawn at the time of video generation so that the visual field to be displayed at the time of video correction is included as much as possible. You can draw extra. As a result, even if the image is moved by the correction, an extra drawn portion newly enters the screen, so that an undrawn region is less likely to occur. In addition, it is possible to control so as not to draw a portion that may become unnecessary due to correction. Therefore, the waste of drawing processing can be reduced.

For example, FIG. 7B calculates a wider field of view to be drawn based on the field of view to be displayed at the time T _Nn of video generation, draws the range of the field of view, and based on the field of view to be displayed at the time T _N. It is a figure which shows the example of a video before and after correction | amendment at the time of correct | amending. In the example shown in FIG. 7B, the corrected image (lower image) is translated to the left according to the change in the visual field. However, since the drawing is performed in the range R2 wider than the original visual field range R1 so as to cover the parallel movement, no undrawn region is generated.

  At the timing for obtaining the visual field to be drawn in the drawing visual field calculation unit 1, the video correction time point corresponds to the future. Therefore, it is difficult to ensure that the visual field to be displayed at the time of future correction is included in the visual field to be surely drawn in any case. Therefore, based on the visual field movement tendency in the simulator, data for obtaining the visual field to be drawn based on the visual field to be displayed at the drawing start time can be determined in advance and recorded in the memory. For example, the extension direction from the visual field to be displayed at the start of drawing to the visual field to be drawn, and a parameter indicating the degree of expansion can be recorded in the memory in advance. Using this parameter, the drawing visual field calculation unit 1 can predict the visual field movement range according to the characteristics of the simulator and set an appropriate drawing range. For example, in a driving simulator, the field of view generally moves only in the horizontal direction, and the vertical movement of the field of view tends to be limited mainly to vehicle shake. Considering this tendency, the visual field to be drawn can be obtained by extending the visual field to be displayed mainly in the horizontal direction. For example, the drawing visual field calculation unit 1 may perform the expansion by increasing the angle of view in the horizontal direction by a certain amount, or may extend the drawing range on the screen by a certain number of pixels in the horizontal direction. The angle of view and the extension direction and extension amount of the drawing range can be recorded in advance. The expansion amount recorded in advance can be calculated, for example, by estimating the maximum value of the lateral movement of the visual field from the calculation result of the simulation unit 5.

(Prediction of field of view)
In order to reduce the waste of drawing, it is desirable to predict in which direction the visual field to be displayed moves and to determine the visual field to be drawn so that it is included and is as narrow as possible. Therefore, the drawing visual field calculation unit 1 can predict the time at which video correction is performed and the visual field to be displayed at the video correction time by the following processing, for example.

  For example, a history of time taken for video generation in the drawing units 22a, 22b, and 22c is recorded, and the drawing visual field calculation unit 1 performs prediction calculation using them to obtain a predicted value of time required for video generation. Can be calculated. The time at which the next video correction is performed can be obtained by adding the time required for video compression / decompression and transmission / reception and the current time to the value obtained by the prediction calculation. As the prediction calculation, for example, there is a method of applying a linear function or a polynomial function to historical data of the time taken for drawing. Alternatively, an average value of time taken for drawing can be calculated as a predicted value.

  For the field of view to be displayed at the time of video correction, record the history of the field of view to be displayed, perform prediction calculation using those history, and obtain the field of view at the predicted time of the video correction obtained above. it can. The field of view to be displayed can be defined by a combination of elements such as a viewpoint position, a line-of-sight vector, a field angle, and a projection parameter. Therefore, time-series data of element values that determine the visual field to be displayed can be recorded as a history. As the prediction calculation, for example, there is a method of applying a linear function or a polynomial function to the history. In addition, the average value of each element value that determines the field of view to be displayed can be calculated as a predicted value. In this way, the visual field can be represented by a combination of elements such as a viewpoint position, a line-of-sight vector, and a visual field angle, so that a predicted value can be calculated individually for each element.

  Prediction with high accuracy can be calculated based on changes in the visual field in a span that is long to some extent, and it is difficult to accurately predict a quick visual field movement in a short time. Therefore, for example, it is possible to follow a rough movement in units of several seconds by predicting the visual field by the drawing visual field calculation unit 1 and set a visual field to be drawn with less waste. A quick visual field movement of about 1 second or less can be responded with good response by setting the visual field to be drawn by the drawing visual field calculation unit so that the image can be corrected within the visual field to be drawn. .

  For example, the drawing visual field calculation unit 1 has a range obtained by adding a visual field predicted and calculated based on a history of visual fields to be displayed in the past to a visual field obtained by extending the visual field to be displayed at the time of video generation by a certain amount in all directions. The field of view can be calculated as the field of view to be drawn. In this way, by expanding the visual field to be displayed by a certain amount in all directions, it is possible to cover the movement of the visual field in a short time and cover the field of view for a relatively long time within the range of the visual field predicted from the history. Can cover the movement.

  The correction process and the drawing visual field calculation process described above can also be applied to the first embodiment.

(Division into parts to be corrected and parts not to be performed)
When the visual field to be displayed includes a plurality of objects having greatly different distances from the viewpoint position, when the video correction unit 3 corrects a video including the plurality of objects at once, a plurality of object images in the video are displayed. Will be moved by the same amount. As a result, a sense of incongruity may increase. For example, in a driving simulator, if the field of view to be displayed includes a part that moves differently from the outside world, such as the body of the vehicle (including the interior), the image of the body and The interior moves with the outside world. As a result, a sense of incongruity may increase.

  Therefore, a video generation unit that generates video of some objects in the virtual world included in the field of view to be displayed, and a second that generates video of other objects that are different in distance from the viewpoint from the some objects. And a video generation unit. Then, after correcting the video generated by the video generation unit, it can be combined with the video generated by the second generation unit. As a result, even an image including an object having a different distance from the viewpoint can be corrected without increasing a sense of incongruity. In the present embodiment, as an example, the server 12 includes the video generation unit 2 and the client 13 includes the local drawing unit 11a which is an example of the second video generation unit.

  For example, in the above driving simulator, it is possible to draw the outside world and the vehicle body separately and synthesize both after performing correction. FIG. 8 is a diagram illustrating an example of an image that is synthesized after a plurality of objects in the virtual world are drawn separately. In the example shown in FIG. 8, a frame P2 drawn with the vehicle body object as the foreground Z and a frame P1 drawn with the outside building as the foreground K and the mountain as the background (background) E are generated independently. The frame P1 is corrected by the video correction unit 3 and then combined with the frame P2. As a result, a frame P3 is generated.

  In the case of the example shown in FIG. 8, the outside world that is the subject of the virtual world can be drawn by the drawing unit 22 on the server 12 side, and the vehicle body with a small drawing load can be drawn by the local drawing unit 11 a on the client 13 side. In this case, the video on the server 12 side is delayed and needs to be corrected. However, the video on the client 13 side needs to be corrected because the time required for video generation is short and the influence of delay due to video transmission is small. .

  In the present embodiment, the client 13 is provided with a local drawing unit 11a. Also, for example, local drawing data including model data and texture data for drawing the vehicle body is recorded in the client 13. The local drawing data may be recorded in advance on the client 13 side, or may be downloaded from the server 12 as much as necessary after connecting to the server 12. The object information (position, posture, attribute, etc.) updated by the simulation unit 5 is sent to the video generation unit 2 of the server 12 and the local drawing unit 11a of the client 13. As an example, information relating to the outside world for generating near and far view images is sent to the image generating unit 2, and information for drawing the vehicle body and interior is sent to the local drawing unit 11a. The update information sent to the local drawing unit 11a is sent through the update information transmitting unit 15 and the update information receiving unit 25. The update information can be, for example, data indicating an updated attribute among the attributes of the drawing target object of the local drawing unit 11a. The data size of the update information is preferably small so that the transmission / reception delay time is negligible compared to the video information.

  The local drawing unit 11a transmits the latest visual field to be displayed sent from the simulation unit 5 via the visual field transmission unit 16 and the visual field reception unit 26, and the transmission information via the update information transmission unit 15 and the update information reception unit 25. Video generation is performed using the local drawing data. Then, the combining unit 9 combines the image corrected by the image correcting unit 3 with the image generated by the local drawing unit 11a. The local drawing unit 11a makes one area by superimposing the divided drawing results by keeping the area of the video in which the object in charge of drawing is not shown (in the example of FIG. 8, the area other than the vehicle body) transparent. Can be combined with video. In the example of FIG. 8, the vehicle body and the external world that is visible beyond the vehicle body are correctly combined.

  As in the example of dividing the vehicle body and the outside shown in FIG. 8, when the objects are separated so as not to cross each other, the composition of the video by the composition unit 9 may be simple superposition. However, if there is an overlap in the depth direction, such as when there is an object riding on the vehicle body, there may be cases where the composition cannot be performed correctly by simple composition. In that case, the synthesizing unit 9 can correctly synthesize by utilizing the contents of the Z buffer used when drawing each video. The Z buffer is a memory area for recording information in the depth direction of video. For example, the drawing result receiving unit 28 and the update information receiving unit 25 receive the contents of the Z buffer attached to each video, and the synthesizing unit 9 compares the values of the Z buffer of each pixel and is in front. It is possible to perform superposition synthesis while adopting the color of each pixel. For the video drawn on the server 12 side, the same correction as the video itself can be performed on the contents of the Z buffer before the composition. As a result, even when the positional relationship in the depth direction cannot be correctly displayed only by simple superposition, the video can be appropriately synthesized.

  Note that the example of the second video generation process and the synthesis process in the present embodiment can also be applied to the first embodiment.

[Concealment of delay]
FIG. 9 is a process flow diagram illustrating an operation example of the simulation system illustrated in FIG. 3. In the example illustrated in FIG. 9, the simulation unit 5 updates the virtual world object data at the time points T _N−4 , T _N−3 , T _N−2 , T _N−1 , and T _N when the operation input is performed. Update the field of view to be displayed. In addition, the simulation part 5 may perform an update process with a fixed period separately from the timing of operation input.

For example, in the update process (Op1) at time T _N-4 , the simulation unit 5 notifies the drawing field calculation unit 1 of the updated field of view to be displayed. When receiving the visual field to be displayed from the simulation unit 5, the drawing visual field calculation unit 1 starts calculating the visual field to be drawn (Op2). The visual field to be drawn calculated by the drawing visual field calculation unit 1 is transferred to the drawing unit 22a, and the drawing unit 22a generates an image of the visual field to be drawn (Op3). The generated video undergoes compression by the video encoder 17, transmission to the client 13 by the drawing result transmission unit 18, reception by the drawing result reception unit 28 of the client 13, and decompression by the video decoder 27 (Op 4), and then the video correction unit 3. Passed to. Upon receiving the video, the video correction unit 3 receives the latest visual field to be displayed updated at the time T _N by the simulation unit 5 in Op5. The video correction unit 3 corrects the video so that the latest visual field range to be displayed is displayed (Op6).

In the process shown in FIG. 9, a delay is actually generated by a time TD1 from the start of the update process of the simulation unit 5 at time T _N-4 to the completion of correction by the video correction unit 3. However, the video corrected by the video correction unit 3 is close to the visual field image to be displayed updated at time T _N. For this reason, the delay felt by the user who has viewed the corrected video, that is, the apparent delay, is the time TD2 from the time T _N to the completion of the correction.

  As described above, in this embodiment, even if there is a certain delay in the generation and transfer of the video, it is possible to make it appear that the video is responding to the user's operation input without delay. Therefore, it is possible to greatly reduce the uncomfortable feeling caused by the delay of the video. Thereby, for example, it is possible to construct a system in which video is generated by a large-scale server that performs parallel drawing and the video is used in real time from the client via the network. In other words, in a simulator that can generate video using high quality or a large amount of data, but delay is inevitable, it can be used over a network with real time and high response, which has been difficult until now.

  For example, it is possible to collect high rendering performance and a large amount of data on the server computer side by adopting a configuration in which video generation of the virtual world is performed by the server computer and the resulting video is transferred to the client computer through the network. Become. As a result, the performance required for the client computer is reduced, and it is easy in terms of cost and labor to deploy the simulator to a large number of users. If it is an online 3D game, even if the computer provided by each player is inexpensive and has a standard performance, it will be possible to enjoy high-quality video that exceeds that performance.

  In such a simulator that can be used via a network, if the video drawn by the server computer is transmitted through the network while being compressed / decompressed by a codec to prevent a decrease in the frame rate, a delay in the video with respect to the user's operation is avoided. I can't. According to the present embodiment, this delay can be concealed, and the uncomfortable feeling of the output video can be reduced. Therefore, an application that requires a quick response to an operation can be used via the network. For online 3D games, it is possible to realize a configuration in which high-quality drawing is performed by a server computer and video is displayed by a client.

[Computer configuration]
The server 12 and the client 13 of the simulation system shown in FIG. 3 can be configured by independent computers. Each computer may be a general-purpose computer or a dedicated machine as in the first embodiment. In addition to the CPU, the computer may include a processor dedicated to image processing such as a DSP. For example, the drawing units 22a, 22b, and 22c in the video generation unit 2 of the server 12 can each be configured by three independent processors.

  Further, the simulation unit 5, the drawing field calculation unit 1, and the image generation unit 2, the image generated by the image generation unit 2, the field of view to be displayed which is the basis of the field of view of the image to be drawn, An embodiment of the present invention also includes a simulated video generation apparatus including a transmission unit that transmits a new field of view to be displayed calculated after the start of video generation to a client device having the video correction unit 3.

  Furthermore, from the server device having the simulation unit 5, a visual field to be drawn which is a visual field wider than the visual field to be displayed at the time of video generation, a video of the visual field to be drawn generated based on data representing a virtual world, A receiving unit that receives a new field of view to be displayed calculated after generation of the video, a video correction unit 3, and a display unit 4 that displays the video corrected by the video correction unit 3 (an example of an output unit) A simulated video display device including the above is also included in the embodiment of the present invention.

DESCRIPTION OF SYMBOLS 1 Drawing visual field calculation part 2 Video generation part 3 Video correction part 4 Display part 5 Simulation part 6 Input part 7 Drawing visual field holding part 8 Display visual field holding part 9 Synthesis | combination part 10, 10a Simulation apparatus 11 2nd video generation part 12 Simulator server (Server device)
13 Simulator client (client device)

Claims (11)

A simulation unit that updates data representing a virtual world according to an operation input from a user or the passage of time, and calculates a visual field to be displayed in the virtual world;
A drawing field calculation unit that calculates a field of view to be drawn which is a field of view wider than the first field of view calculated by the simulation unit at a first time;
A video generation unit that starts generating the first video corresponding to the visual field to be drawn immediately after the first time based on the data representing the virtual world;
Using the second field of view to be displayed calculated by the simulation unit at a second time after the generation of the first video, the first video is displayed immediately after the second time. A video correction unit that corrects the second video corresponding to the field of view to be displayed,
The drawing visual field calculation unit predicts and calculates the second visual field to be displayed at the second time by a predetermined method at the first time, and the visual field to be drawn is the first display. as the inclusive vision field to should do field and predicted the second display field to be simulated image generating apparatus, characterized in that the calculation of the visual field to be the drawing.   The drawing visual field calculation unit calculates a range that can be displayed after starting the generation of the first video based on the history of the visual field to be displayed calculated by the simulation unit, and includes a visual field including the range The simulated video generation apparatus according to claim 1, wherein the visual field to be drawn is set as the visual field to be drawn. A second video generation unit that generates a third video showing a part of the virtual world in the new field of view to be displayed based on the new field of view to be displayed calculated after the generation of the first video is started. When,
The said 2nd image | video corrected by the said image | video correction | amendment part and the synthetic | combination part which synthesize | combines the said 3rd image | video produced | generated by the said 2nd image | video production | generation part are further provided. Simulated video generation device.   The drawing visual field calculation unit calculates a predicted value of time taken from the start of generation of the first video to the start of correction of the first video using a history of time taken to generate the first video. The simulated video generation apparatus according to claim 1, wherein the range of the visual field to be drawn is adjusted according to the predicted value.   The video correction unit corrects, as a video to be displayed, a video of a region corresponding to the second visual field to be displayed calculated after the start of generation of the video among the video of the visual field to be drawn. The simulated video production | generation apparatus of any one of Claims 1-4.   The video correction unit is configured to change a change from the first visual field to be displayed that is a basis of the visual field to be drawn of the video to the second visual field to be displayed that is calculated after the generation of the video is started. The simulated video generation apparatus according to claim 1, wherein the display position of the video of the visual field to be drawn is moved according to the amount and direction. A simulation system including a server device and a client device,
The server device
A simulation unit that updates data representing a virtual world according to an operation input from a user or the passage of time, and calculates a visual field to be displayed in the virtual world;
A drawing field calculation unit that calculates a field of view to be drawn which is a field of view wider than the first field of view calculated by the simulation unit at a first time;
A video generation unit that starts generating the first video corresponding to the visual field to be drawn immediately after the first time based on the data representing the virtual world;
The first video, the first visual field to be displayed that is the basis of the visual field to be drawn of the first video, and the simulation at a second time after the start of the generation of the first video A transmission unit that transmits the second field of view to be displayed calculated by the unit to the client device immediately after the second time,
The client device is
Using the received second field of view to be displayed calculated after the start of generation of the image, the received first image is corrected to a second image corresponding to the received second field of view to be displayed. A video correction unit that
The drawing visual field calculation unit predicts and calculates the second visual field to be displayed at the second time by a predetermined method at the first time, and the visual field to be drawn is the first display. as the inclusive vision field to should do field and predicted the second display field to be a simulation system, characterized in that the calculation of the to be rendered field. A simulation unit that updates data representing a virtual world according to an operation input from a user or the passage of time, and calculates a visual field to be displayed in the virtual world;
A drawing field calculation unit that calculates a field of view to be drawn which is a field of view wider than the first field of view calculated by the simulation unit at a first time;
A video generation unit that starts generating the first video corresponding to the visual field to be drawn immediately after the first time based on the data representing the virtual world;
The first video, the first visual field to be displayed that is the basis of the visual field to be drawn of the first video, and the simulation at a second time after the start of the generation of the first video To the client device having a video correction unit that corrects the first visual field calculated by the unit to a second video corresponding to the second visual field to be displayed. A transmission unit that transmits immediately after the time of 2,
The drawing visual field calculation unit predicts and calculates the second visual field to be displayed at the second time by a predetermined method at the first time, and the visual field to be drawn is the first display. as the inclusive vision field to should do field and predicted the second display field to be simulated image generating apparatus, characterized in that the calculation of the visual field to be the drawing. Calculated by the simulation unit at a first time from a server device provided with a simulation unit that updates data representing the virtual world and calculates the visual field to be displayed in the virtual world according to an operation input from the user or the passage of time. A visual field to be drawn which is a wider visual field than the first visual field to be displayed, a first video corresponding to the visual field to be drawn generated based on data representing the virtual world, and the first A receiving unit that receives the second field of view to be displayed calculated by the simulation unit at a second time after the start of video generation;
Using the received second field of view to be displayed calculated at the second time after the start of generation of the first image, the first image corresponding to the field of view to be drawn is the second A video correction unit that corrects the second video corresponding to the field of view to be displayed;
An output unit for displaying the corrected video,
The receiving unit receives the second field of view to be displayed, which is predicted by the simulation unit at a first method at the first time immediately after the second time,
The image correction unit, the first of said calculated for the first time by the simulation unit such that the inclusive vision field of vision to be displayed and the predicted second display field of view to be A simulated video display device, wherein the first video is corrected to the second video using the received second visual field to be displayed based on the visual field to be drawn. A simulation process for updating the data representing the virtual world according to the operation input from the user or the passage of time, and calculating the visual field to be displayed in the virtual world,
A drawing visual field calculation process for calculating a visual field to be drawn which is a visual field wider than the first visual field to be displayed calculated by the simulation process at a first time;
Video generation processing for starting generation of the first video corresponding to the visual field to be drawn immediately after the first time based on the data representing the virtual world;
Using the second field of view to be displayed calculated by the simulation process at a second time after the start of generation of the first video, the first video is displayed immediately after the second time. And a video correction process for correcting the second video corresponding to the field of view to be displayed.
The drawing visual field calculation process predicts and calculates the second visual field to be displayed at the second time by a predetermined method at the first time, and the visual field to be drawn is the first display. as will be inclusive vision field to should do field and predicted the second display field to be simulated image generation program, characterized in that the calculation of the visual field to be the drawing. A simulation process in which a processor of a computer updates data representing a virtual world in response to an operation input from a user or the passage of time, and calculates a visual field to be displayed in the virtual world;
A drawing visual field calculation step in which the processor calculates a visual field to be drawn which is a visual field wider than the first visual field to be displayed calculated by the simulation step at a first time;
A video generation step in which a processor starts generating a first video corresponding to the visual field to be drawn immediately after the first time based on data representing the virtual world;
The processor uses the second field of view to be displayed calculated by the simulation process at a second time after the start of the generation of the first video, and immediately after the second time, the first video And a video correction step of correcting the second video corresponding to the field of view to be displayed,
The drawing visual field calculation step predicts and calculates the second visual field to be displayed at the second time by a predetermined method at the first time, and the visual field to be drawn is the first display. as the inclusive vision field to should do field and predicted the second display field to be simulated image generating method characterized in that the calculation of the to be rendered field.

Images