WO2016190193A1 - Information processing apparatus, output control apparatus, information processing system, and video data output method - Google Patents

Abstract

The present invention connects output control apparatuses 20a and 20b serially to an information processing apparatus 10 so that a display apparatus such as an HMD 18a, an HMD 18b, and a flat-type display 16 can be connected to each of the output control apparatuses 20a and 20b. The information processing apparatus 10 determines the connection states thereof and outputs a composited image 124 generated by reducing and connecting the images to be displayed on the HMDs 18a and 18b, respectively. Each of the output control apparatuses 20a and 20b retrieves the composited image, extracts areas corresponding to the display apparatus connected thereto, and generates and outputs final display images 126, 128, and 130, thereby causing them to be displayed.

Description

Information processing apparatus, output control apparatus, information processing system, and moving image data output method

The present invention relates to a technique for transmitting and processing image data associated with display of a captured image and a drawn image.

A head mounted display (hereinafter referred to as “HMD”) that displays an image in front of the user when the user wears it on the head has been put into practical use. In recent years, an HMD that presents a stereoscopic image by displaying parallax images on the left and right eyes of the user, and a sense of immersion in a virtual space by tracking the movement of the user's head and reflecting the result in the image, HMDs that give a sense of reality have been proposed (see, for example, Patent Document 1).

JP 2014-93704 A

Unlike the flat display, the HMD is basically based on the enjoyment of content such as video that is displayed only to the user wearing it. When displaying a virtual space with the viewpoint changed according to the movement of the head, the user's personal movement interacts with the video, so the HMD is used for personal play and video content viewing. The form tended to be limited. Therefore, it is desired to realize a content providing technology that allows users to share experiences with other users while wearing an HMD and enjoying an immersive feeling.

The present invention has been made in view of these problems, and an object of the present invention is to provide a technology that allows a plurality of people to enjoy content including image display on the HMD.

An aspect of the present invention relates to an information processing apparatus. The information processing apparatus includes an output data generation unit that generates moving image data to be displayed on the display device, and a communication unit that sequentially outputs moving image data generated by the output data generation unit. When there are a plurality of connections, the output data generation unit causes the communication unit to output combined image data obtained by combining a plurality of moving images to be displayed on each display device for each frame.

Another aspect of the present invention relates to an output control device. The output control device includes an input unit that acquires display moving image data output from the information processing device, and a plurality of moving images to be displayed independently on a plurality of display devices in the display moving image data for each frame. When the synthesized composite image data is included, a partial area to be displayed on the display device connected to itself is extracted from the synthesized image data, and the final display image is output data forming unit. And an output unit for outputting display image data to a connected display device.

Still another aspect of the present invention relates to an information processing system. This information processing system is an information processing system comprising: an information processing device that outputs moving image data; and an output control device that acquires the moving image data and causes the display device to display the information. Includes an output data generation unit that generates moving image data to be displayed on the display device, and a communication unit that sequentially outputs the moving image data generated by the output data generation unit to the output control device. When there are a plurality of connections, the output data generation unit causes the communication unit to transmit combined image data obtained by combining a plurality of moving images to be displayed independently on each display device for each frame. When the composite image data is included in the input unit for acquiring the image data and the moving image data, a partial area to be displayed on the display device connected to the mobile device is extracted from the composite image data. Characterized in that it comprises an output data forming section to specific display image, and an output unit for outputting the data of the final displayed image to the connected display device.

Still another aspect of the present invention relates to a moving image data output method. The moving image data output method includes a step of generating moving image data to be displayed on a display device and a step of sequentially outputting the generated moving image data. The step of generating moving image data includes: When a plurality of devices are connected, the method includes a step of generating combined image data formed by combining a plurality of moving images to be displayed independently on each display device for each frame.

Still another aspect of the present invention also relates to a moving image data output method. A step of acquiring display moving image data output by the information processing device; and a composite image data in which a plurality of moving images to be displayed independently on a plurality of display devices are combined for each frame with the display moving image data A partial region to be displayed on a display device connected to the display device is extracted from the composite image data to form a final display image, and the final display image data is connected to the display device. And a step of outputting the data.

Note that any combination of the above-described components, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded, and the like are also effective as an aspect of the present invention. .

According to the present invention, contents including image display on the HMD can be diversified.

It is a figure which shows the structural example of the information processing system which can apply this Embodiment. It is a figure which shows the example of the external appearance shape of HMD in this Embodiment. It is a figure which shows the internal circuit structure of the information processing apparatus in this Embodiment. It is a figure for demonstrating the image displayed on HMD and a flat type display in this Embodiment. It is a figure which shows the structure of the functional block of the information processing apparatus and output control apparatus in this Embodiment. It is a figure for demonstrating the data of the composite image which an information processing apparatus outputs in this Embodiment. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the structural example of the image data transmitted between the connection state of each apparatus in the information processing system of this Embodiment, and the space | interval. It is a figure which shows the data structure example of the corresponding | compatible information of the parameter used for the connection state and process of an apparatus which an information processing apparatus manages in this Embodiment. It is a flowchart which shows the process sequence which outputs the data of the image which the information processing apparatus in this Embodiment displays on each display apparatus. It is a flowchart which shows the process sequence which the output control apparatus in this Embodiment displays an image on each display apparatus. It is a figure for demonstrating the synthesized image which connected the display image to the vertical direction in this Embodiment. It is a figure for demonstrating the modification of the transmission aspect of the image data displayed on several HMD in this Embodiment. It is a figure for demonstrating the modification of the transmission aspect of the image data displayed on several HMD in this Embodiment. In this Embodiment, it is a figure for demonstrating the method of embedding the correspondence information of the area | region in an image, and HMD in the image which an information processing apparatus outputs.

FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied. In this example, the information processing system 8 includes an imaging device 12 that captures an object, an information processing device 10 that performs information processing based on the captured image, a flat panel display 16 that displays an image obtained as a result of information processing, and each HMDs 18a and 18b worn by users and input devices 14a and 14b operated by the users are included.

The information processing device 10 and the imaging device 12, the input devices 14a and 14b, the flat display 16, and the HMDs 18a and 18b may be connected by a cable or a known wireless communication technology such as Bluetooth (registered trademark). Good. Further, depending on the information processing performed by the information processing device 10, the imaging device 12, the flat display 16, and the input devices 14a and 14b may be omitted. Further, the external shapes of these devices are not limited to those shown in the drawings. Further, the number of users wearing the HMD 18a and the like is not limited to two, and may be one or three or more.

The imaging apparatus 12 generates output data of a captured image by performing general processing such as demosaic processing on a camera that captures an object such as a user at a predetermined frame rate, and outputs the output data to the information processing apparatus 10. A delivery mechanism. The camera includes a visible light sensor used in general digital cameras and digital video cameras, such as a CCD (Charge-Coupled Device) sensor and a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The imaging device 12 may include only one camera, or may be a so-called stereo camera in which two cameras are arranged on the left and right sides at a known interval as illustrated.

Alternatively, the imaging device 12 may be configured by a combination of a device that irradiates an object with reference light such as infrared rays and measures the reflected light and a monocular camera. When a stereo camera or a reflected light measurement mechanism is introduced, the position of a subject in a three-dimensional real space can be obtained, and information processing and display images can be further diversified. Using stereo images taken by the stereo camera from the left and right viewpoints, the subject is determined by the triangulation principle, the subject's distance from the camera, and the reflected light is measured by TOF (Time of Flight) or pattern irradiation. Any method for specifying the distance from the camera is widely known.

The information processing apparatus 10 outputs image data to be displayed on the HMDs 18 a and 18 b and the flat display 16. The data of the image may be data saved as a complete form such as a movie or a captured moving image, or may be drawn in real time inside the information processing apparatus 10. In the latter case, the information processing apparatus 10 performs, for example, general face detection and tracking processing on a captured image acquired from the imaging apparatus 12 at a predetermined frame rate, thereby reflecting a user's action as a subject. Progress the game in which appears, or convert the user's movement into a command input to perform information processing.

At this time, the information processing apparatus 10 acquires each user's movement using markers and acceleration sensors provided in the HMDs 18a and 18b and the input devices 14a and 14b, and acquires user operation information for the input devices 14a and 14b. Alternatively, a different image may be generated for each user. For example, by tracking the image of the marker of the HMD 18a, 18b in the captured image, the position and movement of each user's head is specified. And by changing the viewpoint with respect to the virtual world displayed on HMD18a, 18b corresponding to the movement, each user can see a virtual world with an own viewpoint, and can obtain a sense of presence and immersion. The information processing apparatus 10 may output audio data such as BGM in addition to such image data.

The HMDs 18a and 18b are display devices that display an image on a display panel such as an organic EL panel positioned in front of the user when the user wears the head. For example, a parallax image viewed from the left and right viewpoints is generated and displayed in the left and right areas obtained by dividing the display screen into two, thereby allowing the image to be viewed stereoscopically. However, the present embodiment is not limited to this, and one image may be displayed on the entire display screen. The HMDs 18a and 18b further include an element that emits light of a predetermined color or a light emitting marker composed of a set thereof, a speaker or earphone that outputs sound at a position corresponding to the user's ear, and an acceleration that detects the tilt of the head of the user wearing the device. It may be equipped with a sensor.

The flat display 16 may be a television having a display that outputs a two-dimensional image and a speaker that outputs sound, and specifically, any of a liquid crystal television, an organic EL television, a plasma television, a PC display, and the like. Or the display and speaker of a tablet terminal or a portable terminal may be sufficient. The input devices 14 a and 14 b are operated by the user to accept requests for processing start and end, function selection, various command inputs, and the like, and supply them to the information processing apparatus 10 as electrical signals.

The input devices 14a and 14b may be realized by any one of general input devices such as a game controller, a keyboard, a mouse, a joystick, a touch pad provided on the display screen of the flat display 16, or a combination thereof. The input devices 14a and 14b may further include a light emitting marker composed of an element that emits light of a predetermined color or a set thereof. In this case, when the information processing apparatus 10 tracks the movement of the marker using the captured image, the movement itself of the input devices 14a and 14b can be a user operation. In addition, you may comprise the input devices 14a and 14b only with the light emission marker which has a holding part.

Thus, in this embodiment, output data can be transmitted from one information processing apparatus 10 to a plurality of HMDs 18a and 18b. Thus, for example, it is possible to realize a mode in which a plurality of users participate in one game and operate the input devices 14a and 14b while watching the game situation from their respective viewpoints. In addition, by displaying either or both of the images displayed on the HMDs 18a and 18b on the flat panel display 16, a user as a gallery can watch the game.

At this time, the data output by the information processing apparatus 10 is first transmitted to the HMD 18a, and then transmitted from the HMD 18a to the HMD 18b, as indicated by the arrows. When the number of users increases, transmission is performed sequentially between the HMDs. By doing in this way, the number of users wearing HMD can be increased without increasing the number of communication terminals of each device. A flat panel display 16 can be included as one such chain of HMDs. In the following description, the HMDs 18a and 18b may be collectively referred to as the HMD 18. In addition, as described above, the information processing apparatus 10 can output audio data to each display device, but the following description will be made with the focus on image data.

FIG. 2 shows an example of the external shape of the HMD 18. In this example, the HMD 18 includes an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that goes around the head when the user wears to fix the device. The wearing band 106 is made of a material or a structure whose length can be adjusted according to the head circumference of each user. For example, an elastic body such as rubber may be used, or a buckle, a gear, or the like may be used.

The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the HMD 18 and includes a display panel inside so as to face the eyes when worn. The display panel is realized by a liquid crystal panel or an organic EL panel. The housing 108 may further include a pair of lenses that are positioned between the display panel and the user's eyes when the HMD 18 is mounted and that enlarge the viewing angle of the user. The HMD 18 may further include a speaker or an earphone at a position corresponding to the user's ear when worn.

On the outer surface of the housing 108, light emitting markers 110a, 110b, 110c, and 110d are provided. The number and arrangement of the light emitting markers are not particularly limited. In the illustrated example, the light emitting markers are provided at the four corners on the front surface of the casing of the output mechanism unit 102. Further, it may be provided on both side surfaces behind the mounting band 106. Since the light emitting markers 110c and 110d are below the output mechanism unit 102 and are not originally visible from the viewpoint of FIG. 2, the outer periphery is represented by dotted lines.

As shown in the figure, the HMD 18 realizes display and audio output by being connected to an output control device 20 that relays data output from the information processing device 10. The output control device 20 basically includes one input terminal 5 and two output terminals 7a and 7b. The input terminal 5 is an interface that acquires data output from the information processing apparatus 10 directly or indirectly. The output terminals 7 a and 7 b are interfaces for outputting data to the HMD 18, the flat display 16, or another output control device 20. However, this is not intended to limit the number of input terminals and output terminals and the installation location.

FIG. 3 shows an internal circuit configuration of the information processing apparatus 10. The information processing apparatus 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These units are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a peripheral device interface such as USB or IEEE1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a nonvolatile memory, and an output for outputting data to an external device. A unit 36, an input unit 38 for inputting data from an external device, and a recording medium driving unit 40 for driving a removable recording medium such as a magnetic disk, an optical disk or a semiconductor memory are connected.

The CPU 22 controls the entire information processing apparatus 10 by executing an operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium and loaded into the main memory 26 or downloaded via the communication unit 32. The GPU 24 has a function of a geometry engine and a function of a rendering processor, performs a drawing process according to a drawing command from the CPU 22, and outputs data from the output unit 36 to the output control device 20.

The main memory 26 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing. The output control device 20 may basically have the same configuration. However, depending on the application to be executed and the design of the apparatus, the information processing apparatus 10 may perform almost all processes, and the output control apparatus 20 may be sufficient with only simple processes. In this case, the output control device 20 can omit a part of the illustrated configuration.

FIG. 4 is a diagram for explaining images displayed on the HMD 18 and the flat display 16. The left image (a) is an image displayed on the HMD 18, and the right image (b) is an image displayed on the flat display 16. In the figure, for the sake of simplicity, a grid-like pattern is the display target, but in reality, various images are displayed depending on the contents of information processing such as a virtual world and a game image. The image (a) for the HMD 18 assumes that an image is stereoscopically viewed, and a pair of left-eye images and right-eye images is formed in left and right regions obtained by dividing an image plane corresponding to a display panel into two. The parallax images are arranged respectively.

This example assumes that a lens is provided in front of the display panel in the HMD 18 to widen the user's field of view. In this case, a so-called “pincushion” distortion is generated in the image displayed on the display panel, which is stretched by the lens with a larger displacement as the pixel is further away from the center. Since the amount of displacement varies depending on the color, chromatic aberration also occurs. Accordingly, distortion correction is performed in which the distortion and the chromatic aberration are taken into consideration in the opposite direction, and a “barrel-shaped” image like the image (a) is displayed. Thus, an original image such as the image (b) can be seen stereoscopically when viewed through the lens.

Such correction processing may be performed by the information processing apparatus 10 or the output control apparatus 20. In the former case, the output control device 20 that has received the data of the image (a) outputs the data to the HMD 18 as it is, and the flat display 16 displays either the left eye or the right eye of the image (a). An image (b) is generated by performing the reverse correction to the above-described distortion correction and output.

In the latter case, the output control device 20 performs distortion correction on the uncorrected parallax image transmitted from the information processing device 10 to generate an image (a) and outputs the image (a) to the HMD 18. One of the parallax images, that is, the image (b) is output as it is. In the following description, the former mode is assumed, but the latter mode can be similarly applied. A general method can be applied to distortion correction for a lens and vice versa.

FIG. 5 shows a configuration of functional blocks of the information processing apparatus 10 and an output control apparatus (referred to as an output control apparatus 20a) that outputs image data to a certain HMD 18a. As described above, in the present embodiment, the output control device 20 is sequentially connected to the information processing device 10 to transmit output data, and an appropriate format is transmitted to the HMD 18 or the flat display 16 connected to each output control device 20. To output the image. FIG. 5 shows the configuration of the information processing apparatus 10 and the output control apparatus 20a directly connected thereto, but the other output control apparatuses 20 have the same configuration. In the following description, the side closer to the information processing device 10 in the array of the output control devices 20 connected in this way is expressed as a higher level, and the far side is expressed as a lower level.

Each functional block shown in FIG. 5 can be realized in terms of hardware by the CPU, GPU, various memories shown in FIG. 3, and in terms of software, the data input function and data loaded from the recording medium into the memory. It is realized by a program that exhibits various functions such as a holding function, an image processing function, and a communication function. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

The information processing device 10 includes an input data acquisition unit 72 that acquires input data from the imaging device 12 and the input devices 14a and 14b, an information processing unit 74 that performs information processing according to an application to be executed such as a game, a game image and its sound The output data generation unit 78 that generates data to be output as a result of information processing, the communication unit 76 that communicates with the highest-level output control device 20a, and the connection status specification that specifies the connection status of various devices including the output control device 20a Part 80 and a data storage part 81 for storing basic data necessary for generating output data.

The input data acquisition unit 72 sequentially acquires images taken by the imaging device 12 at a predetermined frame rate. Also, information indicating the contents of user operations via the input devices 14a and 14b is acquired. Here, the user operation may be performed by general information processing such as selection of an application to be executed, start / end of processing, and command input. Then, the input data acquisition unit 72 supplies the acquired information to the information processing unit 74.

The input data acquisition unit 72 may further acquire the result measured by the acceleration sensor in each HMD 18 from the communication unit 76 and supply the result to the information processing unit 74. The input data acquisition unit 72 also controls the start / end of shooting in the imaging device 12 according to the processing start / end request from the user acquired from the input devices 14a and 14b, or according to the processing result in the information processing unit 74. Thus, the type of data acquired from the imaging device 12 may be controlled. However, the data acquired by the input data acquisition unit 72 may differ depending on the content of information processing executed by the information processing unit 74.

The information processing unit 74 performs information processing such as a game designated by the user. This information processing may include a process of detecting an image of an object from a captured image and tracking it. For example, a part of a user's body such as a head or a hand may be tracked using a contour line, or an object having a specific pattern or shape may be tracked by pattern matching. A general image processing technique can be applied to these tracking processes. Further, the position and posture of the user's head may be specified in detail by integrating information such as measurement values transmitted from the HMDs 18 by the acceleration sensor. The information processing unit 74 uses the movement and position of the user thus obtained as an input value, or uses a user operation via the input devices 14a and 14b as an input value to progress the game or perform a corresponding process. Or do it. However, the contents of information processing are not limited to these.

The output data generation unit 78 generates image and audio data to be output as a result of information processing in accordance with a request from the information processing unit 74. For example, as described above, the virtual world viewed from the viewpoint corresponding to the position and posture of the user's head is rendered as a left and right parallax image, and distortion correction considering the lens is performed and the image data arranged on the left and right Is generated. If this image is displayed in front of the left and right eyes at a predetermined frame rate on the HMD 18 and the sound in the virtual world is further output, the user can feel as if he / she entered the virtual world.

The communication unit 76 establishes communication with the output control device 20a using a predetermined protocol such as HDMI (registered trademark) (High-Definition Multimedia Interface) or USB (Universal Serial Bus), and the image and sound generated by the output data generation unit 78. Send the output data. Further, the communication unit 76 detects a device connected to the destination via the output control device 20a. The connection state specifying unit 80 integrates information on the detected devices, specifies information related to the connection state such as the type, number, and connection relationship, and notifies the information processing unit 74 of the information.

When a plurality of HMDs 18 are detected and different images are displayed, the information processing unit 74 requests the output data generation unit 78 to generate a display image corresponding to each HMD 18. The information processing unit 74 further requests the output data generation unit 78 to generate a combined image obtained by reducing the data of the plurality of display images generated as described above and connecting them for each frame. The communication unit 76 similarly transmits the composite image data generated in this manner to the output control apparatus 20a. A specific synthesis example will be described in detail later.

The output control device 20a includes a first communication unit 82 that acquires output data transmitted from the information processing device 10, an output data forming unit 84 that processes the acquired output data into an appropriate format as necessary, and a connected HMD 18a. A second communication unit 86 that outputs data to the other device, and a third communication unit 88 that outputs data to another connected device. The first communication unit 82 establishes communication with a device connected to a higher level by a predetermined protocol, and acquires output data transmitted from the information processing device 10 directly or indirectly. Since the output control apparatus 20a shown in FIG. 5 is directly connected to the information processing apparatus 10, it can also directly acquire output data. The lower-order output control device 20 sequentially acquires output data from the higher-order output control device 20.

The output data forming unit 84 processes the output data acquired by the first communication unit 82 as necessary, and outputs the processed data to the second communication unit 86 as a state that can be output to the HMD 18a. Specifically, when the obtained output data includes composite image data, only the image to be displayed on the HMD 18a is extracted and enlarged to the original image size, whereby the image (a) in FIG. An image like this is generated. When the output data acquired by the first communication unit 82 does not include a composite image, that is, when the original image is in the state of FIG. 4A, the output data forming unit 84 uses the data as it is as the second communication unit. 86 may be output. As the latter case, a case where the HMD 18 other than the HMD 18a is not connected, or a case where a common image is displayed on all of the plurality of HMDs 18 can be considered.

The output data forming unit 84 further appropriately generates and outputs data that can be output to the device connected to the third communication unit 88. As a device that may be connected to the third communication unit 88, another output control device 20b, a flat panel display 16, and another HMD 18b may be considered. When another output control device 20 b is connected, the output data forming unit 84 may output the output data acquired by the first communication unit 82 to the third communication unit 88 as it is. When the flat display 16 is connected, the output data forming unit 84 processes the output data acquired by the first communication unit 82 and generates an image for the flat display. Specifically, as described above, an image such as the image (b) in FIG. 4 is generated by applying reverse correction to the left or right image of the HMD image.

When another HMD 18b is connected to the third communication unit 88, the same processing as that for generating data to be output to the HMD 18a connected to the second communication unit 86 is performed. That is, when the obtained output data includes composite image data, the image to be displayed on the HMD 18b is extracted and enlarged to the original image size. When the composite image is not included, the third communication is performed as it is. To the unit 88. As described above, the output data forming unit 84 appropriately switches the data to be output to the third communication unit 88 depending on the type of device connected to the third communication unit 88. In addition, the output data forming unit 84 performs processing such as decoding of audio data included in the output data as necessary.

The second communication unit 86 establishes communication with the connected HMD 18a according to a predetermined protocol, and transmits the output data that the output data forming unit 84 is able to output, thereby causing the HMD 18a to output moving images and audio. The third communication unit 88 establishes communication with another output control device 20b, the flat display 16, or another HMD 18b, and the output data forming unit 84 can output to those devices. Send the output data.

Note that the communication unit 76 of the information processing device 10, the first communication unit 82, the second communication unit 86, and the third communication unit 88 of the output control device 20a are connected to each other by transmitting and receiving signals defined by the protocol used for communication. Detect device type. By aggregating this information on the information processing apparatus 10 side according to the processing procedure according to the protocol, the connection status specifying unit 80 determines the number of output control devices 20 connected and the type of display device connected to each. Can be identified.

General communication technology can be applied to such processing. For example, when connected by HDMI, a connected device can be detected by a hot plug detection signal, or the type of device can be recognized using the HDMI-CEC (Consumer Electronics Control) mechanism, and the connection order can be specified by addressing. Alternatively, a device detection function and an addressing function by USB can be used.

FIG. 6 is a diagram for explaining the composite image data output by the information processing apparatus 10. The images 208a and 208b in the upper part of the figure are images that appear appropriately when displayed on the HMD 18 as they are. For example, the image 208a is displayed on the HMD 18a and the image 208b is displayed on the HMD 18b. At this time, the output data generation unit 78 first generates images 208a and 208b corresponding to the HMDs 18a and 18b. Next, the generated images 208a and 208b are reduced by a factor of 1/2 in the horizontal direction and connected to the left and right to generate a composite image 210 for output.

That is, in this case, in the composite image 210, the reduced image of the first image 208a is arranged in the left region 212a and the reduced image of the second image 208b is arranged in the right region 212b. It has a configuration. The information processing apparatus 10 generates such a composite image 210 for each frame and sequentially outputs it. When displaying different images on all N HMDs 18, the first image, the second image,..., The Nth image are each reduced horizontally by 1 / N times. Just connect. By doing so, the amount of data to be output is constant regardless of the number of HMDs 18 connected, so that it is not necessary to expand communication hardware and the transfer rate is also constant.

When generating the composite image 210, the output data generation unit 78 includes information associating the area in the composite image 210 with the HMD 18 to be enlarged and displayed as additional information in the output data. This information may be transmitted at the start of output of a series of moving image data, or may be periodically transmitted, for example, added to each frame. The information processing unit 74 requests the output data generation unit 78 to generate each display image in association with the identification information of the HMD 18 that has been assigned in advance, and the correspondence between the identification information and the arrangement of data in the composite image. Is generated and notified to the output data generation unit 78 to request generation of a composite image.

The model data necessary for generating the display image, the arrangement rule for generating the composite image, and the like are stored in the data storage unit 81. The output data forming unit 84 of the output control device 20 extracts, for example, the region 212a from the composite image 210 in FIG. 6 and doubles it in the horizontal direction to restore the image 208a and output it to the HMD 18. At this time, the area to be extracted is specified by referring to the additional information of the output data transmitted from the information processing apparatus 10. For image reduction / enlargement processing, any of general methods such as nearest neighbor method, bilinear interpolation method, bicubic interpolation method, and Lanczos method may be used.

7 to 12 show an example of the configuration of the image data transmitted between the connection states of the respective devices in the information processing system 8 of the present embodiment and between them. First, FIG. 7 shows a case where the HMD 18 and the flat display 16 are connected to one output control device 20 as a basic example. In this case, after generating the image 120 to be displayed on the HMD 18, the information processing apparatus 10 transmits the image 120 as it is to the output control apparatus 20 (S10). The output control device 20 outputs and displays the image 120 as it is on the connected HMD 18 (S12), generates a display image 122 for a flat panel display, and transmits it to the flat panel display 16 (S14).

In the figure, the HMD image 120 corresponds to the image (a) in FIG. 4, and an ellipse labeled “L” in the rectangle indicates the left-eye image, and an ellipse labeled “R” indicates the right-eye image. Yes. The flat display image 122 corresponds to the image (b) in FIG. 4. In this case, an image obtained by reversely correcting the left-eye image of the image 120 is represented by a rectangular portion denoted as “L”. However, the right-eye image may be reversely corrected as an image for a flat panel display, or both the left-eye image and the right-eye image may be reverse-corrected and simultaneously displayed on the flat panel display for stereoscopic viewing. Also good.

FIG. 8 is a diagram in which a second HMD 18b is connected instead of the flat panel display 16 in the connected state shown in FIG. It is assumed that the same image is displayed on the two HMDs 18a and 18b. In this case, when the information processing apparatus 10 generates an image to be commonly displayed on the HMDs 18a and 18b, the information processing apparatus 10 transmits the image 120 as it is to the output control apparatus 20 (S20). The output control device 20 outputs the image 120 as it is to the two connected HMDs 18a and 18b, and displays them on each (S22 and S24). This allows two users to watch the same video or share experiences in the virtual world.

FIG. 9 shows a case where two HMDs 18a and 18b are connected to one output control device 20 as in the connection state shown in FIG. At this time, when the information processing apparatus 10 generates images to be displayed on the HMDs 18a and 18b, the information processing apparatus 10 transmits the composite image 124 that is reduced in the horizontal direction by a factor of 1/2 and connected to the left and right to the output control apparatus 20 (S30). ). In the rectangle of the composite image 124, the ellipses denoted as “L1” and “R1” are the left and right display images for the HMD 18a, and the ellipses denoted as “L2” and “R2” are the left and right display images for the HMD 18b, respectively. An image reduced in the horizontal direction is shown.

The output control device 20 generates images 126 and 128 to be displayed on the HMDs 18a and 18b based on the correspondence between the region in the composite image and the identification information of the HMD 18 included in the additional information transmitted from the information processing device 10, respectively. Then, each HMD 18a, 18b is output and displayed (S32, S34). In the example shown in the figure, for the HMD 18a, the left half area including the ellipses labeled “L1” and “R1” is extracted from the composite image 124, and is doubled in the horizontal direction to be used for the HMD 18a. The display image 126 is generated and transmitted to the HMD 18a (S32).

Similarly, for the HMD 18b, the right half area including the ellipses denoted as “L2” and “R2” is extracted from the composite image 124, and is expanded by a factor of 2 in the horizontal direction to display for the HMD 18b. The image 128 is generated and transmitted to the HMD 18b (S34). According to this aspect, it is possible to realize an aspect in which two users view the same virtual world from different viewpoints or view different screens of the same game.

FIG. 10 shows a case where two HMDs 18a and 18b are connected to two output control devices 20a and 20b, respectively, and a flat panel display 16 is connected to a lower output control device 20b. When displaying different images on the HMDs 18a and 18b in this connected state, the information processing apparatus 10 first reduces the image displayed on the HMDs 18a and 18b by a factor of 1/2 in the horizontal direction as described with reference to FIG. The composite image 124 connected to is transmitted to the upper output control apparatus 20a (S40).

Based on the correspondence between the area in the composite image and the identification information of the HMD 18 included in the additional information transmitted from the information processing apparatus 10, the output control apparatus 20a corresponds to the area corresponding to the connected HMD 18a, in the example of FIG. A region including an ellipse labeled “L1” and “R1” is specified. Then, the region is extracted from the composite image 124, and the display image 126 for the HMD 18a is generated by enlarging it twice in the horizontal direction, and transmitted to the HMD 18a (S42). The output control device 20a also transmits the data of the composite image 124 transmitted from the information processing device 10 to the output control device 20b connected to the lower level as it is (S44).

The output control device 20b, based on the correspondence between the region in the composite image and the identification information of the HMD 18 included in the additional information transmitted from the information processing device 10, the region corresponding to the connected HMD 18b, an example of FIG. Then, an area including an ellipse written as “L2” or “R2” is specified. Then, the region is extracted from the composite image 124, and the display image 128 for the HMD 18b is generated by enlarging it twice in the horizontal direction, and transmitted to the HMD 18b (S46). The output control device 20b also extracts a region corresponding to the image for the left eye of the HMD 18a from the composite image 124, enlarges it twice in the horizontal direction, and applies reverse correction to thereby display a display image 130 for a flat panel display. Is generated and transmitted to the flat display 16 (S48).

As shown in FIG. 10, even if each output control device 20 has only two output terminals, by connecting another output control device 20 to one of the two output terminals, it is possible to connect three or more display devices in parallel. Display is possible. If the number of output control devices 20 is further increased, the number of display devices to be displayed in parallel can also be increased. FIG. 11 shows an example in which four HMDs 18 are connected. In this example, four output control devices 20a, 20b, 20c, and 20d are connected in series to the information processing device 10, and HMDs 18a, 18b, 18c, and 18d are connected to the output control devices 20a, 20b, 20c, and 20d, respectively. ing.

When displaying individual images on the four HMDs 18a, 18b, 18c, and 18d, the information processing apparatus 10 reduces the display images corresponding to the HMDs 18a, 18b, 18c, and 18d by a factor of 1/4 in the horizontal direction. Then, the composite image 134 connected in the horizontal direction is generated and transmitted to the highest-level output control device 20a (S50). In the example shown in the figure, the display images corresponding to the HMDs 18a, 18b, 18c, and 18d are represented as “L1 / R1”, “L2 / R2”, “L3 / R3”, and “L4 / R4”, respectively. The output control device 20a refers to the additional information, extracts a region corresponding to the connected HMD 18a from the synthesized image 134, and expands the image by four times in the horizontal direction, thereby generating a display image 136 for the HMD 18a and transmitting it to the HMD 18a. (S52).

The output control device 20a also transmits the data of the composite image 134 transmitted from the information processing device 10 to the output control device 20b connected to the lower level as it is (S54). Thereafter, the data of the composite image 134 is sequentially transmitted to the output control devices 20c and 20d (S58 and S62), and also to the HMDs 18b, 18c and 18d connected to the respective output control devices 20b, 20c and 20d. The respective display images 138, 140, 142 are restored and transmitted (S56, S60, S64). Thereby, individual images are displayed on the four HMDs 18a to 18d.

By adopting the same configuration, it is possible to display different images on any number of HMDs 18 while keeping the necessary transfer rate constant. Further, by connecting the flat display 16 to the output control device 20d or by replacing at least one of the HMDs 18a to 18d with the flat display 16, any number of flat displays 16 or any number of HMDs 18 can be obtained. The image can be displayed in parallel on a display system that combines the flat panel display 16 and the flat panel display 16. In the figure, the connection order of the HMDs 18a to 18d and the arrangement order of the display images in the composite image 134 correspond to each other. However, as long as they are associated in the additional information as described above, the order is not particularly limited.

10 and 11, the data of the composite images 124 and 134 output from the information processing apparatus 10 are transmitted as they are between the plurality of output control apparatuses 20. On the other hand, when the plurality of HMDs 18 all display different images as in these examples, the higher-level output control device 20 excludes the image extracted by itself from the original composite image, and then uses the remaining image data as the next. You may make it transmit to the low-order output control apparatus 20 connected. In this way, the size of the image data to be transmitted can be reduced as it passes through the output control device 20.

FIG. 12 shows a case where four HMDs 18a, 18b, 18c, and 18d connected in the same manner as in FIG. 11 form a plurality of groups, and individual images are displayed for each group. In the example shown in the drawing, it is assumed that the HMDs 18a, 18c, and 18d belong to the group “a” and the HMD 18b belongs to the group “b” as appended to the HMDs 18a to 18d. In this case, the information processing apparatus 10 reduces the display image corresponding to each group (denoted as “La / Ra” and “Lb / Rb”) by a factor of 1/2 in the horizontal direction, and the combined image 154 connected to the left and right. Is transmitted to the highest-level output control device 20a (S70).

The output control device 20a refers to the additional information, extracts an area corresponding to the group “a” to which the connected HMD 18a belongs from the composite image 154, and doubles the region in the horizontal direction, so that the group for the group “a” A display image 156 is generated and transmitted to the HMD 18a (S72). The output control device 20a also transmits the data of the composite image 154 transmitted from the information processing device 10 to the output control device 20b connected to the lower level as it is (S74). The output control device 20b generates a display image 158 for the group “b” by extracting the region corresponding to the group “b” to which the connected HMD 18b belongs from the composite image 154 and expanding the region twice in the horizontal direction. Is transmitted to the HMD 18b (S76).

The output control device 20b also transmits the data of the composite image 154 transmitted from the information processing device 10 to the output control device 20c connected to the lower level as it is (S78). Similarly to the output control device 20a, the output control device 20c generates a display image 156 for the group “a”, transmits it to the connected HMD 18c (S80), and sends it to the output control device 20d connected to the lower level. The data of the composite image 154 transmitted from the information processing apparatus 10 is transmitted as it is (S82). The output control device 20d also generates a display image 156 for the group “a” and transmits it to the connected HMD 18d (S84).

In the illustrated example, the four HMDs 18a to 18d are divided into two groups a and b, but the number of HMDs 18 and groups is not particularly limited. Such an aspect can be applied to a case where an individual field of view is displayed for each team in a team-matching game, or a case where a display screen is changed depending on positions such as game participants and spectators. Further, when the flat display 16 is connected as described above, the screen of the flat display 16 may be divided into a plurality of areas and, for example, images for the left eye of the images displayed on the respective HMDs 18 may be displayed in parallel. .

In this way, for example, images that each user participating in the race game is viewing individually on the HMD 18 can be shown in an aligned state, and users who are not participating in the game can enjoy watching the game. The composite image output by the information processing apparatus 10 is not limited to being composed of only an HMD image, but may include an image that can be displayed on the flat display 16 as it is, or a plurality of images that are displayed on each of the multiple flat displays 16. You may include images. The information processing apparatus 10 may transmit such composite image data to another information processing apparatus or server via a network and display the data on a display device at a remote location.

FIG. 13 shows an example of the data structure of the connection status of the devices managed by the information processing device 10 and the correspondence information of the parameters used for processing. The device management information 160 includes a display device ID column 162, a physical ID column 164, an output control device ID column 166, a display device type column 168, a group ID column 170, and an image region column 172. As described above, when the output control device 20, the HMD 18, and the flat display 16 are connected to the information processing device 10, the connection state specifying unit 80 detects the devices connected using the communication function, and Identification information (logical address) is assigned to the physical address.

The information described in the display device ID column 162 in the illustrated device management information 160 is an identification number assigned to the display device, that is, the HMD 18 or the flat display 16 among the detected devices. Information related to the order of connection is also represented by assignment. The physical ID column 164 represents a physical address that each display device has. The output control device ID column 166 represents identification information of the output control device 20 to which each display device is connected. The display device type column 168 represents information for identifying the type of each display device, that is, whether it is an HMD or a flat panel display (shown as “TV” in the figure).

In the illustrated example, the display devices with the identification numbers “01” to “04” are HMDs, and the display device with the identification numbers “05” is “TV”, that is, a flat panel display. The HMDs with the identification numbers “01” to “04” are connected to the output control devices 20 with the identification information “A” to “D”, respectively. The output control device 20 with the identification information “D” is further connected to the identification number “05”. ”Is connected. The connection state identification unit 80 collects these pieces of correspondence information and notifies the information processing unit 74 of them. On the other hand, the information processing unit 74 determines whether to display a common image on all display devices, display a different image on all, or display a different image for each group depending on the content of the game. Make a decision.

When specifying a different image for each group, specify the group to which each display device belongs. For example, if each HMD 18 is determined for each user, separately holding information that associates the user ID with the physical address of the HMD 18 and grouping the user IDs by user operation at the start of the game, etc. As a result, the identification number and group of each display device can be associated. However, it should be understood by those skilled in the art that various methods can be considered for associating the HMD 18 with the user and the HMD 18 with the group.

In the device management information 160 shown in the figure, the group ID column 170 shows the identification information of the group associated with each HMD 18 in this way. In the illustrated example, it is assumed that three HMDs 18 with identification numbers “01”, “03”, and “04” belong to the group “a”, and HMDs with the identification number “02” belong to the group “b”. This example corresponds to the grouping shown in FIG. In addition, since the identification number “05” is the flat display 16, in this example, both images are set to be displayed in parallel on the divided screen by belonging to both the groups “a” and “b”. However, only one of the images may be displayed as described above. Of course, the group ID column 170 can be omitted for games that do not require grouping.

Subsequently, the information processing unit 74 determines the arrangement of the image data to be displayed on the HMD 18 of each group in the composite image. The lower part of FIG. 13 schematically shows an example of the arrangement. As described with reference to FIG. 12, in this example, the left half of the composite image 174 is the left and right image data of the group “a”, and the right half is the data of the left and right images of the group “b”. In this way, when the image area is divided in the horizontal direction and the data of each display image is arranged, the horizontal range of the composite image is set in the image area column 172 of the device management information 160. For example, as shown in the figure, when the horizontal size of the composite image 174 is X pixels, the range on the coordinate axis where the left end of the image is 0 and the right end is X is specified in a format such as (start point coordinate, end point coordinate). .

In the example shown in the figure, an area of (0, X / 2-1) representing the left half of the composite image 174 is designated for the HMDs 18 having the identification numbers “01”, “03”, and “04” belonging to the group “a”. Has been. For the HMD 18 with the identification number “02” belonging to the group “b”, an area (X / 2, X) representing the right half of the composite image is designated. Further, for the flat panel display 16 with the identification number “05” for simultaneously displaying the images of both groups, (0, X / 4-1) and (X / 2, 3X) corresponding to the left eye images of each group. The area of / 4-1) is designated.

Such device management information 160 is stored in the data storage unit 81 and is shared inside the information processing device 10. The output data generation unit 78 performs the image generation process requested from the information processing unit 74 in association with the identification number of the display device or the identification information of the group to generate a display image, and then reduces them horizontally. Then, the composite image 174 is generated by connecting as specified in the image area column 172. Further, the information processing apparatus 10 transmits the correspondence relationship represented by the apparatus management information 160 to the output control apparatus 20 as additional information.

On the other hand, each output control device 20 that has received the data of the composite image 174 specifies an area corresponding to the display device connected to itself based on the additional information, and extracts the final display image from the composite image 174. Form. Note that the fields of the device management information 160 and the format of the notation shown here are merely examples, and various forms are conceivable depending on the content of the game, the communication protocol, and the like. For example, the user ID may be further associated, or may be associated with the input device ID of each user.

Next, the operation of the information processing system 8 realized by the configuration described so far will be described. FIG. 14 is a flowchart illustrating a processing procedure in which the information processing apparatus 10 outputs image data to be displayed on each display device. First, when the information processing system 8 is constructed by connecting the output control device 20 or the like to the information processing device 10, the communication unit 76 and the connection state specifying unit 80 detect the connected device and assign an identification number. (S100). The information is notified to the information processing unit 74 and each detected device.

When the user forms a group according to the contents of the game (Y in S102), the HMD 18 is divided into groups and the number of groups is set to a variable N in accordance with separately set grouping information or correspondence information between the user and the HMD. Substitute (S104). On the other hand, when a group is not formed (N in S102), the number of HMDs is substituted into a variable N (S106). Then, the region obtained by dividing the image region into N in the horizontal direction is associated with each HMD or each group (S108), and the corresponding information is notified to each output control device 20 (S110). When only one HMD 18 is connected, since N = 1, no division is performed.

Then, a display image is generated by the cooperation of the information processing unit 74 and the output data generation unit 78, and if necessary, the display image is reduced to 1 / N times in the horizontal direction and arranged in the associated region for output. Are generated and output (S112). The output image is a composite image composed of one display image when N = 1 and a plurality of display images when N> 1. While it is not necessary to end the processing, the generation and output processing of the output image is continued at a predetermined frame rate (N in S114, S112), and the user inputs an end request or the game ends. When it is necessary to end the processing, all the processing is finished (Y in S114).

FIG. 15 is a flowchart showing a processing procedure for causing the output control device 20 to display an image on each display device. First, the output control device 20 acquires information indicating the correspondence between the display device or group and the region in the composite image (S120). This information is notified from the information processing apparatus 10 in S110 of FIG. However, as described above, the timing of notifying the correspondence information is not particularly limited, and may be periodically added to the output image such as every frame or every predetermined number of frames.

Next, the output control device 20 acquires the image data transmitted from the information processing device 10 (S122). If the acquired image is a composite image (Y in S124), the region corresponding to the HMD 18 connected to itself or a group to which it belongs is extracted from the composite image and expanded in the horizontal direction to obtain a final display image. Is generated (S126). Whether the image transmitted from the information processing apparatus 10 is a composite image, the area to be extracted, and the enlargement ratio can be specified from the correspondence information acquired in S120.

If the image transmitted from the information processing apparatus 10 is not a composite image, whether there is only one HMD 18 connected or a common image is displayed on all of the plurality of HMDs. (N in S124). Next, when the flat display 16 is connected to itself (Y in S128), the output control device 20 extracts data of one of the left-eye images and applies reverse correction to the flat display. A display image for use is generated (S130). If the flat display 16 is not connected (N of S128), the process of S130 is skipped.

Then, the display image data generated in S126 is output to the connected HMD 18 and the display image data generated in S130 is output to the flat panel display 16, thereby displaying the image on each display device (S132). At this time, if another output control device 20 is connected to the lower level, the image data acquired from the information processing device 10 in S122 is output to the other output control device 20 as it is. While it is not necessary to end the processing, the processing of S124 to S132 is repeated for the image data transmitted directly or indirectly from the information processing apparatus 10 (N of S134, S122). As a result, a moving image such as a game screen is displayed on each HMD 18 and the flat display 16. When it is necessary to terminate the processing due to a notification from the information processing apparatus 10 or the like, all the processing is terminated (Y in S134).

In the example described so far, as illustrated in FIG. 6, an image obtained by reducing the display image of each HMD 18 to 1 / N times in the horizontal direction in an area obtained by dividing the image area of the composite image into N in the horizontal direction. By arranging, images of the same size can be transmitted regardless of the number of HMDs 18. The composition of the composite image is not limited to this. For example, a reduced display image may be arranged in an area that is divided into N parts in the vertical direction. FIG. 16 is a diagram for explaining a composite image in which display images are connected in the vertical direction. The upper part of the figure is a composite image 180 in this case, and the lower part is images 182 and 184 to be displayed on the two HMDs 18 respectively.

In this example, since the images 182 and 184 are originally obtained by arranging the left and right parallax images in a region obtained by dividing the screen in the horizontal direction, in order to make the composite image 180 in the vertical direction as illustrated, Rearrangement is also necessary for the left and right images constituting one display image. That is, in the image 182, the left-eye image labeled “L1” is arranged in the first stage of the composite image 180, and the right-eye image written “R1” is arranged in the second stage of the composite image 180. Similarly, in the image 184, the image for the left eye labeled “L2” is arranged in the third row of the composite image 180, and the image for the right eye written “R2” is arranged in the fourth row of the composite image 180. . Here, the images “L1”, “R1”, “L2”, and “R2” are each enlarged by 2 times in the horizontal direction, reduced by 1/4 times in the vertical direction, and then connected in the vertical direction. As shown, a composite image 180 having the same size as the original images 182 and 184 is obtained.

Similarly, when N HMD images are included in the composite image, the left-eye and right-eye images of each image are doubled in the horizontal direction and reduced to 1 / (2N) times in the vertical direction. In addition, the left eye image and the right eye image are connected in a predetermined order. As a result, a composite image in which the left-eye image and the right-eye image corresponding to one display image are vertically arranged in each region obtained by dividing the image region into N in the vertical direction can be generated. Each output control device 20 extracts a region corresponding to the connected HMD 18, enlarges the left-eye image and the right-eye image by 2N times in the vertical direction, reduces the image by 1/2 times in the horizontal direction, The images 182 and 184 are restored. In this case, the processing procedure may be the same as that shown in FIGS. 14 and 15 except that the area designation in the image area column 172 of the device management information 160 indicates the vertical coordinate axis.

When a plurality of display images are connected in the horizontal direction as shown in FIG. 6, when the information processing device 10 outputs the pixel values of the composite image in raster order, the output control device 20 outputs the pixels in the corresponding region. Since it is sufficient to output to the HMD 18 while interpolating in order, the required capacity of the buffer memory can be reduced. When a plurality of display images are connected in the vertical direction as shown in FIG. 16, “L1”, “R1”, “L2”, and “R2” are output in raster order. Until the data on the first line of “R1” is output as a display image, it is necessary to temporarily store the data up to the last line of “L1”.

That is, in the aspect of connecting in the vertical direction, a buffer memory having a capacity for storing at least the entire reduced image of the left-eye image is required. Also, as the number of images to be included in the composite image increases, the input of data located below the composite image is delayed, and the time until the display of the image using the data is started is delayed by a minute time that is equal to or shorter than the frame period. It will be. On the other hand, when connecting in the vertical direction, it is not necessary to reduce the original display image in the horizontal direction, so that the information in the horizontal direction is not lost. In the aspect in which the left and right parallax images are displayed on the HMD 18 and stereoscopically viewed, the information in the horizontal direction represents the left and right parallax, and thus tends to affect the stereoscopic effect.

As a result, the three-dimensional effect is less likely to be lost in the vertical connection mode. For these reasons, whether the connection direction of the display image is the horizontal direction or the vertical direction is appropriately determined according to the memory capacity that can be used, the display contents, and whether or not stereoscopic viewing is performed. In the example of FIG. 16, it is assumed that the composite image is composed only of the data of the left and right parallax images. However, when including a single image to be displayed on the flat display 16 or the like, the vertical direction is simply 1 in the vertical direction. It may be reduced to / N times.

In the embodiment described so far, the images to be displayed on the respective HMDs 18 are reduced and combined to form a combined image having the same size as the original image, so that the amount of data (transmission band) to be transmitted per unit time is the number of HMDs 18. It was made not to be influenced by. On the other hand, the following modifications are also conceivable. FIGS. 17 and 18 are diagrams for explaining a modification of the transmission mode of image data to be displayed on a plurality of HMDs. First, FIG. 17 shows an example in which a plurality of images to be displayed are connected and transmitted without being reduced. The upper part of the figure shows the composite image 220 in this case, and the lower part shows the images 222, 224, 226, and 228 to be displayed on the four HMDs 18, respectively.

As shown in the figure, the composite image 220 is an image in which four display images 222, 224, 226, and 228 are connected as they are, two in the horizontal direction and two in the vertical direction. As a result, the composite image 220 has a size four times that of the original image. In this way, the transmission band between the information processing device 10 and each output control device 20 is quadrupled. However, since the display image extracted from the composite image 220 can be displayed on each HMD 18 without enlarging, the output control device 20 processing load is reduced. Note that the number of images to be connected as a composite image is not limited to four, and the connection order is not limited to that illustrated. In some cases, one display image may be divided and connected to a plurality of rows or columns. Furthermore, it may be combined with a mode of reducing the display image, and more display images may be combined while allowing the size of the combined image to be expanded.

FIG. 18 shows an example in which a plurality of images to be displayed are transmitted at a high rate without being reduced. The upper part of the figure shows an image sequence 230 to be output in a normal output period for one frame, and the lower part shows images 232 and 234 to be displayed on the two HMDs 18 respectively. In other words, in this example, the images 232 and 234 to be displayed on each HMD 18 are arranged on the time axis direction indicated by “t” in FIG. Since each of the images 232 and 234 is a moving image frame image, as a result, two moving image frame images are alternately output at a double rate. Even when three or more moving images are included, the frame images are output in order.

In this case, the output control device 20 extracts the data at the timing when the data of the frame to be displayed on the corresponding HMD 18 arrives. Therefore, the information processing apparatus 10 determines the output order according to the number of HMDs 18 and transmits additional information that associates the order with the HMDs 18 to be displayed. In this aspect, when the number of images to be displayed is N, the output rate is increased N times, so the transmission band between the information processing device 10 and each output control device 20 is increased N times. On the other hand, similarly to the mode of FIG. 17, each HMD 18 can display the acquired image without enlarging, thereby reducing the processing load of the output control device 20. In this case as well, it may be combined with a mode in which the display image is reduced, and more display images may be output while allowing an increase in output rate.

Next, a method of transmitting correspondence information between the area in the image transmitted from the information processing apparatus 10 and the HMD 18 will be exemplified. As described above, the correspondence information may be transmitted to the output control device 20 prior to transmitting a series of output data from the information processing device 10, or may be included as additional information in the output data. FIG. 19 is a diagram for explaining a method of embedding information in an image output from the information processing apparatus 10 as one embodiment of the latter. The image 190 is an image transmitted from the information processing apparatus 10 to the output control apparatus 20. In the example of FIG. 12, as illustrated in FIGS. 12 and 13, the display image for the group “a” and the group “b” are displayed. This is a composite image obtained by connecting display images for use.

When a parallax image subjected to distortion correction corresponding to a lens is displayed as in this example, contents to be displayed such as a game screen and a virtual world in the display image 190 are represented by “La” and “Ra”. , “Lb”, “Rb”, and other regions are invisible regions. Therefore, by displaying the corresponding information using the invisible region, necessary information can be transmitted without increasing the data amount. For example, in the image 190, the left half invisible area 192a including the display image data of the group “a” and the right half invisible area 192b including the display image data of the group “b” are filled with different colors.

If the color information corresponding to the group to which the connected HMD 18 belongs is given to the output control device 20 in advance, the output control device 20 determines which region of the image 190 should be extracted in the invisible region. Can be identified based on color. In an aspect in which no group is formed, the HMD 18 and the color, or the output control device 20 and the color can be directly associated with each other. In any aspect, the information processing apparatus 10 and each output control apparatus 20 may share the correspondence relationship. For example, if the physical address and color of the HMD 18 and the output control apparatus 20 are set in advance in both of them. No need to send or receive information about correspondence.

Alternatively, several correspondence patterns are created, and the information processing apparatus 10 determines which pattern to use depending on the content, the connection state of the display device, and the like at the start of the game, and notifies the output control device 20 of it. It may be. In the case of a composite image composed of three or more display images, the operation is the same if each invisible region is filled with a different color. Further, even if it is not a composite image, it can be represented that it is not a composite image because the colors of all invisible regions are the same. As the fill color, for example, by selecting three colors of red, green, and blue from 0 or eight patterns of colors represented by the maximum gradation, the output control device 20 can easily determine the region to be extracted without misunderstanding.

The information processing apparatus 10 outputs data of the image 190 in which the correspondence information is represented by the color of the invisible region for all the frames to be output or frames at a predetermined interval. In this way, the area determination process in the output control apparatus 20 can be simplified, and even if there is a change in the area to be extracted due to the increase or decrease in the number of connected HMDs 18 in the middle, there is no delay in the same process. Appropriate areas can be extracted. Note that not all invisible areas may be filled, but only some areas of a predetermined size may be filled. Even in an invisible region corresponding to one display image, the amount of information to be embedded may be increased by using different colors or displaying marks or patterns depending on the position.

For example, in the image 190, the left half invisible area 192a and the right half invisible area 192b are each composed of six small areas. Of these, information such as the area in the upper left corner corresponding to the group and the area in the upper center indicating the enlargement ratio of the extracted image may be expressed in color. The above-described pattern of the correspondence relationship between the color and the apparatus may be represented by a color at any location. Further, the information may be expressed by making the invisible region a striped pattern and changing the direction, width, color combination, and the like.

According to the present embodiment described above, an output control device is connected in series to an information processing device that generates image data, and a display device such as an HMD or a flat panel display is connected to each output control device. The information processing apparatus generates and outputs one display image or a combined image obtained by reducing a plurality of display images according to the connection state and the content of the image to be displayed on each display device. This increases the number of connected display devices and the number of images displayed on them without increasing the number of output terminals of each information processing device or output control device, and without increasing the size of data to be transmitted. Can do.

This mode is particularly effective for content using an HMD in which a user views a display image individually, and a unique mode in which a plurality of users view the same game or virtual world from individual viewpoints can be easily realized. . If a pair of HMD and output control device is provided as a set, the user can enjoy the desired content at a minimum cost by purchasing additional sets according to the content and the number of participants.

Also, a composite image is composed of images that have been subjected to distortion correction corresponding to the lens for display by the HMD. Information for specifying an area to be extracted from the composite image by each output control device is represented by a color or a graphic in the invisible area generated thereby. Thereby, it is not necessary to separately transmit data for the information, and data transfer and extraction processing can be made efficient. Even if the composition of the composite image changes midway, the output control device can extract an appropriate region by the same processing.

The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

8 information processing system, 10 information processing device, 12 imaging device, 14a input device, 16 flat panel display, 18 HMD, 20 output control device, 72 input data acquisition unit, 74 information processing unit, 76 communication unit, 78 output data generation Part, 80 connection state specifying part, 81 data storage part, 82 first communication part, 84 output data forming part, 86 second communication part, 88 third communication part.

As described above, the present invention can be used for a computer, a display device, a game device, an information processing device, an image processing device, a system including them, and the like that process various electronic contents.

Claims (22)

1. An output data generation unit for generating moving image data to be displayed on the display device;
   A communication unit that sequentially outputs moving image data generated by the output data generation unit,
   When a plurality of the display devices are connected, the output data generation unit causes the communication unit to output combined image data obtained by combining a plurality of moving images to be displayed independently on each display device for each frame. An information processing apparatus characterized by the above.
2. The output data generation unit generates the composite image by connecting each frame image of the plurality of moving images after reducing the connection at a magnification corresponding to the number of the frame images. The information processing apparatus described in 1.
3. A connection state specifying unit for specifying a connection state of the display device;
   The output data generation unit generates correspondence information that associates the layout of each frame image to be combined in the composite image with a display device that displays it based on the connection state,
   The information processing apparatus according to claim 1, wherein the communication unit also outputs the correspondence information.
4. The output data generation unit generates data of a moving image to be displayed on a head-mounted display having a lens on the front surface of the display panel as the display device, and when viewed through the lens among each frame image The information processing apparatus according to claim 3, wherein an image corresponding to the correspondence information is represented in a region that is invisible.
5. When the number of frame images to be synthesized is N (N is a natural number of 2 or more), the output data generation unit connects the frame images in the horizontal direction with the horizontal size being multiplied by 1 / N. The information processing apparatus according to claim 1, wherein the composite image is generated.
6. The output data generation unit generates moving image data including a frame image in which a pair of a left-eye image and a right-eye image for stereoscopic display displayed on a head-mounted display as the display device is displayed side by side, 6. The information processing apparatus according to claim 5, wherein the composite image is generated by connecting the left-eye image and the right-eye image by reducing each of them in the horizontal direction by 1 / N times.
7. When the number of frame images to be combined is N (N is a natural number equal to or greater than 2), the output data generation unit connects each frame image in the vertical direction with a vertical size of 1 / N times. The information processing apparatus according to claim 1, wherein the composite image is generated.
8. The output data generation unit generates moving image data including a frame image in which a pair of a left-eye image and a right-eye image for stereoscopic display displayed on a head-mounted display as the display device is displayed side by side, The composite image is generated by enlarging the left-eye image and the right-eye image twice in the horizontal direction, reducing the image in the vertical direction to 1 / N times, and connecting them in the vertical direction. Item 8. The information processing device according to Item 7.
9. When the plurality of connected display devices form a group, the output data generation unit generates the combined image by generating and combining the moving image data for each group. The information processing apparatus according to any one of claims 1 to 8.
10. An input unit that acquires display moving image data output from the information processing apparatus;
    When the display moving image data includes composite image data in which a plurality of moving images to be displayed independently on a plurality of display devices are combined for each frame, a partial region corresponding to the display device connected to the display device Is extracted from the data of the composite image, and an output data forming unit to be a final display image;
    An output unit for outputting the data of the final display image to the connected display device;
    An output control device comprising:
11. The input unit also acquires correspondence information that associates the arrangement of each frame image to be synthesized in the synthesized image with a display device that displays the arrangement.
    The output control device according to claim 10, wherein the output data forming unit specifies and extracts a partial region corresponding to a display device connected to the output data forming unit based on the correspondence information.
12. The composite image is formed by connecting images obtained by reducing the frame images to be combined at a magnification corresponding to the number of frames.
    The output data forming unit generates the final display image by enlarging the extracted partial region at a magnification according to the number of the frame images to be synthesized specified based on the correspondence information. The output control device according to claim 11, characterized in that:
13. The plurality of display devices include a head mounted display including a lens on the front surface of the display panel,
    The output data forming unit identifies the correspondence information associated with the frame data by reading out an image represented in a region that is invisible when viewed through the lens in the synthesized frame image. The output control device according to claim 11 or 12,
14. The output control device according to any one of claims 10 to 13, wherein the output unit further outputs the display moving image data output from the information processing device to another output control device.
15. An information processing system comprising: an information processing device that outputs moving image data; and an output control device that acquires the moving image data and causes the display device to display the data.
    The information processing apparatus includes:
    An output data generation unit for generating moving image data to be displayed on the display device;
    A communication unit that sequentially outputs moving image data generated by the output data generation unit to the output control device;
    When a plurality of the display devices are connected, the output data generation unit causes the communication unit to transmit combined image data obtained by combining a plurality of moving images to be displayed independently on each display device for each frame,
    The output control device includes:
    An input unit for acquiring data of the moving image;
    An output data forming unit that extracts a partial area to be displayed on a display device connected to the moving image data from the synthesized image data when the moving image data includes the synthesized image data; ,
    An output unit for outputting the data of the final display image to the connected display device;
    An information processing system comprising:
16. A plurality of the output control devices are connected in series to the information processing device,
    Each output control device outputs the data of the final display image to a head mounted display connected to the output control device, and transmits the data of the moving image output by the information processing device in the serial order. The information processing system according to claim 15.
17. Generating moving image data to be displayed on the display device;
    And sequentially outputting the generated moving image data,
    The step of generating the moving image data includes a step of generating combined image data obtained by combining a plurality of moving images to be displayed on each display device for each frame when a plurality of the display devices are connected. A method for outputting moving image data by an information processing apparatus.
18. Obtaining display moving image data output by the information processing apparatus;
    When the display moving image data includes composite image data in which a plurality of moving images to be independently displayed on a plurality of display devices are combined for each frame, a partial region to be displayed on a display device connected to the display moving image data Are extracted from the data of the composite image and set as a final display image;
    Outputting the data of the final display image to the connected display device;
    A moving image data output method by an output control device comprising:
19. A function of generating moving image data to be displayed on the display device;
    A computer program for causing a computer to perform a function of sequentially outputting generated moving image data,
    The function of generating moving image data is to generate combined image data obtained by combining a plurality of moving images to be displayed on each display device for each frame when a plurality of the display devices are connected. A computer program characterized by the above.
20. A function of acquiring display moving image data output by the information processing apparatus;
    When the display moving image data includes composite image data in which a plurality of moving images to be independently displayed on a plurality of display devices are combined for each frame, a partial region to be displayed on a display device connected to the display moving image data Is extracted from the composite image data, and the final display image,
    A function of outputting the final display image data to the connected display device;
    A computer program for causing a computer to realize the above.
21. A function of generating moving image data to be displayed on the display device;
    A computer program for causing a computer to perform a function of sequentially outputting generated moving image data,
    The function of generating the moving image data is a computer that generates combined image data obtained by combining a plurality of moving images to be displayed on each display device for each frame when a plurality of the display devices are connected. A computer-readable recording medium having a program recorded thereon.
22. A function of acquiring display moving image data output by the information processing apparatus;
    When the display moving image data includes composite image data in which a plurality of moving images to be independently displayed on a plurality of display devices are combined for each frame, a partial region to be displayed on a display device connected to the display moving image data Is extracted from the composite image data, and the final display image,
    A function of outputting the final display image data to the connected display device;
    A computer-readable recording medium characterized by recording a computer program that causes a computer to realize the above.

Images