JP5653771B2 - Video display device and program - Google Patents

Description

  The present invention relates to a video display device, a program, and the like.

  For example, Patent Literature 1 discloses a technique for providing a content sharing experience among a plurality of people by constructing a so-called virtual reality environment. Further, as shown in Patent Document 2, a technique is known in which an image to be viewed is displayed as if it was fixed at real space coordinates.

Japanese Patent No. 36553463 Japanese Patent No. 3368672

  The prior art has not been able to realize an effective method for communicating when viewing content with a plurality of head-mounted display devices attached.

  According to some aspects of the present invention, it is possible to provide a video display device, a program, and the like that realize smooth and natural communication with the outside.

  One embodiment of the present invention is a first video acquisition unit that acquires a first video, and a second video that acquires a second video generated based on a captured image from an imaging unit or communication information from a communication unit. An acquisition unit, a processing unit, and a display unit, wherein the processing unit communicates with another video display device whose video display mode is a viewing mode or different from its own video display device. A video display mode determination unit that determines whether the video communication mode is the communication mode, and when the video display mode determination unit determines that the video display mode is the viewing mode, the display unit A first video is displayed, and when the video display mode determination unit determines that the video display mode is the communication mode, the video display device displays the second video. Engaged to.

  In one embodiment of the present invention, after the first video and the second video are acquired, the video to be displayed can be switched according to the video viewing mode. Thereby, in a situation where communication is performed, smooth communication is realized by displaying a video for communication.

  Moreover, in one aspect of the present invention, the video display mode determination unit is configured to determine whether the video display mode is based on positional relationship information representing a positional relationship between the video display device of its own and the other video display device. It may be determined whether the viewing mode is set or the communication mode is set.

  Thereby, determination of a video display mode can be performed based on the positional relationship information between its own video display device and other video display devices (or users wearing them). Therefore, it is possible to shift to the communication mode when the positional relationship is expected to be communicated.

  In the aspect of the invention, the processing unit includes an image recognition unit that performs image recognition processing based on the captured image, and the video display mode determination unit is a recognition result of the captured image by the image recognition unit. Based on the above, it may be determined whether the video display mode is the viewing mode or the communication mode.

  This makes it possible to determine the video display mode based on the image recognition process.

  Further, in one aspect of the present invention, the video display mode determination unit may determine that the other video display device is more than the given distance from the video display device as a result of the image recognition process in the image recognition unit. When it is determined that the image is present at a close position, the video display mode may be determined to be the communication mode.

  Thereby, it becomes possible to shift to the communication mode based on the distance information between the other display device and its own video display device.

  In one aspect of the present invention, the image recognition unit performs size recognition of the other video display device in the captured image by performing image recognition processing of the other video display device in the captured image. The video display mode determination unit acquires the video display mode in the communication mode when the size represented by the size information of the other video display device is larger than a given threshold. You may determine that there is.

  Thereby, it becomes possible to shift to the communication mode based on the size information in the captured image of another video display device.

  Moreover, in one aspect of the present invention, the image recognition unit performs image recognition processing of an identification marker provided to the other video display device in the captured image, so that the size of the identification marker in the captured image is increased. The video display mode determination unit determines that the video display mode is the communication mode when the size represented by the size information of the identification marker is greater than a given threshold. You may determine that there is.

  Thereby, it becomes possible to shift to the communication mode based on the size information in the captured image of the identification marker given to another video display device.

  Moreover, in one aspect of the present invention, the processing unit includes a sensor information acquisition unit that acquires sensor information from a sensor, and the video display mode determination unit is configured so that the video display mode is based on the sensor information. It may be determined whether the mode is the viewing mode or the communication mode.

  This makes it possible to determine the video display mode based on the sensor information.

  Further, in one aspect of the present invention, when it is determined by the sensor information that the other video display device exists within a given angle range set for the video display device of its own, The video display mode determination unit may determine that the video display mode is the communication mode.

  This makes it possible to shift to the communication mode when another video display device is within a given angle range.

  In one aspect of the present invention, the processing unit includes a reception processing unit that receives position transmission information transmitted by the other video display device within a receiving range having directivity, and the video display mode determination unit includes: The sensor information acquisition unit may determine that the video display mode is the communication mode when the position transmission information is acquired as the sensor information.

  This makes it possible to determine whether or not another video display device is in a given angle range based on the reception result in the reception processing unit that receives in the reception range having directivity.

  In the aspect of the invention, the reception processing unit may receive the position transmission information in a reception range corresponding to the visual field range of the user.

  Thereby, the process corresponding to a user's visual field range is attained.

  In the aspect of the invention, the processing unit may include a transmission processing unit that transmits position transmission information in a transmission range that does not have directivity.

  This makes it possible to transmit the position transmission information within a transmission range that does not have directivity.

  Further, according to an aspect of the present invention, the video switching unit includes a video switching unit that performs switching processing between the first video and the second video, and the video switching mode is determined by the video display mode determination unit. If it is determined that the viewing mode is selected, a video switching process is performed so that the first video is displayed on the display unit, and the video display mode determination unit determines that the video display mode is the communication mode. If it is determined that the second video is displayed, the video switching process may be performed so that the second video is displayed on the display unit.

  This makes it possible to switch the video to be displayed based on the determination of the video display mode.

  In one aspect of the present invention, a second detection area is set around a user wearing the video display device, and the first detection area is in an angle range corresponding to the visual field range of the user. When it is determined that the other video display device is present in the second area, the processing unit performs a second process with a light load, and the other video is displayed in the first area. If it is determined that a display device is present, the first process having a higher load than the second process may be performed.

  Accordingly, when another video display device exists in the second detection area, the second process with a light load is performed, and when another video display device exists in the first detection area, the second process is performed. It is possible to perform the first process having a heavier load than the first process.

  In the aspect of the invention, the processing unit may include at least one of reception processing by the reception processing unit, transmission processing by the transmission processing unit, and identification marker recognition processing by the image recognition unit as the second process with a light load. Processing may be performed.

  This makes it possible to perform reception processing, transmission processing, and marker recognition processing as the second processing with a light load.

  In one embodiment of the present invention, the processing unit is based on other user information, which is information on another user who has received the other video display device received from the communication unit, as the first processing with a heavy load. At least one of state determination processing, communication start processing by the communication unit, and video switching processing by the video switching unit may be performed.

  As a result, it is possible to perform a state determination process based on user information, a communication start process, and a video switching process as the first process that is heavier than the second process.

  In one aspect of the present invention, the second video acquisition unit acquires other user information, which is information of another user wearing the other video display device, from the other video display device, A video generated based on the other user information may be acquired as the second video.

  Thereby, it is possible to use a video generated based on the information of another user as the second video that is a video for communication.

  In the aspect of the invention, the second video acquisition unit may acquire the avatar video of the other user generated based on the other user information as the second video.

  Thereby, it becomes possible to use an avatar image of another user as the second image, and a richer communication can be realized.

  In the aspect of the invention, the second video acquisition unit acquires the own user information that is information of a user wearing the own video display device, and the second video is the own user information. You may acquire the chat image | video showing speech information and speech information which is the said other user information.

  This makes it possible to use a chat video representing the utterance information of the user and other users as the second video. Therefore, it is possible to communicate by text chat without interrupting the audio content of the first video.

  In the aspect of the invention, the video display device may be a head-mounted display device that is mounted on a user's head and has a display unit in at least a part of the user's field of view.

  This makes it possible to use a head-mounted display device mounted on the user's head as the video display device.

  Another aspect of the present invention provides a first video acquisition unit that acquires a first video, and a second video that is generated based on a captured image from the imaging unit or communication information from the communication unit. The computer functions as a display control unit that performs display control of a video acquisition unit, a processing unit, and a display unit, and the processing unit is in a viewing mode or different from its own video display device. A video display mode determination unit that determines whether the communication mode is for communication with another video display device, and the display control unit is configured to determine whether the video display mode is the viewing mode by the video display mode determination unit. If it is determined that the mode is a mode, control is performed to display the first video on the display unit, and the video display mode determination unit controls the video display mode to be the communication mode. If it is determined that the Nmodo, there is provided a program for controlling display of said second image on said display unit.

The system configuration example of this embodiment. The example of the condition using the video display apparatus which concerns on this embodiment. Example of the first video. An example in which a captured image is displayed as a second video. 2 is a detailed system configuration example of the present embodiment. An example in which only a video display device is displayed as the second video. The system structural example in the case of using sensor information. The figure explaining the transmission / reception range of a beacon. An example in which another video display device can be detected when a beacon is used. The system structural example in the case of producing | generating a 2nd image | video by processing with respect to a captured image. An example in which a picture-in-picture image is displayed as the second video. An example in which a superimposed image is displayed as the second video. The system configuration example in 2nd Embodiment. An example in which an avatar image is displayed as the second video. An example in which a chat image is displayed as the second video. An example of setting multiple detection areas. Another example of setting a plurality of detection areas. Examples of sensors for realizing multiple detection areas The flowchart for demonstrating the detail of the process in a modification. An example of a monocular head-mounted display device used as a video display device according to the present embodiment.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

  1. Method of this embodiment

  First, the method of this embodiment will be described. The video display device according to the present embodiment is, for example, a head-mounted display device (HMD). In particular, the present embodiment assumes an HMD that has a wide display angle of view (a wide video display range in the user's visual field range) and enables video display with a sense of realism.

  The problem in this case is that the outside world can hardly be seen when the HMD is mounted. Since the video displayed by the HMD (mostly the first video here) occupies most of the user's visual field range, for example, even when trying to communicate with other users nearby, The problem is that it is difficult to communicate smoothly. Further, even in an HMD of a type where the visual field range is not as wide as in the present embodiment, if the display unit is held so that the display unit of the HMD enters a part of the visual field range, it is hidden by the display unit. The problem that it becomes difficult to confirm the appearance of the user will also occur.

  Therefore, the present applicant proposes the following method. That is, when the user wearing the HMD faces the direction of another user (assuming that other users are also wearing the HMD), the first video is replaced with the assistance of communication with the other user. A second video image is displayed. Specifically, as illustrated in FIG. 4, a captured image obtained by capturing another user may be displayed as the second video, or the captured image and the first image may be displayed as appropriate images as illustrated in FIG. 11. An image that is synthesized by processing may be displayed as the second video. By doing in this way, since a display image switches to the image for communication as needed, it becomes possible to perform communication with other users smoothly.

  Hereinafter, the above-described method using a video based on a captured image as the second video will be described in the first embodiment. A method of using an avatar image or a chat image generated based on information of another user as the second video will be described in the second embodiment. Finally, a modification for the purpose of reducing the processing load will be described.

  2. First embodiment

  FIG. 1 shows a first embodiment of a video display device of the present invention. The embodiment of the present invention is constructed as a system having at least two or more video display devices (head-mounted display devices, HMDs) as follows. The first video display device 10 includes a display unit 15, a display control unit 17, a video switching unit 14, a first video acquisition unit 11, a second video acquisition unit 12, a processing unit 16, and a video A display mode determination unit 13 is included. The second video display device 20 includes a display unit 25, a display control unit 27, a video switching unit 24, a first video acquisition unit 21, a second video acquisition unit 22, and a processing unit 26. The video display mode determination unit 23 is provided. Hereinafter, since the first video display device 10 and the second video display device 20 have the same configuration, the first video display device 10 will be described, and the second video display device 20 will not be described in detail. Omitted. In the following, the first video display device 10 is considered as a reference, and the video display device of itself represents the first video display device 10 and the other video display device is the second video display device 20 (or not). It represents the third video display device shown in the figure.

  The first video acquisition unit 11 acquires a video for viewing, that is, a video for viewing content such as a movie or a photo album. The second video acquisition unit 12 plays back video for communication. In the first embodiment, an image of the outside world or an image obtained by processing the image is acquired.

  When a plurality of users (two users here, but not limited to them) are watching a video side by side using a head-mounted display device in the situation shown in FIG. As shown in FIG. 3, the video signal (first video) from the first video acquisition unit 11 is displayed on the display unit 15.

  The video display mode determination unit 13 determines the video display mode based on, for example, positional relationship information between its own video display device and another video display device. Specifically, another head-mounted display device (for example, the second video display device 20) has a predetermined range, for example, the front left and right of its own head-mounted display device (first video display device 10). When the distance is within 15 degrees and within a distance of 1.5 m, the video switching unit 14 is activated, and the display video is changed from the first video from the first video acquisition unit 11 to the second video acquisition unit 12. Switch to the second video from Thereby, it can switch to the image | video for communication as shown in FIG.

  A more detailed configuration will be described in detail with reference to FIG. As shown in FIG. 5, the first video display device 100 (head-mounted display device) includes an imaging unit 106, a second video acquisition unit 116, a processing unit 102, a video switching unit 104, and video playback. Unit 101, first video acquisition unit 115, display unit 105, and display control unit 117. The processing unit 102 includes an image recognition unit 107 and a video display mode determination unit 103. Similarly, the second video display device 200 (head-mounted display device) includes an imaging unit 206, a second video acquisition unit 216, a processing unit 202, a video switching unit 204, and a video playback unit 201. , A first video acquisition unit 215, a display unit 205, and a display control unit 217. The processing unit 202 includes an image recognition unit 207 and a video display mode determination unit 203. Hereinafter, since the first video display device 100 and the second video display device 200 have the same configuration, the first video display device 100 will be described, and the second video display device 200 will not be described in detail. Omitted. In the following, the first video display device 100 is considered as a reference, and the video display device of itself represents the first video display device 100, and the other video display devices are the second video display device 200 (or not). It represents the third video display device shown in the figure. The same applies to the description of the block diagrams in FIG.

  The imaging unit 106 includes an imaging system such as an optical system for image formation, a shutter and diaphragm for light amount adjustment, an ND filter, and other imaging elements such as a CCD and a CMOS, and an image for outputting a video signal by NTSC or digital image transmission. It consists of a processing unit (for example, a video processing circuit). It is good also as a structure similar to what is called a video camera and a digital still camera. Video signals from these are input to the image recognition unit 107 of the processing unit 102. The image recognition unit 107 is configured by a computer or the like on which a computer program for digitally converting input video is installed. The video display mode determination unit 103 may be implemented as a computer program.

  The video playback unit 101 may be a conventional video source such as a video cassette recorder, but is connected to a device that decodes and plays back an image file recorded on a medium such as a semiconductor memory, HDD, or Blu-ray disc, or an online video distribution service. It may be a computer program. The video switching unit 104 is a device that switches a video signal (second video) from the imaging unit 106 and a video signal (first video) from the video playback unit 101, and is implemented by, for example, a signal switching circuit. . The video signal selected by the switching is connected to the display unit 105 via the display control unit 117.

  In this embodiment, the display unit 105 is shown as a head mounted display (HMD) provided with display panels on both the left and right eyes. This is a monocular type or a screen virtual image by a mirror installed in the near field of view. It may be a helmet type display. Note that in the case of an unconstrained display, an imaging device that operates in conjunction with the movement of the head may be provided.

  Next, the operation of this embodiment will be described. As shown in FIG. 3, when viewing a video from the video playback unit 101 (for example, a video playback device such as a DVD player), the video switching unit 104 in FIG. Are sent to the display unit 105 (specifically, an HMD panel) for display.

  Here, as shown in FIG. 4, when facing the direction of another user (HMD user), the imaging unit 106 shown in FIG. 5 causes the HMD image (an image obtained by photographing the outside world, particularly worn by other users). The image of the HMD being taken) is taken. The image recognition unit 107 recognizes the HMD image and calculates the presence / absence of another HMD on the other side, the size in the screen, and the like. These object recognitions are omitted because they use general object recognition technology. From such recognition, an HMD detection signal for notifying that another HMD is present in the field of view and an HMD size signal indicating the size of the HMD recognition area in the screen are sent to the video display mode determination unit 103.

  The video display mode determination unit 103 determines the video display mode by acquiring positional relationship information between its own video display device and another video display device based on the received signal. Specifically, if the HMD size signal is present in the captured image at a predetermined ratio, for example, a screen ratio when the distance from the video display device is within a range of 50 cm to 2 m, the video display mode determination is performed. Unit 103 outputs a switching signal for switching the display video to the second video. This signal is input to the video switching unit 104, switched from the first video from the video playback unit 101 to the second video captured by the imaging unit 106 and displayed on the display unit 105.

  At this time, as shown in FIG. 4, the state of another user captured by the imaging unit 106 is displayed by video see-through, so that the start of communication can be grasped and the state of the other party can be monitored.

  Therefore, this embodiment has the following specific effects in addition to the above-described effects. That is, two processes, that is, another HMD detection and measurement process and a second video signal acquisition process for communication displayed on the display unit 105 can be performed by one imaging apparatus (imaging unit 116). The configuration is simple.

  Of course, each configuration of the present embodiment can be variously modified and changed. For other HMD recognition, an identification marker may be added to the HMD to recognize it. In this case, it is necessary to have size information for knowing the distance and geometric information for knowing the direction. For example, LED bright spot markers that are separated by a predetermined distance between three points and do not lie on the same straight line. Alternatively, a flat marker or the like that indicates the position and orientation by providing a one-point characteristic code pattern such as a QR code may be used.

  For example, as illustrated in FIG. 6, the HMD detection process (positional relationship information determination process) may be performed based on a beacon signal transmitted and received by the transmission / reception unit 108. As shown in FIG. 7, the transmission / reception unit 108 that transmits and receives beacon signals is connected to the video display mode determination unit 103. When no other HMD transmission range is included in the beacon reception range as shown in FIG. 8, the video signal from the video reproduction unit 101, that is, the first video is displayed on the display unit 105. Also, as shown in FIG. 9, when the transmission range and the reception range overlap and HMD2 enters the reception range of HMD1, the video display mode determination unit 103 outputs a switching signal, and the video switching unit 104 is an imaging unit. The video is switched to the video signal (second video) from 106. In the present embodiment, the target HMD can determine whether or not the HMD exists in the communication range from the radio wave intensity and the direction.

  In this case, the image recognition unit 107 may not be provided, but it is desirable that the field angle range of the imaging unit 106 and the HMD detection range of the beacon substantially match.

  In addition, the video from the imaging unit 106 is not output to the display unit 105 as it is, but the video processed through the video generation unit 112 as shown in FIG. 10 is output as the second video to the video switching unit 104. Also good. This video processing may be, for example, picture-in-picture display (PinP) in which an image from the imaging unit 106 is displayed as a child screen as shown in FIG. 11, and of course, the parent-child screen may be reversed. Further, as shown in FIG. 12, the second video may be displayed in a superimposed manner after the contrast of the first video is displayed as a background.

  In the present embodiment described above, the video display device (the first video display device 10 that is its own video display device) includes the first video acquisition unit 11 that acquires the first video as shown in FIG. The second video acquisition unit 12 that acquires the second video, the processing unit 16, and the display unit 15 are included. The processing unit 16 includes a video display mode determination unit 13 that determines a video display mode. Further, when the video display mode determination unit 13 determines that the video display mode is the viewing mode, the display unit 15 displays the first video, and the video display mode is determined to be the communication mode. If it does, the second video is displayed.

  Here, the first video is, for example, video from a video playback device such as a DVD player, and specifically, content such as a movie can be considered. The first video may be a video from a network-type distribution service, a television broadcast, or the like. The second video is a video generated based on a captured image from the imaging unit (corresponding to 106 in FIG. 5) or communication information from the communication unit (corresponding to 113 in FIG. 13 described later). The term video means here all contents including multimodal information such as moving images and audio. However, the video may be multimodal, as well as a single modal signal such as a moving image or audio alone.

  The video display mode includes a viewing mode and a communication mode. The viewing mode is a mode for viewing content such as a movie, and the communication mode is a mode for communicating with other users. As a situation when the communication mode is executed, the case as shown in FIG. 2 can be considered. For example, a plurality of users wearing HMDs communicate with each other when they exist in a limited area such as the same room. It is for smoothness. However, it is not necessary for users to be within a given distance, and specifically, it is not necessary to be in the same room. In this embodiment, each user is assumed to be wearing an HMD, and communication with a person not wearing the HMD may not be considered. Note that the video display device according to the present embodiment is not limited to the HMD, and includes a small display that interlocks with the movement of the head.

  As a result, it is possible to acquire the first video and the second video, display the first video in the viewing mode, and display the second video in the communication mode. Therefore, it is possible to smoothly communicate with other users even while using the video display device. In particular, the present invention is effective when the video display device is a device whose external field of view is narrowly limited, such as an HMD (particularly, a display image occupies a large proportion of the user's visual field range).

  In addition, the video display mode determination unit 13 determines whether the video display mode is the viewing mode or the communication mode based on the positional relationship information indicating the positional relationship between the video display device of its own and another video display device. May be performed.

  Thus, the video display mode can be determined based on the positional relationship information between the video display devices (that is, between users who wear the video display devices). Therefore, for example, if the setting is made so as to shift to the communication mode in a positional relationship where communication is likely to be performed, the video display mode can be changed appropriately and effectively according to the situation. Note that the positional relationship information is specifically acquired by a captured image from the imaging unit 106 in FIG. 5, sensor information from the sensor information acquisition unit (transmission / reception unit 108 in FIG. 7), and the like, as will be described later. Can be considered.

  Further, as illustrated in FIG. 5, the processing unit 102 may include an image recognition unit 107 that performs image recognition processing based on the captured image. The video display mode determination unit 103 determines the video display mode based on the recognition result of the captured image by the image recognition unit 107.

  Here, in particular, the positional relationship information between the own video display device and another video display device may be acquired based on the recognition result of the captured image. The video display mode determination unit may determine the video display mode based on the positional relationship information. Specifically, as will be described later, it is conceivable to determine the positional relationship based on size information in a captured image of a target video display device (another video display device).

  This makes it possible to switch the video display mode based on the image recognition process for the captured image. Here, when the second video is acquired based on the captured image, both of the second video acquisition process and the video display mode determination process can be performed based on the captured image. become. Therefore, the configuration of the video display device 100 can be simplified.

  Also, the video display mode determination unit 103 determines that another video display device (for example, the second video display device 200) is close to its own video display device (for example, a place) as a result of the image recognition processing by the image recognition unit 107. If it is determined that the video display mode is the communication mode, the video display mode may be determined to be the communication mode.

  As a result, as a method for switching the video display mode based on the image recognition process, whether or not another video display device (or another user using the same) and its own video display device (or own user) are in the vicinity. It is possible to use a method for making a determination based on the above.

  Further, the image recognition unit 107 may acquire size information of another video display device in the captured image by performing image recognition processing of the other video display device on the captured image. The video display mode determination unit 103 determines the video display mode as the communication mode when the size represented by the size information of the other video display device is larger than a given threshold.

  Thereby, in the captured image, when another video display device is captured with a certain size or larger, it is possible to switch to the communication mode. In the captured image, the other image display device being imaged largely means that the other image display device is present at a position close to its own image display device. Furthermore, in view of the fact that the angle of view of the imaging unit 106 (imaging device) that acquires the captured image is limited to a certain angle range unless a special device such as a fisheye lens is used, other video display devices display their own video. It will be located in a given angular range set for the instrument. If the angle of view of the imaging unit is considered to be the same as the visual field range of the own user, the case of satisfying this condition means that the user is looking at the direction of another user nearby. . Therefore, it can be said that it is possible to shift to the communication mode in a natural situation, considering that there is a high possibility that communication such as conversation is usually started in such a situation. Note that the apparent size (for example, the number of pixels) on the captured image and the size information of the video display device do not necessarily correspond one-to-one. When the imaging unit 106 has a zoom function, the size information needs to be changed in conjunction with the size of the zoom magnification.

  In addition, the image recognition unit 107 may acquire the size information of the identification marker in the captured image by performing image recognition processing of the identification marker given to another video display device in the captured image. The video display mode determination unit 103 determines that the video display mode is the communication mode when the size represented by the size information of the identification marker is larger than a given threshold value.

  Thereby, in the captured image, when the identification marker is captured with a certain size or larger, the video display mode can be switched to the communication mode. Recognizing the size of the identification marker given to the video display device is equivalent to acquiring distance information to the identification marker, that is, another video display to which the identification marker is given. The distance information to the device is acquired. The following processing is the same as that in the case of directly performing image recognition processing of another video display device.

  Further, as illustrated in FIG. 7, the processing unit 102 may include a sensor information acquisition unit (transmission / reception unit 108) that acquires sensor information from the sensor. Then, the video display mode determination unit 103 determines the video display mode based on the sensor information.

  Here, in particular, the positional relationship information between the own video display device and another video display device may be acquired based on the sensor information. The video display mode determination unit may determine the video display mode based on the positional relationship information. Specifically, as will be described later with reference to FIGS. 8 and 9, it is conceivable to determine the positional relationship using a beacon or the like.

  This makes it possible to determine the video display mode using the sensor information. In particular, when using positional relationship information, it is possible to grasp the positional relationship between multiple video display devices using a sensor that acquires positional relationship information and determine that communication is likely to be performed. It becomes possible to shift to the communication mode. Since it suffices to process the information by the sensor, the processing is easier than the image recognition processing.

  If it is determined by the sensor information that there is another video display device within a given angle range set for the video display device (or the user wearing the video display device). The video display mode determination unit 103 may determine that the video display mode is the communication mode.

  As a result, not only the position is simply detected, but also the angle range in which the user is present is detected. Therefore, it is appropriate to determine whether or not the current situation is a communication situation. Can be performed.

  Further, as illustrated in FIG. 7, the processing unit 102 may include a reception processing unit 109 that receives position transmission information transmitted by another video display device within a reception range having directivity. The video display mode determination unit 103 determines that the video display mode is the communication mode when the sensor information acquisition unit (transmission / reception unit 108) acquires the position transmission information as the sensor information. In that case, the said reception range is good also as an angle range corresponding to the visual field range of a user (own user).

  As a result, the position transmission information transmitted by the other video display device is received within the receiving range having directivity, so that the other video display device exists within a given angle range with respect to its own video display device. It is possible to determine whether or not to do so. In particular, by associating the reception range with the user's visual field range, it is possible to determine whether there is another video display device within the user's visual field range. Therefore, in a situation where the user is looking at the direction of another video display device, it is possible to shift to the communication mode, and natural transition can be performed.

  The processing unit 102 may include a transmission processing unit 110 that transmits position transmission information in a transmission range that does not have directivity.

  Thereby, it becomes possible to inform the other video display device of the position of its own video display device, and the other video display device can shift to the communication mode.

  Further, as shown in FIG. 1, the video display device 10 may include a video switching unit 14 that performs a video switching process between the first video and the second video. Then, the video switching unit 14 performs video switching processing so that the first video is displayed on the display unit 15 when the video display mode is the viewing mode. Further, when the video display mode is the communication mode, the video switching process is performed so that the second video is displayed on the display unit 15.

  Thereby, the video displayed on the display unit 15 can be switched based on the video display mode. As will be described later, this switching is performed by detection and non-detection of another video display device (for example, the second video display device 20) by the processing unit 16. Here, the video display device of its own is a video display device that is a criterion for current judgment. Further, the other video display device is a video display device other than the above-described own video display device. A user who uses his / her video display device is simply called a user or his / her user, and a user who uses another video display device is called another user or another user.

  The video display device according to the present embodiment may be a head-mounted display device that is mounted on the user's head and has a display unit in at least a part of the user's field of view.

  Here, the head-mounted display device is specifically an HMD or the like as described above. Here, as shown in FIG. 2, the HMD may have a large proportion of the display image with respect to the user's visual field, such as viewing the display image with both eyes. Further, as shown in FIG. 20, the display image may occupy a small proportion of the user's field of view, such as viewing the display image with one eye.

  This makes it possible to use a head-mounted display device as the video display device according to the present embodiment.

  In addition, as illustrated in FIG. 1, the present embodiment includes a first video acquisition unit 11 that acquires a first video, a second video acquisition unit 12 that acquires a second video, a processing unit 16, The display control unit 17 relates to a program that causes a computer to function. The processing unit 16 includes a video display mode determination unit 13 that determines a video display mode. In addition, when the video display mode determination unit 13 determines that the video display mode is the viewing mode, the display control unit 17 performs control to display the first video on the display unit 15, and the video display mode Is determined to be in the communication mode, control is performed to display the second video on the display unit 15.

  As a result, the present embodiment is not limited to a case where the video display device is realized in hardware, but can be applied to a computer system such as a PC that executes video display processing in software. It becomes possible. The program is recorded on an information storage medium. Here, as the information recording medium, various recording media that can be read by an optical detection system, such as an optical disk such as a DVD or a CD, a magneto-optical disk, a hard disk (HDD), a memory such as a nonvolatile memory or a RAM, can be assumed. .

  3. Second embodiment

  Next, a second embodiment will be described with reference to FIG. Compared to the first embodiment, the communication unit 113 and the video generation unit 112 are added.

  Next, the operation of the second embodiment will be described. The communication unit 113 communicates information such as video, audio, text input, and the like between the first HMD and the second HMD. Note that the number of HMDs is not limited to two. Various methods shown in the first embodiment can be used as a method for switching images.

  As the second video, an avatar video is generated based on information obtained by communication. The image generated in this way is displayed as a second video as shown in FIG.

  Of course, each configuration of the present embodiment can be variously modified and changed. The text chat may be realized by exchanging the text input information via the communication unit 113 and displayed on the display unit 105 by switching the video according to the switching condition of the first embodiment.

  With the configuration as described above, communication between appropriate HMD wearers can be easily realized by switching to the second video.

  In the present embodiment, as shown in FIG. 13, the second video acquisition unit 116 (again, as in the first embodiment, the first video display device 100 in FIG. 13 is used as its own video display device. Is to acquire other user information, which is information of another user, from another video display device (second video display device 200), and generate a video generated based on the other user information as a second video. get.

  As a result, images adjusted for communication will be exchanged, thus realizing richer communication. In particular, in this embodiment, since it is assumed that each user wears an HMD, it is difficult to obtain information such as a user's facial expression when using a captured image. The use of images is particularly effective.

  Moreover, the 2nd image | video acquisition part 116 may acquire the avatar image | video of another user produced | generated based on other user information as a 2nd image | video.

  Thereby, communication using an avatar as shown in FIG. 14 becomes possible. By appropriately setting the avatar's facial expression, posture, movement, etc., richer communication is possible compared to a picked-up image.

  In addition, the second video acquisition unit 116 acquires own user information, which is information on a user wearing the own video display device (first video display device 100), and uses the user information as the second video. You may acquire the chat image | video showing the speech information of a certain own user, and the speech information of the other user which is other user information.

  As a result, rich communication is realized as in the case of an avatar. At the same time, since the uttered voice is displayed as text as shown in FIG. 15, appropriate communication can be performed without hindering viewing of the first video content centered on the voice.

  4). Modified example

  Next, a modified example will be described. Here, a plurality of areas for detecting other HMDs are provided, and the processing when the HMD is detected is changed for each area.

  Specifically, for example, a case where two detection areas are set can be considered. As shown in FIG. 16, the user's pointing direction area is set as the first detection area, and the area around the user is set as the second detection area. Here, the first detection area is a gaze range of the user wearing the HMD (generally, a range of 40 to 50 ° as a standard horizontal angle of view). The user's field of view and the HMD display unit 105 (display) This corresponds to the central field angle range of the apparatus. Further, the second detection area corresponds to a region in the same room (with a radius of about 2 m in the case of a home) mainly centering on the HMD. Moreover, you may set the 1st detection area combining the distance centering on HMD similarly to the 2nd detection area.

When another HMD is detected in the first detection area, the other party's state determination (posture, speech, facial expression, etc.), communication start processing (communication start, video generation start, etc.), and video switching processing (Display of the second video) and the like. On the other hand, when another HMD is detected in the second detection area, the processing unit 102 has a light load such as determination of the presence of the HMD user, beacon reception processing / transmission processing, and marker recognition processing. A second process is performed. Compared to the second process, the first process is a heavy process.

  By doing in this way, when other HMD is detected in a detection area relatively far from the user wearing the HMD, that is, when it can be determined that the will of communication between the users is not strong, Perform processing with a light load. When another HMD is detected in the user's pointing direction area, that is, when communication between users is assumed to be performed, processing with a heavy load for communication is performed. Therefore, as compared with the case where processing with heavy load is always performed, it is only necessary to perform heavy processing only when necessary, so that the processing load as a whole can be reduced.

  Further, the detection area is not necessarily limited to two. As shown in FIG. 17, in consideration of personal space, the third and fourth detection areas may be defined within a peripheral radius of 1 m to forcibly shift to the communication mode.

  Specifically, in FIG. 17, when divided into four detection areas, the first detection area notifies both sides of a dialog requesting permission to start communication with each other. In the fourth detection area (about several tens of centimeters in diameter), since the proximity is significant, the mode is forcibly shifted to the communication mode. In the second detection area, the presence of other users around is displayed inconspicuously with a simple icon or the like. Since the third detection area is extremely close from the rear, it is displayed as an alert icon or the like as danger information.

  This makes it possible to divide the detection area more finely and perform different processes.

  Also, the user ID may be registered in advance such as who is in proximity, and the information may be displayed in advance. Further, as the proximity increases and the object moves to the front of the field of view, the communication mode and various information displays for notifying and asking for consent may be changed in stages.

  Furthermore, it is possible to display azimuth information. Even if the other party's HMD is behind the right, as if the eyes are also behind, a notice lamp blinks on the right side of the HMD screen to notify you when you are in the second detection area. You may make it display the arrangement | positioning condition of surrounding HMD on the outer peripheral part of radar like radar.

  If comprised in this way, a corresponding HMD can be caught in the 1st detection area by turning around according to a notice lamp, and it can shift to communication mode. At this time, pre-communication of simple partner information (ID information such as icons and names) may be simultaneously performed, and the contents thereof may be displayed on the display unit 105.

  As an operation example in the movie appreciation mode, the following modes are conceivable. By making it insensitive to the detection of other HMDs, you will be slowly appreciated and produce a more immersive feeling. Specifically, the first and second detection areas may be narrowed (sensitivity becomes dull) on condition that the same image is viewed for a certain period of time as the first video. At the same time, in order to enable a response when a call is made in close proximity, processing that eliminates the first and second detection areas is not performed.

  Next, an example of a sensor for realizing such a modification will be described. As shown in FIG. 18, the reception has directivity and the transmission has no directivity (however, the strength is determined).

  Hereinafter, an example using an infrared transmission / reception module will be described. The transmitting part projects infrared rays to at least a few tens of degrees in the horizontal direction by providing an infrared LED with an omnidirectional diffusing optical system with a milky food cover for diffusion, a cone, a rotating parabolic mirror, and a rotating hyperbolic mirror. To do. On the other hand, an infrared light receiving unit installed with the optical axis directed in the front direction of the wearer through a condensing lens having an angle of view (below the horizontal angle of view of the display device, for example, substantially the same as the viewing angle of 40 to 50 °).

  Thereby, the infrared rays diffused from the light emitting units of other HMDs are received when other HMDs exist in the light receiving range of the wearer's HMD. If it is out of range, the light is not received or the intensity is low because of reflection. The distance is determined by this received light intensity.

  In addition, in order to detect the second detection area, it is necessary to provide another infrared non-directional light receiving unit. In FIG. 18, the omnidirectional infrared light-emitting unit and light-receiving unit have a light-receiving characteristic in which the range of the vertical angle of view is panoramically rotated around the entire circumference. It doesn't matter. For further simplification, the detection intensity of the light receiving element may be detected in the first detection area and the second detection area by combining the omnidirectional imaging optical system and the infrared division light receiving element.

  Further, when the imaging device (imaging unit 106) is also used as a sensor, it has directivity corresponding to an angle range determined by the angle of view of the imaging device.

  In the case of ultrasonic waves, an effect equivalent to that of the above configuration may be obtained by a combination of an omnidirectional speaker (transmitting unit), an omnidirectional microphone, and a directional microphone (receiving unit).

  Next, details of processing when a plurality of detection areas are set as described above will be described with reference to the flowchart of FIG. In this flowchart, when there is another HMD nearby when close to the surroundings, the function of notifying the presence, the function of switching to the communication mode, and the function of forcibly switching to the communication mode when another HMD is present in the vicinity and in the front have.

  Specifically, when this process is started, first, the first video is displayed on the display unit 105 (S301). Next, it is determined whether or not to end the viewing (S302). If Yes, the viewing ends. In the case of No, it is determined whether or not another HMD is detected in the first detection area and the second detection area (S306) (S303), and in the case of No, the process returns to S301. Note that the detection process in the first detection area and the detection process in the second detection area are performed in parallel using, for example, a multithread.

  In S303, when another HMD is detected in the second detection area, the display unit displays that there is another user nearby (S304), and determines whether or not to shift to the communication mode. (S305). In the case of Yes, it transfers to S307 and a communication mode is started. In No, it returns to S301.

  If it is determined Yes in S305, the communication mode is started from S307. First, information of another HMD is acquired (S307), a second video is generated (S308), and the second video is displayed. The information is displayed on the part 105 (S309). Then, it is determined whether to end the communication mode (S310). If No, the process returns to S307 and the communication mode is continued. In the case of Yes in S310, the process returns to S301.

  In parallel with the detection process in the second detection area, it is determined whether another HMD has been detected in the first detection area (S306). If No in S306, the process returns to S301. If Yes, the process proceeds to S311 and the communication mode is started.

  The processing when it is determined Yes in S306 is S311 to S314. Since this process is the same as the process of S307-S310 mentioned above, detailed description is abbreviate | omitted.

  In the present embodiment described above, the first detection area is set in an angle range corresponding to the visual field range of the user wearing his / her own video display device (also described here for the first video display device 100), and the user's When the second detection area is set around, and the processing unit 102 determines that another video display device (such as the second video display device 200) exists in the second detection area, The second process with a light processing load is performed. In addition, when it is determined that another video display device exists in the first detection area, the first process having a higher load than the second process is performed.

  Here, the first process is, for example, a state determination process based on other user information (information of a user wearing another video display device received from the communication unit 113), a communication start process by the communication unit 113, and video switching. For example, video switching processing by the unit 104. Further, the second processing includes, for example, reception processing by the reception processing unit 109, transmission processing by the transmission processing 110 unit, and identification marker recognition processing by the image recognition unit 107.

  Thus, the processing to be performed on the own video display device (first video display device 100) can be changed according to the position where the other video display device (second video display device 200) is present. It becomes possible. As a reference at that time, processing with a heavy load is performed in the first detection area, and processing with a light load is performed in the second detection area. This is because the processing necessary for realizing the communication mode is heavy. In other words, if the process for the constant communication mode is performed, the load on the system is heavy and unnecessary processing is included in the viewing mode, which is wasteful. In view of this, usually, the process is limited to the minimum process necessary for the transition to the viewing mode and the communication mode. Then, when another video display device is found within a given angle range (for example, within the visual field range of the user), that is, when there is a high possibility of communication, the heavy load processing is performed. Resources can be used efficiently.

  The two embodiments 1 and 2 to which the present invention is applied and the modifications thereof have been described above, but the present invention is not limited to the embodiments 1 and 2 and the modifications as they are, The constituent elements can be modified and embodied without departing from the spirit of the invention. Various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described first and second embodiments and modifications. For example, you may delete a some component from all the components described in each Embodiment 1-2 or the modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

10 video display device, 11 first video acquisition unit, 12 second video acquisition unit,
13 video display mode determination unit, 14 video switching unit, 15 display unit, 16 processing unit,
17 display control unit, 100 video display device, 101 video playback unit, 102 processing unit,
103 video display mode determination unit, 104 video switching unit, 105 display unit,
106 imaging unit, 107 image recognition unit, 108 transmission / reception unit, 109 reception processing unit,
110 transmission processing unit, 112 video generation unit, 113 communication unit

Claims (22)

A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
A display unit;
Including
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from the video display device;
The display unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the first video is displayed;
When the video display mode determination unit determines that the video display mode is the communication mode, the second video is displayed ;
The video display mode determination unit
It is determined whether the video display mode is the viewing mode or the communication mode based on positional relationship information representing a positional relationship between the own video display device and the other video display device. A video display device characterized by that. A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
A display unit;
Including
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from its own video display device ;
An image recognition unit that performs image recognition processing based on the captured image ;
The display unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the first video is displayed;
When the video display mode determination unit determines that the video display mode is the communication mode, the second video is displayed ;
The video display mode determination unit
A video display device that determines whether the video display mode is the viewing mode or the communication mode based on a recognition result of the captured image by the image recognition unit . In claim 2 ,
The video display mode determination unit
As a result of the image recognition processing in the image recognition unit, if it is determined that the other video display device is present at a position closer than a given distance from the video display device of its own, the video display mode is A video display device that is determined to be in the communication mode. In claim 2 ,
The image recognition unit
In the captured image, by performing image recognition processing of the other video display device, to obtain size information of the other video display device in the captured image,
The video display mode determination unit
When the size represented by the size information of the other video display device is larger than a given threshold, the video display mode is determined to be the communication mode. machine. In claim 2 ,
The image recognition unit
In the captured image, by performing image recognition processing of the identification marker given to the other video display device, the size information of the identification marker in the captured image is acquired,
The video display mode determination unit
A video display device, wherein when the size represented by the size information of the identification marker is larger than a given threshold, the video display mode is determined to be the communication mode. A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
A display unit;
Including
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from its own video display device ;
Including a sensor information acquisition unit for acquiring sensor information from the sensor ;
The display unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the first video is displayed;
When the video display mode determination unit determines that the video display mode is the communication mode, the second video is displayed ;
The video display mode determination unit
A video display device that determines whether the video display mode is the viewing mode or the communication mode based on the sensor information . In claim 6 ,
When it is determined by the sensor information that the other video display device exists within a given angle range set for the video display device of its own,
The video display mode determination unit
A video display device that determines that the video display mode is the communication mode. In claim 6 or 7 ,
The processor is
Including a reception processing unit for receiving position transmission information transmitted by the other video display device in a reception range having directivity;
The video display mode determination unit
When the sensor information acquisition unit acquires the position transmission information as the sensor information, the video display device determines that the video display mode is the communication mode. In claim 8 ,
The reception processing unit
An image display device, wherein the position transmission information is received within a reception range corresponding to a user 's visual field range. In claim 6 or 7 ,
The processor is
An image display device comprising a transmission processing unit that transmits position transmission information within a transmission range that does not have directivity. A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
A display unit;
Including
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from the video display device;
The display unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the first video is displayed;
When the video display mode determination unit determines that the video display mode is the communication mode, the second video is displayed ;
A second detection area is set around the user wearing the video display device and the first detection area is set in an angle range corresponding to the visual field range of the user.
The processor is
When it is determined that the other video display device exists in the second detection area, a second process with a light load is performed, and it is determined that the other video display device exists in the first detection area. In such a case, the video display device is characterized in that the first process, which is heavier than the second process, is performed . In claim 11 ,
The processor is
A video display device characterized by performing at least one of reception processing by a reception processing unit, transmission processing by a transmission processing unit, and identification marker recognition processing by an image recognition unit as the second processing with a light load. In claim 11 ,
The processor is
As heavy the first processing load, the state determination process based on other user information is information other user wearing the other image display device received from the communication unit, the start process of the communication by the communication unit and A video display device that performs at least one of video switching processing by a video switching unit. In claim 1,
A video switching unit that performs a video switching process between the first video and the second video;
The video switching unit
When the video display mode determination unit determines that the video display mode is the viewing mode, a video switching process is performed so that the first video is displayed on the display unit,
When the video display mode determination unit determines that the video display mode is the communication mode, a video switching process is performed so that the second video is displayed on the display unit. Video display equipment. In any one of Claims 1 thru | or 14 .
The second video acquisition unit
Other user information, which is information of another user wearing the other video display device, is acquired from the other video display device, and the second video is generated based on the other user information. A video display device characterized by acquiring video. In claim 15 ,
The second video acquisition unit
An image display device that acquires an avatar image of the other user generated based on the other user information as the second image. In claim 15 ,
The second video acquisition unit
While acquiring the own user information which is information of the user wearing the own video display device, the second video represents the utterance information which is the own user information and the utterance information which is the other user information. A video display device characterized by acquiring a chat video. In any one of Claims 1 thru | or 17 ,
The image display equipment is mounted on the head of the user, the image display apparatus, which is a head-mounted display device having the display portion in at least a portion of the field of view of the user. A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
As a display control unit that performs display control of the display unit,
Make the computer work,
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from the video display device;
The display control unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the video display mode determination unit performs control to display the first video on the display unit,
By the image display mode decision unit, when the video display mode is determined to be the communication mode, you have row control to display the second image on the display unit,
The video display mode determination unit
It is determined whether the video display mode is the viewing mode or the communication mode based on positional relationship information representing a positional relationship between the own video display device and the other video display device. A program characterized by that. A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
As a display control unit that performs display control of the display unit,
Make the computer work,
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from its own video display device ;
An image recognition unit that performs image recognition processing based on the captured image ;
The display control unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the video display mode determination unit performs control to display the first video on the display unit,
By the image display mode decision unit, when the video display mode is determined to be the communication mode, you have row control to display the second image on the display unit,
The video display mode determination unit
A program for determining whether the video display mode is the viewing mode or the communication mode based on a recognition result of the captured image by the image recognition unit . A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
As a display control unit that performs display control of the display unit,
Make the computer work,
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from its own video display device ;
Including a sensor information acquisition unit for acquiring sensor information from the sensor ;
The display control unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the video display mode determination unit performs control to display the first video on the display unit,
By the image display mode decision unit, when the video display mode is determined to be the communication mode, you have row control to display the second image on the display unit,
The video display mode determination unit
A program for determining whether the video display mode is the viewing mode or the communication mode based on the sensor information . A first video acquisition unit for acquiring a first video;
A second video acquisition unit that acquires a second video generated based on a captured image from the imaging unit or communication information from the communication unit;
A processing unit;
As a display control unit that performs display control of the display unit,
Make the computer work,
The processor is
A video display mode determination unit that determines whether the video display mode is a viewing mode or a communication mode for communicating with another video display device that is different from the video display device;
The display control unit
When the video display mode determination unit determines that the video display mode is the viewing mode, the video display mode determination unit performs control to display the first video on the display unit,
By the image display mode decision unit, when the video display mode is determined to be the communication mode, you have row control to display the second image on the display unit,
A second detection area is set around the user wearing the video display device and the first detection area is set in an angle range corresponding to the visual field range of the user.
The processor is
When it is determined that the other video display device exists in the second detection area, a second process with a light load is performed, and it is determined that the other video display device exists in the first detection area. If so, the program performs the first process that is heavier than the second process .

Images