CN108141559B - Image system, image generation method and computer readable medium - Google Patents
Image system, image generation method and computer readable medium Download PDFInfo
- Publication number
- CN108141559B CN108141559B CN201580083269.3A CN201580083269A CN108141559B CN 108141559 B CN108141559 B CN 108141559B CN 201580083269 A CN201580083269 A CN 201580083269A CN 108141559 B CN108141559 B CN 108141559B
- Authority
- CN
- China
- Prior art keywords
- image
- gaze point
- head
- video
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/66—Transforming electric information into light information
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
- H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/167—Position within a video image, e.g. region of interest [ROI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Controls And Circuits For Display Device (AREA)
Abstract
The video system of the present invention includes a head-mounted display used by being mounted on the head of a user, and a video generation device for generating a video provided to the user by the head-mounted display. The image demonstration part of the head-mounted display is used for demonstrating images to a user. The photographing section is used for photographing an image including eyes of a user. The first communication unit transmits a captured image to the video generation device and receives a video from the video generation device. The second communication unit of the video generation device receives the captured image from the head-mounted display and transmits the video to the head-mounted display. A fixation point acquisition unit acquires a fixation point of a user on a video based on a captured image. The calculation unit sets a predetermined region based on the acquired gaze point, and generates a video having a smaller data amount per unit pixel number than a video calculated for the predetermined region, for the outside of the predetermined region.
Description
Technical Field
The present invention relates to an imaging system, and more particularly, to an imaging system having a head mounted display and an image generating device.
Background
The head-mounted display is used by a user wearing the head, and displays an image on a screen provided at a close distance to the eyes of the user. Since the user cannot see contents other than the displayed image when wearing the head-mounted display, the user can enjoy the feeling of integration with the virtual space. As a technique related to the above, patent document 1 discloses an image generating apparatus and an image generating method that can detect a movement of a user and can display an image corresponding to the movement of the user on a head-mounted display.
Documents of the prior art
Patent document
Patent document 1: japanese patent laid-open publication No. 2013-258614
Disclosure of Invention
Problems to be solved by the invention
With the above technology, the head-mounted display can display an image corresponding to the user's gaze direction on the screen. However, in most cases, the images displayed by the head-mounted display are animated. Therefore, the amount of data is large and when an original video is transferred from the video generation device to the head mounted display, there is a possibility that the update of the image is delayed and the video is interrupted. Also, recently, high-definition monitors have increased, and it is desired to process a large volume of image data. When transmission and reception of image data are taken into consideration, the image generating apparatus and the head-mounted display can be integrated, but since the head-mounted display needs to be used by being attached by a user, miniaturization is desired and incorporation into a housing is difficult. Therefore, in practice, although the image generating apparatus is connected to the head-mounted display by wireless, the image data amount is large, and therefore, the image provided to the user may be in a state of being stopped.
The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a technique relating to a video system capable of suppressing communication stagnation between a head-mounted display and a video generating device.
Means for solving the problems
In order to solve the above-described problems, one embodiment of the present invention is a video system including a head-mounted display used by being mounted on a head of a user and a video generation device for generating a video to be presented to the user by the head-mounted display. In the imaging system, the head-mounted display includes: an image demonstration part for demonstrating images to a user; a photographing section for photographing an image including eyes of a user; and a first communication unit that transmits the image captured by the imaging unit to the video generation device and receives the video presented by the video presentation unit from the video generation device. The image generation device includes: a second communication unit that receives the image captured by the imaging unit from the head-mounted display and transmits the image to the head-mounted display; a fixation point acquisition unit that acquires a fixation point of a user on the image based on the image captured by the imaging unit; and a calculation unit that sets a predetermined region based on the gaze point acquired by the gaze point acquisition unit, and generates a video having a smaller data amount per unit pixel than the video calculated for the predetermined region, for the outside of the predetermined region.
The image generating apparatus may further include a communication determining unit for determining a communication environment between the first communication unit and the second communication unit, and the calculating unit may reduce the amount of image data in a case where the communication environment is poor, as compared with a case where the communication environment is good.
The communication determination section may further determine the communication environment based on information of the latest data including a communication parameter including at least one of electric wave intensity, communication speed, data loss rate, throughput, noise situation, or physical distance from the router.
The image generating device may further include a gaze point movement acquiring unit that acquires movement of the gaze point of the user based on the gaze point acquired by the gaze point acquiring unit, and the calculating unit may change at least one of a size and a shape of the predetermined region in accordance with the movement of the gaze point.
The calculation unit sets the form of the predetermined region to a form having a long axis and a short axis, or to a form having a long side and a short side, and sets the long axis or the long side direction of the predetermined region according to the movement direction of the gaze point.
The calculation unit may generate a video in which the data amount per unit pixel number is changed according to the distance from the gaze point, in addition to the predetermined region.
The calculation unit may generate a video in which the data amount per unit pixel number is continuously decreased as the distance from the gaze point is increased, in addition to the predetermined region.
The calculation unit may generate the video so that the data amount per unit pixel number is not less than the lower limit value.
In addition, any combination of the above constituent elements, expressions of the present invention converted between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as embodiments of the present invention.
Effects of the invention
According to the present invention, the image system including the head-mounted display can appropriately reduce the amount of communication data, whereby an image with less sense of incongruity can be provided to the user without stagnation.
Drawings
Fig. 1 is a diagram schematically showing an overview of an imaging system according to an embodiment.
Fig. 2 is a block diagram showing an example of a functional configuration of the video system according to the embodiment.
Fig. 3 is a diagram illustrating an example of the gaze point of the user acquired by the gaze point acquisition unit according to the embodiment.
Part (a) and part (b) of fig. 4 are diagrams showing examples of the predetermined region set by the calculation unit.
Fig. 5 is a diagram schematically showing a relationship between the X-axis of the video display area and the data amount per unit pixel number.
Fig. 6 is a diagram illustrating an example of movement of the gaze point acquired by the gaze point movement acquiring unit according to the embodiment.
Part (a) and part (b) of fig. 7 are schematic diagrams showing another example of the relationship between the X-axis of the image display region and the data amount per unit pixel number.
Fig. 8 is a timing chart showing an example of processing of the video system according to the embodiment.
Fig. 9 is a flowchart showing an example of a process of communication determination according to the embodiment.
Detailed Description
A brief description of embodiments of the present invention will be given. Fig. 1 is a diagram schematically showing an overview of an imaging system 1 according to an embodiment. The video system 1 according to the embodiment includes a head-mounted display 100 and a video generation device 200. As shown in fig. 1, the head-mounted display 100 is used by being mounted on the head of a user 300.
The image generation apparatus 200 is used to generate an image that is presented to the user by the head-mounted display 100. Although not limited, the image generating apparatus 200 is, for example, a device that can reproduce images such as a stationary game machine, a portable game machine, a Personal Computer (PC), a tablet PC, a smart phone, a tablet phone, a video player, and a television. The image generating apparatus 200 is connected to the head-mounted display 100 by wireless or wired connection. In the example shown in fig. 1, the image generating apparatus 200 is wirelessly connected to the head-mounted display 100. The wireless connection between the image generation apparatus 200 and the head mounted display 100 may be realized by a wireless communication technique such as known wireless fidelity (Wi-Fi) or Bluetooth (registered trademark). Although not limited thereto, the image transmission between the head mounted display 100 and the image generating apparatus 200 is performed according to standards such as wireless display (Miracast, trademark), wireless gigabit (WiGig, trademark), and wireless home digital interface (WHDI, trademark).
The head-mounted display 100 includes a frame 150, a wearing part 160, and a headphone 170. The housing 150 accommodates an image display system such as an image display unit for presenting images to the user 300, and a wireless transmission module such as a wireless fidelity (Wi-Fi) module or a Bluetooth (registered trademark) module, which are not shown. The fitting 160 is used to fit the head mounted display 100 on the head of the user 300. The attachment 160 is implemented by, for example, a band or a stretchable band. When the user 300 wears the head-mounted display 100 with the wearing tool 160, the frame 150 is disposed at a position covering the eyes of the user 300. Therefore, when the user 300 wears the head mounted display 100, the view of the user 300 is blocked by the frame 150.
The headphone 170 is used to output the audio of the video reproduced by the video generating apparatus 200. The headset 170 may not be fixed to the head mounted display 100. The headphone 170 can be freely attached and detached even in a state where the user 300 has the head mounted display 100 attached by the attachment 160.
Fig. 2 is a block diagram showing an example of a functional configuration of the video system 1 according to the embodiment. The head-mounted display 100 includes an image presentation unit 110, an image capturing unit 120, and a first communication unit 130.
The video presentation unit 110 is used to present a video to the user 300. The image demonstration unit 110 is realized by, for example, a liquid crystal display or organic electroluminescence (electroluminescence). The photographing section 120 is used to photograph an image including the eyes of the user. The imaging unit 120 is realized by an image sensor such as a charge-coupled device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS) housed in the housing 150, for example. The first communication unit 130 is connected to the image generating apparatus 200 by wireless or wire, and is used to perform information transmission between the head-mounted display 100 and the image generating apparatus 200. Specifically, the first communication unit 130 is configured to transmit the image captured by the image capturing unit 120 to the video generation apparatus 200 and receive the video presented by the video presentation unit 110 from the video generation apparatus 200. The first communication part 130 may be implemented by a wireless transmission module, for example, a wireless fidelity (Wi-Fi) module or a Bluetooth (Bluetooth) module.
Next, the video image generation device 200 of fig. 2 will be described. The video image generating apparatus 200 includes a second communication unit 210, a communication determination unit 220, a gaze point acquisition unit 230, a gaze point movement acquisition unit 240, a calculation unit 250, and a storage unit 260. The second communication part 210 is connected to the head mounted display 100 by wireless or wire. The second communication unit 210 is configured to receive the image captured by the image capturing unit 120 from the head mounted display 100 and transmit the video to the head mounted display 100. The term "video" in the present specification refers to a video generated by the calculation unit 250 described below. The gaze point acquisition unit 230 acquires the gaze point P of the user on the video based on the image captured by the imaging unit 120. The position of the gaze point P is acquired, for example, by a known line-of-sight detection technique. For example, the gaze point acquisition unit 230 acquires the relationship between the image display position and the reference point and the movement point of the user's eye as calibration information in advance. When reproducing the video, the image capturing unit 120 captures an image of the eyes of the user 300 in the same manner as in the calibration, and the gaze point acquiring unit 230 acquires the position information of the reference point and the moving point based on the image. Based on the acquired position information and the calibration information acquired in advance, the gaze point acquisition unit 230 estimates the gaze point P of the user on the image. Here, the "reference point" refers to, for example, a point such as an eye corner that moves relatively little with respect to the head-mounted display, and the "moving point" refers to an iris, a pupil, or the like that moves according to the position at which the user 300 is looking. Hereinafter, the "gaze point P" refers to the gaze point of the user estimated by the gaze point acquisition unit 230.
Fig. 3 is a diagram illustrating an example of the gaze point P of the user 300 acquired by the gaze point acquisition unit 230 according to the embodiment. The video presentation unit 110 actually displays a three-dimensional object in a video on two-dimensional orthogonal coordinates by the display pixels of the video display area. In fig. 3, the horizontal and vertical directions of the image display area of the head mounted display 100 are set as the X axis and the Y axis, respectively, and the coordinates of the gaze point P are represented by (X, Y). As shown in fig. 3, the position of the gaze point P may also be represented on the image the user is looking at.
Returning to the description of fig. 2. Based on the gaze point P acquired by the gaze point acquisition unit 230, the calculation unit 250 sets a predetermined region a with the gaze point P as a reference. The calculation unit 250 generates a video having a smaller data amount D per unit pixel number than the video calculated for the predetermined area a, for the outer area B other than the predetermined area a. As will be described in detail below, the "data amount per unit pixel D" is an index for comparing how the calculation unit 250 performs different processes in the predetermined area a and the external area B, and indicates the amount of data D per pixel, for example, of the video generated by the video generation apparatus 200 and transmitted to the head-mounted display 100.
Next, the processing executed by the calculation unit 250 will be described with reference to fig. 4 and 5. Part (a) and part (b) of fig. 4 are diagrams illustrating an example of the predetermined region a set by the calculation unit 250. A case where the calculation unit 250 sets a region having a distance a or less from the gaze point P as the predetermined region a will be described with reference to part (a) of fig. 4. The predetermined region a may be a closed region, but an example of the case of setting to a circular shape is shown in part (a) of fig. 4, and an example of the case of setting to a rectangular shape is shown in part (b) of fig. 4. In this way, if the predetermined area a is a simple form, the calculation for setting the predetermined area a in accordance with the movement of the gaze point P by the calculation unit 250 can be reduced.
Generally, the vision of a human eye is as high as the central vision zone, which includes the fovea, and drops sharply when deviated from the fovea. As is well known, the range that can be seen in detail by the human eye is a range up to within 5 ° of the center of the fovea. Therefore, the calculation unit 250 may calculate the distance between the display pixels of the head mounted display 100 and the fovea of the eye of the user 300 approximately, and then set the range on the image display area corresponding to the area of 5 ° fovea as the predetermined area a with reference to the gaze point P of the user 300. The size of the specific defined area a when the user 300 looks can be determined through experimentation in view of the optical system employed by the liquid crystal display of the head mounted display 100 and the human visual characteristics (e.g., central vision, age, viewing angle, etc.) described above.
Fig. 5 is a graph illustrating a relationship between the X axis of the video display area and the data amount D per unit pixel number. The abscissa of the graph corresponds to the X-axis of the image display region, and the ordinate of the graph represents the data amount D per unit pixel number on a line parallel to the X-axis including the fixation point P. Fig. 5 shows an example in which the calculation unit 250 sets the range of the distance a from the gaze point P as the predetermined region a. First, the calculation unit 250 extracts video data of a video to be presented to the user next time from among the video data stored in the storage unit 260. The calculation unit 250 may also acquire external image data of the image generating apparatus 200. The calculation unit 250 performs calculation for reducing the data amount D per unit pixel number in a position where the X axis is smaller than (X-a) or larger than (X + a) with respect to the video data. As a method of reducing the amount of data, for example, a known method of compressing a video by deleting high-frequency components of the video can be used. This makes it possible to obtain an image in which the data amount at the time of communication is reduced as a whole.
An example of a method of deleting a high-frequency component of a video will be described. Specifically, the calculation unit 250 changes the sampling rate used in creating the two-dimensional image from the video data of the three-dimensional model between the inside and the outside of the predetermined region a. The calculation unit 250 reduces the sampling rate for the outside of the predetermined area a compared to the inside of the predetermined area a. Then, the calculation unit 250 generates an image by interpolation processing for the non-sampled region. The interpolation process is, for example, a well-known bilinear or spline interpolation. This makes the image more blurred than when the image is formed over the entire video area at a high sampling rate. As a result, the high-frequency components of the image are deleted, and thus the amount of data at the time of compression is reduced. Further, the sampling rate at the time of image formation is lowered, and thus an image can be formed at high speed.
Returning to the description of fig. 2. The communication determination unit 220 is configured to determine a communication environment between the first communication unit 130 and the second communication unit 210. In the case where the communication environment is not good, the calculation unit 250 may reduce the amount of data of the video as compared with the case where the communication environment is good.
The calculation section 250 may also reduce the data amount D per unit pixel number in the outer region B according to the determination result of the communication environment. For example, the communication environment is divided into three stages of C1, C2, and C3 from a good point of view, and values of data compression rates used in the respective stages are set to E1, E2, and E3 and stored by the storage section 260. The communication deciding unit 220 is configured to decide which stage among C1 to C3 the communication environment corresponds to. The calculation unit 250 acquires the value of the data compression rate according to the determination result from the storage unit 260, and compresses the video data of the outer region B at the acquired data compression rate to generate a video.
Thus, the amount of data for the video transmitted from the video generation apparatus 200 to the head-mounted display 100 is adjusted according to the communication environment, and therefore, the image can be prevented from being stagnated due to a delay in the transmission time or the like. Further, since the image quality does not change in the vicinity of the gaze point P of the user 300, even if the data amount is reduced, it is possible to prevent the user 300 from being given a sense of incongruity. Further, the user can be provided with a video image in which the information on the gaze point P of the user 300 is reflected, which is captured by the imaging unit 120, without delay.
The communication deciding part 220 may also decide the communication environment based on information of the latest data including communication parameters including at least one of electric wave intensity, communication speed, data loss rate, throughput, noise situation, or physical distance from the router.
The communication deciding part 220 may also be configured to monitor a communication parameter and decide the quality of the communication environment based on the communication parameter. The communication determination unit 220 transmits a message inquiring about the communication status to the head mounted display 100. Then, for example, the first communication unit 130 receives the message, acquires communication parameters on the head mounted display 100 side, and transmits the acquired communication parameters to the image generation apparatus 200. Further, the second communication unit 210 acquires communication parameters on the video generating apparatus 200 side. Thus, the communication determination unit 220 can also determine the quality of the communication environment based on the communication parameters received from the head mounted display 100 and the communication parameters acquired by the second communication unit 210. Here, the information including the latest data may be, for example, a value obtained by the communication determination unit 220 by calculation of a moving average from a certain number of past observation values. Further, similarly to the above configuration, when the data compression rate set in association with the communication environment is used, the calculation unit 250 can generate a video of a data amount suitable for the communication environment at that time. Therefore, even for a location where the communication environment is not good or easily changes, the frame rate of the video presented to the user can be maintained, and a video that does not give a sense of incongruity to the user after viewing can be provided.
The gaze point movement acquisition unit 240 may also acquire the movement of the gaze point P of the user 300 based on the gaze point P acquired by the gaze point acquisition unit 230. The calculation unit 250 changes at least one of the size and the form of the predetermined region a in accordance with the movement of the gaze point P acquired by the gaze point movement acquisition unit 240.
Fig. 6 is a diagram illustrating an example of the movement of the gaze point P acquired by the gaze point movement acquiring unit 240 of the embodiment. Fig. 6 shows a state where the user's gaze point P moves from P1 to P2. The calculation unit 250 sets the predetermined region a by referring to the movement of the gaze point P acquired by the gaze point acquisition unit 230 and the gaze point P acquired by the gaze point movement acquisition unit 240. In the example shown in fig. 6, the gaze point P is at the position of P2, and the direction of movement of the gaze point is indicated by an arrow. The predetermined area a does not need to be arranged with the gaze point P as the center. For example, as shown in fig. 6, the calculation unit 250 may set the boundary of the predetermined area a so as not to be equidistant from the gaze point P2, and so as to have a wide movement direction of the movement of the gaze point P and be within the predetermined area a. Thus, the head mounted display 100 can provide an image for the user 300 that maintains image quality over a wide range including the direction in which the line of sight of the user 300 is directed. As described above, the predetermined area a may be circular or rectangular as shown in fig. 4(a) and (b).
The calculation unit 250 may be set so that the form of the predetermined region a has a form of a long axis and a short axis or a long axis and a short axis, and may set the long axis or the long axis direction of the predetermined region according to the moving direction of the gaze point P.
In fig. 6, the calculation unit 250 sets the form of the predetermined region a to an elliptical shape. The calculation unit 250 sets the form of the predetermined region a to an elliptical shape based on the movement of the gaze point P acquired by the gaze point movement acquisition unit 240. For example, when the predetermined region a is arranged with the gaze point P as a reference, the calculation unit 250 may be set so that the movement direction of the gaze point P is the major axis direction of the ellipse. Here, the gaze point P is not necessarily the center of the ellipse, and the positional relationship between the gaze point P and the ellipse is set so that the gaze point P is wide in the advancing direction of the movement and is within the ellipse. Accordingly, the video presenter 110 can widely display a video with a maintained image quality in a direction that moves more than the direction in which the gaze point P moves less. The form of the predetermined area a set by the calculation unit 250 is not limited to the ellipse described above, as long as it has a major axis and a minor axis or a major axis and a minor axis. For example, when the calculation unit 250 sets the predetermined area a to be rectangular, in the case of using a compression method in which a plurality of pixels are compressed in units of blocks as one block, the calculation of the overlapping portion of the blocks existing at the boundary between the predetermined area a and the predetermined area a can be simplified as compared with the case where the predetermined area a is elliptical.
The calculation unit 250 may generate a video in which the data amount D per unit pixel number is changed according to the distance from the gaze point P, in addition to the predetermined region a.
Part (a) of fig. 7 is a schematic diagram showing a case where the relationship between the X-axis of the video display area and the data amount D per unit pixel number changes in a plurality of stages. The lower graph of part (a) of fig. 7 is a graph showing the change in the data amount D per unit pixel number on the dot-dash line of the image display area shown above. In the example of fig. 7(a), the calculation unit 250 sets the predetermined region a with reference to the gaze point P. Further, in addition to the boundary defining the area a, a boundary is provided for defining the boundary of the first outer area B1 so as to surround a, and a boundary is provided for defining the second outer area B2 so as to surround B1. The outside of the boundary of the second outer region B2 is defined as B3. As described above, dividing the outer area B into a plurality of areas makes the difference in image quality occurring at the boundary between the predetermined area a and the outer area B smaller than that in the case of no division. Thus, the vision system 1 shown in part (a) of fig. 7 can provide the user 300 with the influence of reducing the amount of data, making it more suitable for human visual recognition, compared to the case where the outer area B is not divided into a plurality of areas.
The calculation unit 250 may generate a video image in which the data amount D per unit pixel number is continuously reduced as the distance from the gaze point P is larger, in addition to the predetermined region a.
Fig. 7(b) is a schematic diagram of a case where the relationship between the X-axis of the image display region and the data amount D per unit pixel number is continuously changed. The calculation unit 250 continuously changes the relationship between the vertical axis and the horizontal axis in part (b) of fig. 7 and sets the relationship as a gradient. Thereby, the difference in image quality in the area boundary that changes the data amount D per unit pixel number becomes small, and a smooth image can be acquired.
The calculation unit 250 may generate a video so that the data amount D per unit pixel number does not fall below the lower limit value DL.
The vertical axis of part (a) and part (b) of fig. 7 shows a lower limit value DL of the data amount D per unit pixel number. In general, when a process of reducing the amount of data in a moving image is performed, there is a possibility that a specific motion may occur in the vicinity of an object boundary on a video image, depending on the method of image processing. Also, it is generally known that the human eye has a reduced vision in the peripheral vision field, but is sensitive to movement. Therefore, the calculation unit 250 generates a video with reference to the lower limit value DL so as not to generate such a video. Thus, the imaging system 1 can provide the user 300 with an image in which the sense of incongruity with respect to the peripheral visual field region is suppressed. The specific lower limit value DL can be determined experimentally in view of image processing and the like suitable for the image display system of the head mounted display 100 and the video generating apparatus 200.
Hereinafter, a use example of the present embodiment will be described with reference to fig. 8 and 9. Fig. 8 is a timing chart for explaining the flow of main processing in the head mounted display 100 and the image generating apparatus 200 according to the embodiment. First, the user 300 wears the head mounted display 100 and views the video presented by the video presentation unit 110. The image capturing unit 120 acquires an image including the eyes of the user 300 (step S101), and the first communication unit 130 transmits the image to the video generating apparatus 200 (step S102).
The second communication unit 210 of the video image generation apparatus 200 receives an image including an eye from the head-mounted display 100 (step S201). The gaze point acquisition unit 230 acquires the gaze point P of the user 300 based on the image (step S202). Then, the communication determination unit 220 determines the communication environment based on the communication parameters (step S203). Details regarding the communication decision are described below. Next, the calculation unit 250 sets the compression rate of the data based on the result determined by the communication determination unit 220 (step S204). The calculation unit 250 acquires video data of a video to be displayed to the user from the storage unit 260 (step S205). Next, the calculation unit 250 acquires information on the gaze point P from the gaze point acquisition unit 230, and sets the predetermined region a with the gaze point P as a reference (step S206). The calculation unit 250 generates a video having a smaller data amount D per unit pixel number than the video calculated for the predetermined area a for the outer area B (step S207). When generating a video image having a small data amount D, the calculation unit 250 refers to the compression rate set based on the communication result to determine the data amount D in the outer area B. Next, the second communication unit 210 transmits the video generated by the calculation unit 250 to the head mounted display 100 (step S208). The first communication unit 130 of the head-mounted display 100 receives the generated video (step S103), and the video presentation unit 110 presents the video to the user 300 (step S104).
Fig. 9 is a flowchart showing an example of a process of communication determination according to the embodiment. The communication determination unit 220 acquires the latest data including at least one communication parameter of the radio wave intensity, the communication speed, the data loss rate, the throughput, the noise situation, or the physical distance from the router (step S211). Next, the communication determination unit 220 calculates an average value of the communication parameters based on the latest data acquired and the past communication information in a predetermined period (step S212). Next, the communication determination unit 220 determines the communication environment based on the calculated average value (step S213). While the video is being reproduced, the video system 1 repeats the processing shown in fig. 8 and 9. In step S213, as described above, the communication determination may be performed based on the latest data of the communication parameters on the head mounted display 100 side and the video image generation device 200 side.
As described above, according to the embodiment, in a state where the image quality of the movie near the gaze point P at which the user is looking is maintained, the image quality at a position distant from the gaze point P is reduced, and since the amount of data transmitted to the head-mounted display 100 by the movie generation apparatus 200 is reduced, a movie with less discomfort can be provided to the user. Further, since the amount of data during communication is small, even when the communication environment is deteriorated, the influence of data transmission delay and the like caused by the deterioration can be reduced. Therefore, the video system 1 of the present invention is suitable for an apparatus for performing interactive communication with the user 300, such as an application program or a game used in a game machine, a computer, a mobile terminal, or the like.
The present invention has been described above based on the embodiments. It will be understood by those skilled in the art that this embodiment is exemplary, various modifications may be made to each of these constituent elements or to each combination of processes, and such modifications are also within the scope of the present invention.
The present invention has been described above based on the embodiments. It will be understood by those skilled in the art that this embodiment is exemplary, various modifications may be made to each of these constituent elements or to each combination of processes, and such modifications are also within the scope of the present invention.
In the above, the description has been given of the case where the gaze point acquisition unit 230 is mounted on the image generation device 200. However, the fixation point acquiring unit 230 is not limited to being mounted on the image generating apparatus 200. For example, the gaze point acquisition part 230 may also be installed on the head mounted display 100. In this case, the head-mounted display 100 is provided with a control function, and a program function for realizing the processing performed in the gaze point acquisition unit 230 may be provided by the control function of the head-mounted display 100. Thus, since the step of transmitting the image including the eyes of the user 300 from the head mounted display 100 to the picture generation apparatus 200 can be omitted, the picture system 1 can suppress the bandwidth of communication or contribute to speeding up the processing.
Description of the reference numerals
1: image system
100: head-mounted display
110: image demonstration part
120: image pickup unit
130: first communication unit
150: frame body
160: wearing piece
170: head earphone
200: image generation device
210: second communication unit
220: communication determination unit
230: gaze point acquisition unit
240: gaze point movement acquisition unit
250: calculating part
260: storage unit
Industrial applicability
The invention can be used in an image system comprising a head-mounted display and an image generating device.
Claims (9)
1. An image system is characterized in that,
the method comprises the following steps:
a head-mounted display used by being mounted on a head of a user; and
an image generating device for generating an image to be presented to the user by the head-mounted display,
the head-mounted display includes:
an image demonstration unit for demonstrating the image to the user;
a photographing section for photographing an image including eyes of the user; and
a first communication unit for transmitting the image captured by the image capturing unit to the image generating device and receiving the image presented by the image presenting unit from the image generating device,
the image generating device includes:
a second communication unit that receives the image captured by the image capturing unit from the head-mounted display and transmits the image to the head-mounted display;
a gaze point acquisition unit configured to acquire a gaze point of the user on the image based on the image captured by the imaging unit;
a gaze point movement acquisition unit that acquires movement of the gaze point of the user based on the gaze point acquired by the gaze point acquisition unit; and
the calculation part is used for calculating the calculation result,
the calculation unit generates the image by setting a predetermined region based on the gaze point acquired by the gaze point acquisition unit, setting a predetermined region having a shape defined by a major axis and a minor axis or a major axis and a minor axis using the gaze point as a reference, setting the direction of the major axis or the direction of the major axis of the predetermined region based on the movement direction of the gaze point acquired by the gaze point movement acquisition unit, and generating an image in which the amount of image data per unit pixel number outside the predetermined region is reduced compared to an image calculated for the inside of the predetermined region.
2. The imaging system according to claim 1, wherein the image generating apparatus further includes a communication evaluating portion for evaluating a communication environment between the first communication portion and the second communication portion,
in the case where the communication environment is deteriorated, the calculation unit may reduce the data amount of the video to be smaller than that used when the communication environment is good.
3. The video system according to claim 2, wherein the communication evaluation unit evaluates the communication environment based on information obtained by combining latest data with one or more communication parameters selected from among radio wave intensity, communication speed, data loss rate, throughput, noise level, and physical distance from the router.
4. The imaging system according to claim 1, wherein the calculation unit adjusts the size and shape of the predetermined region or all of the size and the shape in accordance with the movement of the gaze point.
5. The video system according to claim 1, wherein the calculation unit generates the video by adjusting the data amount per unit pixel based on a distance from the gaze point outside the predetermined region.
6. The video system according to claim 1, wherein the calculation unit generates the video by continuously decreasing the data amount per unit pixel as the distance from the gaze point increases outside the predetermined region.
7. The video system according to claim 1, wherein the calculation unit generates the video without reducing each data amount of the total number of the unit pixels to a lower limit value.
8. An image generation method embodied in an image system, the image system comprising: a head-mounted display used by being mounted on a head of a user; and an image generating device for generating an image to be presented to the user by the head-mounted display, the image generating method being characterized in that,
the method comprises the following steps:
an image demonstration step of demonstrating the image to the user through the head-mounted display;
a photographing step of photographing an image of eyes of a user through the head-mounted display;
a transmission step of transmitting the image photographed in the photographing step to the image generating apparatus through the head-mounted display;
a receiving step of receiving, from the image generating device, an image to be demonstrated in the image demonstrating step through the head mounted display;
a receiving step of receiving, by the video generating device, the image captured in the capturing step from the head-mounted display;
a transmission step of transmitting the image to the head-mounted display through the image generation device;
a gaze point acquisition step of acquiring, by the video generation device, a gaze point of the user in the video based on the image captured in the capturing step;
a gaze point movement acquisition step of acquiring, by the image generation device, movement of the gaze point of the user based on the gaze point acquired in the gaze point acquisition step; and
a calculation step of generating the image by the image generation device,
setting a predetermined region having a shape defined by a major axis and a minor axis or a major axis and a minor axis using the gaze point as a reference based on the gaze point acquired in the gaze point acquisition step, setting a direction of the major axis or a direction of the major axis of the predetermined region based on a movement direction of the gaze point acquired in the gaze point movement acquisition step, and generating an image in which an amount of image data per unit pixel number outside the predetermined region is reduced compared to an image calculated for an inside of the predetermined region.
9. A non-transitory computer readable medium storing an image generation program executed by an image generation device for generating an image to be transmitted to a head-mounted display used in a state of being fixed to a head of a user,
the above procedure includes:
a communication function of receiving the image of the user's eyes photographed by the head mounted display and transmitting the image to the head mounted display;
a gaze point acquisition function for acquiring a gaze point of the user in the video based on the image received by the communication function;
a gaze point movement acquisition function that acquires movement of the gaze point of the user based on the gaze point acquired by the gaze point acquisition function; and
a computing function for generating the above-mentioned image,
a predetermined region having a shape defined by a major axis and a minor axis or a major axis and a minor axis is set based on the gaze point acquired by the gaze point acquisition function using the gaze point as a reference, and the direction of the major axis or the direction of the major axis of the predetermined region is set based on the direction of movement of the gaze point acquired by the gaze point movement acquisition function, thereby generating an image in which the amount of image data per unit pixel number outside the predetermined region is reduced compared to an image calculated for the inside of the predetermined region.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/JP2015/076765 WO2017046956A1 (en) | 2015-09-18 | 2015-09-18 | Video system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108141559A CN108141559A (en) | 2018-06-08 |
| CN108141559B true CN108141559B (en) | 2020-11-06 |
Family
ID=58288481
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201580083269.3A Active CN108141559B (en) | 2015-09-18 | 2015-09-18 | Image system, image generation method and computer readable medium |
Country Status (3)
| Country | Link |
|---|---|
| KR (1) | KR101971479B1 (en) |
| CN (1) | CN108141559B (en) |
| WO (1) | WO2017046956A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200195944A1 (en) * | 2018-12-14 | 2020-06-18 | Advanced Micro Devices, Inc. | Slice size map control of foveated coding |
| US11756259B2 (en) * | 2019-04-17 | 2023-09-12 | Rakuten Group, Inc. | Display controlling device, display controlling method, program, and non-transitory computer-readable information recording medium |
| WO2021066210A1 (en) * | 2019-09-30 | 2021-04-08 | 엘지전자 주식회사 | Display device and display system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6211903B1 (en) * | 1997-01-14 | 2001-04-03 | Cambridge Technology Development, Inc. | Video telephone headset |
| JP2004056335A (en) * | 2002-07-18 | 2004-02-19 | Sony Corp | Information processing apparatus and method, display apparatus and method, and program |
| CN104067160A (en) * | 2011-11-22 | 2014-09-24 | 谷歌公司 | Method of using eye-tracking to center image content in a display |
| CN204442580U (en) * | 2015-02-13 | 2015-07-01 | 北京维阿时代科技有限公司 | A kind of wear-type virtual reality device and comprise the virtual reality system of this equipment |
| CN104767992A (en) * | 2015-04-13 | 2015-07-08 | 北京集创北方科技有限公司 | Head-wearing type display system and image low-bandwidth transmission method |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2868389B2 (en) * | 1993-06-14 | 1999-03-10 | 株式会社エイ・ティ・アール通信システム研究所 | Image display device |
| JP3263278B2 (en) * | 1995-06-19 | 2002-03-04 | 株式会社東芝 | Image compression communication device |
| US9344612B2 (en) * | 2006-02-15 | 2016-05-17 | Kenneth Ira Ritchey | Non-interference field-of-view support apparatus for a panoramic facial sensor |
| JP2008131321A (en) * | 2006-11-21 | 2008-06-05 | Nippon Telegr & Teleph Corp <Ntt> | Video transmission method, video transmission program and computer readable recording medium with the program recorded thereon |
-
2015
- 2015-09-18 KR KR1020187008945A patent/KR101971479B1/en active IP Right Grant
- 2015-09-18 CN CN201580083269.3A patent/CN108141559B/en active Active
- 2015-09-18 WO PCT/JP2015/076765 patent/WO2017046956A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6211903B1 (en) * | 1997-01-14 | 2001-04-03 | Cambridge Technology Development, Inc. | Video telephone headset |
| JP2004056335A (en) * | 2002-07-18 | 2004-02-19 | Sony Corp | Information processing apparatus and method, display apparatus and method, and program |
| CN104067160A (en) * | 2011-11-22 | 2014-09-24 | 谷歌公司 | Method of using eye-tracking to center image content in a display |
| CN204442580U (en) * | 2015-02-13 | 2015-07-01 | 北京维阿时代科技有限公司 | A kind of wear-type virtual reality device and comprise the virtual reality system of this equipment |
| CN104767992A (en) * | 2015-04-13 | 2015-07-08 | 北京集创北方科技有限公司 | Head-wearing type display system and image low-bandwidth transmission method |
Also Published As
| Publication number | Publication date |
|---|---|
| KR101971479B1 (en) | 2019-04-23 |
| KR20180037299A (en) | 2018-04-11 |
| WO2017046956A1 (en) | 2017-03-23 |
| CN108141559A (en) | 2018-06-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9978183B2 (en) | Video system, video generating method, video distribution method, video generating program, and video distribution program | |
| US10965868B2 (en) | Apparatus and methods for image encoding using spatially weighted encoding quality parameters | |
| CN105684074B (en) | Image display device, image display method, image output device, image output method, and image display system | |
| JP7408678B2 (en) | Image processing method and head mounted display device | |
| US9509916B2 (en) | Image presentation method and apparatus, and terminal | |
| US8687053B2 (en) | Stereoscopic image display device | |
| US11836885B2 (en) | Dynamic foveated pipeline | |
| US11196975B2 (en) | Image generating device, image display system, and image generating method | |
| RU2649950C2 (en) | Image display device and image display method, information communication terminal and information communication method, and image display system | |
| CN110537208B (en) | Head-mounted display and method | |
| JP2018141816A (en) | Video system, video generation method, video distribution method, video generation program and video distribution program | |
| EP3065413B1 (en) | Media streaming system and control method thereof | |
| WO2018211672A1 (en) | Image generation device, image display system, and image generation method | |
| JP2012095229A (en) | Image display device and computer program for image display device | |
| CN108141559B (en) | Image system, image generation method and computer readable medium | |
| US20130050427A1 (en) | Method and apparatus for capturing three-dimensional image and apparatus for displaying three-dimensional image | |
| WO2020044916A1 (en) | Information processing device, information processing method, and program | |
| JP2013201688A (en) | Image processing apparatus, image processing method, and image processing program | |
| CN113454706B (en) | Display control device, imaging device, and display control method | |
| WO2018173445A1 (en) | Information processing device, information processing method, information processing system, and program | |
| US20150009297A1 (en) | Terminal device, image shooting system and image shooting method | |
| CN114624875A (en) | Image calibration method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |