JP2017068818A - Device, method, and program for forming images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display additional information in an area not often viewed by a user.SOLUTION: An image forming device comprises; image forming means for forming an image to be displayed on a display; frequency statistical means configured to produce statistical data based on a frequency at which each area of the image being displayed on the display is viewed by a user; low frequency area identification means configured to identify an area of the image with a frequency lower than a threshold as a low frequency area based on the statistical data; additional image forming means configured to form an additional image to be superimposed on the image in the low frequency area; and an image output means for outputting the image and the additional image to the display.SELECTED DRAWING: Figure 6

Description

  The present invention relates to an image generation apparatus, an image generation method, and an image generation program.

  A technique for displaying additional information superimposed on content is known. For example, in Patent Document 1, additional information is displayed in a display area determined according to content characteristics.

Japanese Patent No. 5465620

  Displaying additional information in areas that the user does not see often may create a new user experience.

  The present invention has been made in view of the above points, and one of the purposes thereof is an image generation apparatus, an image generation method, and an image generation apparatus that allow additional information to be displayed in an area that a user does not often see. And providing an image generation program.

  In order to solve the above-described problems, one aspect of the present invention provides image generation means for generating an image to be displayed on a display, and statistics based on a frequency at which a user views each part of the image displayed on the display. Frequency statistical means for creating data, low frequency area identifying means for identifying an area in the image in which the frequency is below a threshold value as a low frequency area based on the statistical data, and the image in the low frequency area And an image output device that outputs the image and the additional image to the display.

  Another aspect of the present invention is an image generation method executed by a computer, the step of generating an image to be displayed on a display, the step of outputting the image to the display, and the user performing the display Creating statistical data based on the frequency of viewing each part of the image displayed on the screen, and identifying, based on the statistical data, an area in the image where the frequency is below a threshold as a low-frequency area And generating an additional image for superimposing and arranging the image in the low-frequency area, and outputting the additional image to the display.

  Further, according to another aspect of the present invention, a computer generates an image to be displayed on a display, a step of outputting the image to the display, and each of the images displayed on the display by a user. Creating statistical data based on the frequency of viewing a portion; identifying an area in the image where the frequency is below a threshold based on the statistical data as a low frequency area; and An image generation program for executing the steps of generating an additional image to be superimposed on an image and outputting the additional image to the display.

  According to the present invention, it is possible to display additional information in an area that the user does not often see. Reference number: 151470 Japanese Patent Application No. 2015-191042 (Proof) Submitted date: September 29, 2015 2

FIG. 1 is a hardware configuration diagram of an image display system 100 according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a functional configuration of the image generation apparatus 200 according to an embodiment of the present invention. FIG. 3 shows an example of statistical data created by the frequency statistics unit 232. FIG. 4 shows an example of statistical data created by the frequency statistics unit 232. FIG. 5 shows an example of a low frequency area. FIG. 6 is a flowchart showing a processing procedure of the image generation apparatus 200 according to an embodiment of the present invention.

[Description of Embodiment of the Present Invention]
First, the contents of the embodiment of the present invention will be listed and described. One embodiment of the present invention has the following configuration.

  (Item 1) Image generation means for generating an image to be displayed on a display, frequency statistical means for generating statistical data based on the frequency with which a user views each part of the image displayed on the display, and the statistical data Based on the above, a low frequency area identifying means for identifying an area in the image in which the frequency is below a threshold as a low frequency area, and an additional image for superimposing the image on the low frequency area are generated. An image generation apparatus comprising: an additional image generation unit configured to output the image and the additional image to the display.

  (Item 2) The image generating device according to item 1, wherein the display is configured as a head mounted display, and the image is a virtual reality image presented to the user wearing the head mounted display.

  (Item 3) The image generation device according to item 1 or 2, wherein the frequency statistics unit calculates the frequency based on the user's gaze direction detected by the gaze direction detection unit.

  (Item 4) The image generation device according to item 1 or 2, wherein the frequency statistics unit calculates the frequency based on an output from a sensor that detects a head orientation of the user.

  (Item 5) The additional image generation means generates the additional image according to the current gaze direction of the user detected by the gaze direction detection means or the current head direction of the user detected by the sensor. Item 5. The image generating device according to Item 3 or 4, which is dynamically changed.

  (Item 6) The low-frequency area identifying means is configured such that an area in the image in which the frequency is lower than a first threshold is a first low-frequency area, the frequency is higher than the first threshold, and is greater than the first threshold. Each of the areas in the image that is less than a second threshold value that is greater than the second threshold value is identified as a second low-frequency area, and the additional image generation unit superimposes the first additional image on the first low-frequency area by superimposing the image. The image generating device according to any one of items 1 to 5, wherein the image generating device is arranged and arranges a second additional image having an attribute value different from that of the first additional image by superimposing the image on the second low-frequency area. .

  (Item 7) An image generation method executed by a computer, the step of generating an image to be displayed on a display, the step of outputting the image to the display, and the user displaying the image displayed on the display Creating statistical data based on the frequency of viewing each part; identifying an area in the image where the frequency is below a threshold based on the statistical data as a low frequency area; and An image generation method comprising: generating an additional image to be arranged so as to be superimposed on the image; and outputting the additional image to the display.

  (Item 8) Generating an image to be displayed on a display on a computer, outputting the image to the display, and statistics based on a frequency at which a user views each part of the image displayed on the display Creating data, identifying, based on the statistical data, an area in the image where the frequency is below a threshold as a low-frequency area, and placing the image on the low-frequency area so as to overlap with the image An image generation program for executing an additional image generation step and a step of outputting the additional image to the display.

[Details of the embodiment of the present invention]
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a hardware configuration diagram of an image display system 100 according to an embodiment of the present invention. The image display system 100 includes a head mounted display (hereinafter referred to as HMD) 120 and an image generation apparatus 200. For example, the HMD 120 and the image generation apparatus 200 are electrically connected by a wired cable 150 and can communicate with each other. Instead of the wired cable 150, a wireless connection may be used.

  The HMD 120 is a display device that is used by being worn on the head of the user 160. The HMD 120 includes a display 122, an eye tracking device (hereinafter referred to as ETD) 124, and a sensor 126. Only one of the ETD 124 and the sensor 126 may be provided alternatively. The HMD 120 may further include a speaker (headphone) and a camera (not shown).

Display 122 is configured to present an image to the field of view of user 160 wearing HMD 120. For example, the display 122 can be configured as a non-transmissive display. In this case, the external scene of the HMD 120 is blocked from the view of the user 160 and only the image displayed on the display 122 is delivered to the eyes of the user 160. For example, an image generated using computer graphics is displayed on the display 122. An example of an image by computer graphics is a virtual reality image obtained by imaging a virtual reality space (for example, a world created by a computer game). Alternatively, the real world may be represented by computer graphics based on position coordinate data such as actual topography and objects in the real world. Furthermore, instead of using computer graphics, an image captured from the viewpoint of the user 160 using a camera (not shown) installed in the HMD 120 may be displayed on the display 122.

The ETD 124 is configured to track the movement of the eyeball of the user 160 and detect in which direction the line of sight of the user 160 is directed. For example, the ETD 124 includes an infrared light source and an infrared camera. The infrared light source irradiates infrared rays toward the eyes of the user 160 wearing the HMD 120. The infrared camera captures an image of the eyes of the user 160 irradiated with infrared rays. Infrared rays are reflected on the surface of the eyes of the user 160, but the reflectance of the infrared rays differs between the pupil and a portion other than the pupil. In the image of the eyes of the user 160 captured by the infrared camera, the difference in infrared reflectance appears as the brightness of the image. Based on this contrast, the user 160
A pupil is identified in the eye image of the user's eye, and the direction of the line of sight of the user 160 is detected based on the position of the identified pupil. The line-of-sight direction of the user 160 indicates which part of the image displayed on the display 122 the user 160 is gazing at.

  The sensor 126 is a sensor for detecting which direction the head of the user 160 wearing the HMD 120 is facing. As the sensor 126, for example, any one of a magnetic sensor, an angular velocity sensor, an acceleration sensor, or a combination thereof may be used. When the sensor 126 is a magnetic sensor, an angular velocity sensor, or an acceleration sensor, the sensor 126 is built in the HMD 120 and outputs a value (magnetism, angular velocity, or acceleration value) corresponding to the direction or movement of the HMD 120. The direction of the head of the user 160 wearing the HMD 120 is calculated by processing the output value from the sensor 126 by an appropriate method. The orientation of the head of the user 160 can be used to change the display image of the display 122 so as to follow the movement when the user 160 moves the head. When the display image on the display 122 is changed in accordance with the movement of the head of the user 160, the direction of the head of the user 160 is a rough guide for a portion of the display image on the display 122 that is relatively likely to be viewed by the user 160. Indicates.

  The sensor 126 may be a sensor provided outside the HMD 120. For example, the sensor 126 may be an infrared sensor that is separate from the HMD 120. By using this infrared sensor to detect an infrared reflective marker provided on the surface of the HMD 120, the orientation of the head of the user 160 wearing the HMD 120 can be specified.

  The image generation device 200 is a device for generating an image displayed on the HMD 120. The image generating apparatus 200 includes at least a processor 202, a memory 204, and a user input interface 208. The image generation apparatus 200 may further include a network interface (not shown) for communicating with other apparatuses via a network, for example, as other components. The image generation device 200 can be realized as, for example, a personal computer, a game console, a smartphone, a tablet terminal, or the like.

  The memory 204 stores at least an operating system and an image generation program. The operating system is a computer program for controlling the overall operation of the image generation apparatus 200. The image generation program is a computer program for the image generation apparatus 200 to realize each function of image generation processing described later. The memory 204 can also store data generated by the operation of the image generating apparatus 200 temporarily or permanently. Specific examples of the memory 204 include a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, a flash memory, and an optical disk.

  The processor 202 is configured to read a program stored in the memory 204 and execute processing according to the program. When the processor 202 executes an image generation program stored in the memory 204, each function of image generation processing described later is realized. The processor 202 includes a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit).

  The user input interface 208 is configured to receive an input for operating the image generating apparatus 200 from a user of the image display system 100. Specific examples of the user input interface 208 are a game controller, a touch pad, a mouse, a keyboard, and the like.

  FIG. 2 is a block diagram illustrating a functional configuration of the image generation apparatus 200 according to an embodiment of the present invention. The image generation device 200 includes a storage unit 220 and a processing unit 230. The processing unit 230 further includes an image generation unit 231, a frequency statistics unit 232, a low frequency area identification unit 233, an additional image generation unit 234, and an image output unit 235. The storage unit 220 corresponds to the memory 204 shown in FIG. The processing unit 230 and each of the units 231 to 235 included in the processing unit 230 are realized by the processor 202 illustrated in FIG. 1 reading and executing the image generation program in the memory 204. Represents the function of.

  The image generation unit 231 generates an image to be displayed on the HMD 120. For example, the image generation unit 231 acquires predetermined data from the storage unit 220 and generates an image by computer graphics processing based on the acquired data. As an example, the image generation unit 231 may generate a virtual reality image that allows the user 160 wearing the HMD 120 to recognize a virtual reality space by a computer game. The virtual reality image represents a scene that the user can see in the virtual reality space. For example, the virtual reality image generated by the image generation unit 231 includes characters appearing in computer games, landscapes such as buildings and trees, interior furniture and wall interiors, items falling on the ground, and user operations. A part of the body of an avatar (hand or foot) or an object (gun or sword) held by the avatar by a hand. Moreover, the image generation part 231 is good also as generating the computer graphics image which reproduced the real world based on the actual topographical data etc. of the real world. Furthermore, the image generated by the image generation unit 231 is not based on computer graphics processing, but may be, for example, a video image taken from the viewpoint of the user 160 by an external camera installed in the HMD 120.

  The image generation unit 231 may further change the image based on the output value from the sensor 126. For example, the image generated by the image generation unit 231 is an image representing a state in which the user's field of view in the virtual reality space changes following the movement of the head of the user 160 indicated by the output value from the sensor 126. Good.

  The image generated by the image generation unit 231 is output to the HMD 120 via the image output unit 235 and displayed on the display 122.

  The frequency statistics unit 232 creates statistical data based on the frequency with which the user 160 wearing the HMD 120 views each part of the image displayed on the display 122. The statistical data indicates which part of the image displayed on the display 122 is often seen and which part is not often seen. For example, the frequency statistics unit 232 creates statistical data on the frequency with which the user 160 views each part of the image based on the line-of-sight direction of the user 160 detected by the ETD 124. Further, the frequency statistics unit 232 may create statistical data of the frequency with which the user 160 views each part of the image based on the head direction of the user 160 detected by the sensor 126. This will be specifically described with reference to FIGS.

  FIG. 3 shows a display image 520 displayed on the display 122 of the HMD 120. Display image 520 includes a plurality of partial regions 501. In the example of FIG. 3, the partial area 501 is a small image range formed by dividing the display image 520 vertically and horizontally in a grid pattern. The user 160 wearing the HMD 120 gazes at a specific partial area 501 in the display image 520 at a certain moment, and gazes at another partial area 501 at another moment. The line of sight of the user 160 moves back and forth between different partial areas 501 of the display image 520 with the passage of time. The line-of-sight direction of the user 160 is detected by the ETD 124 and input to the frequency statistics unit 232.

  In the partial area 501 in FIG. 3, any number from “0” to “5” is written. This number represents the frequency value of the partial area 501 where the number is written. For example, the frequency value “0” indicates that the user 160 does not see the partial area 501 at all. The frequency value “1” indicates that the user 160 has viewed the partial area 501 with a slight frequency, and the frequency value “5” indicates that the user 160 has viewed the partial area 501 with the highest frequency. . The intermediate frequency values “2”, “3”, and “4” indicate the frequency obtained by appropriately dividing the frequency values “1” and “5”. The frequency statistics unit 232 collects the line-of-sight direction of the user 160 input from the ETD 124 for a predetermined time period, and assigns a frequency value to each partial region 501 according to the collection result. For example, the frequency statistics unit 232 assigns a frequency value “5” to the partial region 501 classified into the group with the longest stay time of the line of sight of the user 160 in the time period, and then the group with the longest stay time A frequency value “4” is assigned to the partial region 501 classified as “1”. Similarly, other frequency values are assigned.

  FIG. 4 shows a region 540 that is wider than the range 530 that is displayed on the display 122 of the HMD 120 at a time. The region 540 represents the entire scene that the user can see around him / her in a virtual reality space created by a computer game, for example. The display range 530 represents a limited range of scenes that enter the field of view when the user faces a specific direction in the virtual reality space, and corresponds to the range of the display image 520 in FIG. For example, when the user 160 moves the head in various directions, the display range 530 moves around the area 540 following the movement of the head.

  As shown in FIG. 4, the region 540 includes a plurality of partial regions 501 similar to those described with reference to FIG. 3. The user 160 wearing the HMD 120 turns his head in a certain direction in a certain time zone. While the user 160 turns his head in the direction, a frequency value is assigned to the partial area 501 included in the display range 530 at the same time as described in FIG. In addition, the user 160 turns his head in another direction in another time zone. While the user 160 turns his head in the other direction, a frequency value is similarly given to the partial area 501 included in the display range 530 at that time. When the display ranges 530 in two time zones partially overlap, for example, a frequency value is given to each partial area 501 based on the total time that the user's 160 gaze stays in the overlapping partial area 501. .

  In the example in which the display range 530 moves within the region 540 as shown in FIG. 4, the frequency value of each partial region 501 may be determined based on the orientation of the head of the user 160. For example, when the user 160 moves the head, the orientation of the head of the user 160 is detected by the sensor 126, and an image of the display range 530 corresponding to the orientation is generated by the image generation unit 231. At this time, the actual line of sight of the user 160 may be present at any part in the generated image, but here, as one assumption, the user 160 is not allowed to have a predetermined fixed part within the display range 530 (for example, the display range 530). Center) The frequency statistics unit 232 determines the frequency value of each partial region 501 on the assumption that the direction of the fixed portion of the display range 530 is the line-of-sight direction of the user 160. For example, the frequency statistics unit 232 tracks how the display range 530 moves in the region 540 over a predetermined time period, and the partial region 501 having a long time overlapping the center of the display range 530 has a large frequency value and display range. A small frequency value is given to the partial region 501 that overlaps with the center of 530 for a short time.

  As described above, the frequency values of the partial areas 501 are statistically collected based on the viewing direction or the head direction of the user 160. The collected frequency values of each partial area 501 constitute statistical data. The statistical data may be stored in the storage unit 220.

  Note that the dotted lines (the frame lines of the partial area 501) and the numbers (frequency values) depicted in FIGS. 3 and 4 are shown for the above explanation and are not elements constituting the image itself. Therefore, it should be noted that it is not displayed on the display 122 of the HMD 120.

  The low frequency area identifying unit 233 identifies the low frequency area based on the statistical data created by the frequency statistical unit 232. The low frequency area is a partial area of the image displayed on the display 122 of the HMD 120 that is less frequently seen by the user 160. For example, the low-frequency area identification unit 233 compares the frequency value of each partial area 501 constituting the statistical data with a predetermined threshold value, and if the frequency value of a certain partial area 501 falls below the threshold value as a result of the comparison, The partial area 501 is determined as a part of the low frequency area.

  FIG. 5 shows an example of a low frequency area. This example corresponds to the example of the frequency value of each partial area 501 shown in FIG. The low-frequency area 570 illustrated in FIG. 5 is an area including the partial region 501 to which the frequency value “0” or “1” is assigned in FIG. 3. As another example, the partial region 501 to which the frequency value “0”, “1”, or “2” is assigned may be a low-frequency area.

  Moreover, the low frequency area identification part 233 is good also as classifying a low frequency area into several steps according to a frequency value. For example, the low-frequency area identification unit 233 sets the partial region 501 having the frequency value “0” or “1” as the first low-frequency area, and sets the partial region 501 having the frequency value “2” or “3” as the first. 2 It is good also as a low frequency area.

  The additional image generation unit 234 generates an additional image to be arranged in the low frequency area. The generated additional image is output to the HMD 120 via the image output unit 235 and is displayed on the display 122 so as to be superimposed on the image from the image generation unit 231. The additional image can be used for, for example, presenting an advertisement in a virtual reality space or displaying an enemy character or a useful item in a computer game. Since the additional image is displayed in a low-frequency area where the frequency seen by the user 160 is low, the user 160 can visually recognize the image from the image generation unit 231 without being significantly affected by the additional image. On the other hand, for example, assuming a rule that an additional image having a high value for the user 160 can be displayed in the low-frequency area, the user's 160 attention is given only to the image from the image generation unit 231 (an area other than the low-frequency area). And can be directed to low frequency areas. For example, in an example of a computer game, it is possible to increase the fun of the game by displaying rare items with high value or enemy characters that can be scored high when defeated as additional images in a low-frequency area.

  Further, the additional image generation unit 234 may dynamically change the additional image according to the current gaze direction and head direction of the user 160. For example, a character (a ghost or a mole of a mole-tapping game) as an additional image is first displayed in the low-frequency area. Based on the input from the ETD 124 or the sensor 126, the additional image generation unit 234 determines whether or not the user 160 is about (or is turning) the line of sight (or head) to the low-frequency area. When the user 160 tries to turn his / her line of sight toward the low-frequency area, the additional image generation unit 234 displays the additional image of the character, for example, the character escapes from the line of sight of the user 160 (the ghost disappears from view or the mole moves to the ground. Change to an additional image that can be hidden. The degree to which the character escapes may be adjusted according to the degree to which the user's line of sight approaches the low frequency area.

  Further, when the low frequency area is classified into a plurality of stages, the additional image generation unit 234 also adds a different additional image (first addition) for each classified low frequency area (first low frequency area, second low frequency area, etc.). An image, a second additional image, etc.) may be generated. For example, the attribute value of the first additional image displayed in the first low frequency area is different from the attribute value of the second additional image displayed in the second low frequency area. As an example, when the first low-frequency area corresponds to the frequency values “0” and “1” and the second low-frequency area corresponds to the frequency values “2” and “3”, the frequency seen by the user 160 is more In the first low-frequency area, which is a low area, a rare item with higher value and an enemy character that can obtain a higher score when defeated are displayed as a first additional image, thereby further enhancing the fun of the game. Is possible.

  FIG. 6 is a flowchart showing a processing procedure of the image generation apparatus 200 according to an embodiment of the present invention. First, in step S <b> 602, the image generation unit 231 of the image generation apparatus 200 generates an image to be displayed on the HMD 120. Next, in step S604, the image output unit 235 of the image generation device 200 outputs the image generated by the image generation unit 231 to the HMD 120. Thereby, an image is displayed on the display 122 of the HMD 120. Next, in step S606, the frequency statistics unit 232 of the image generation apparatus 200 creates statistical data based on the frequency with which the user 160 wearing the HMD 120 views each part of the image displayed on the display 122. For example, the frequency statistic unit 232 of the image generation apparatus 200 determines the frequency of each partial region based on the direction of the line of sight of the user 160 detected by the ETD 124 or the direction of the head of the user 160 detected by the output value from the sensor 126. Determine the value. Next, in step S <b> 608, the low frequency area identifying unit 233 of the image processing apparatus 200 identifies a low frequency area that is less frequently seen by the user 160 based on the statistical data created by the frequency statistical unit 232. Next, in step S610, the additional image generation unit 234 of the image generation device 200 generates an additional image to be arranged in the low-frequency area. Next, in step S612, the image output unit 235 of the image generation apparatus 200 outputs the additional image generated by the additional image generation unit 234 to the HMD 120. Thereby, the additional image is displayed in the low frequency area so as to be superimposed on the image on the display 122 of the HMD 120.

  As mentioned above, although embodiment of this invention was described, this invention is not limited to this, A various change is possible within the range which does not deviate from the summary.

100 Image Display System 120 Head Mount Display (HMD)
122 Display 124 Eye tracking device (ETD)
126 Sensor 150 Wired Cable 200 Image Generation Device 202 Processor 204 Memory 208 User Input Interface 220 Storage Unit 230 Processing Unit 231 Image Generation Unit 232 Frequency Statistics Unit 233 Low Frequency Area Identification Unit 234 Additional Image Generation Unit 235 Image Output Unit

Claims (8)

Image generating means for generating an image to be displayed on the display;
Frequency statistics means for creating statistical data based on the frequency with which a user views each part of the image displayed on the display;
A low-frequency area identifying means for identifying an area in the image in which the frequency is below a threshold as a low-frequency area based on the statistical data;
An additional image generating means for generating an additional image to be superimposed on the image in the low-frequency area;
Image output means for outputting the image and the additional image to the display;
An image generation apparatus comprising: The display is configured as a head mounted display,
The image is a virtual reality image presented to the user wearing the head mounted display.
The image generation apparatus according to claim 1. The frequency statistic unit calculates the frequency based on the gaze direction of the user detected by the gaze direction detection unit;
The image generation apparatus according to claim 1. The frequency statistics means calculates the frequency based on an output from a sensor that detects the orientation of the user's head.
The image generation apparatus according to claim 1. The additional image generation unit dynamically changes the additional image according to the current gaze direction of the user detected by the gaze direction detection unit or the current head direction of the user detected by the sensor. Let
The image generation apparatus according to claim 3 or 4. The low-frequency area identifying means includes a second low-frequency area in the image in which the frequency is lower than a first threshold, a second higher in frequency than the first threshold, and a second higher than the first threshold. Each identifying an area in the image that is below a threshold as a second infrequent area;
The additional image generating means arranges the first additional image on the first low-frequency area so as to be superimposed on the image, and superimposes the image on the second low-frequency area on the first additional image. Arranging different second additional images of
The image generation apparatus according to claim 1. An image generation method executed by a computer,
Generating an image for display on a display;
Outputting the image to the display;
Creating statistical data based on the frequency with which a user views each part of the image displayed on the display;
Identifying the area in the image where the frequency is below a threshold as a low frequency area based on the statistical data;
Generating an additional image to be superimposed on the image in the low frequency area;
Outputting the additional image to the display;
An image generation method including: On the computer,
Generating an image for display on a display;
Outputting the image to the display;
Creating statistical data based on the frequency with which a user views each part of the image displayed on the display;
Identifying the area in the image where the frequency is below a threshold as a low frequency area based on the statistical data;
Generating an additional image to be superimposed on the image in the low frequency area;
Outputting the additional image to the display;
An image generation program for executing