JP2024023752A - Data generation device, video system, and data generation method - Google Patents

Abstract

The present invention provides a data generation device that easily generates data for providing accurate three-dimensional images to a user, and a video system that can utilize such data. A data generation device according to the present invention includes an image pair acquisition unit that photographs an eye using visible light and infrared light to acquire a pair of a first visible light photographed image and a first infrared photographed image; The features of the first visible light photographed image and the first infrared photographed image are compared to obtain a feature quantity indicating the relationship between the first visible light photographed image and the first infrared photographed image, and the learned model is and a machine learning unit that generates the data. [Selection diagram] Figure 1

Description

The present invention relates to a data generation device, a video system, and a data generation method for generating data that can be used in a video system such as a head-mounted display.

Currently, in order to provide virtual reality (VR) in games and videos, head-mounted displays that can provide stereoscopic images are becoming increasingly popular. A head-mounted display is an image providing device that provides a three-dimensional image to a user inside a casing that covers the user's head. In such head-mounted displays, there are devices that are configured to change the stereoscopic image, such as by installing a line-of-sight detector that detects the user's line-of-sight direction and increasing the resolution of only the image in that line-of-sight direction. (For example, see Patent Document 1).

To detect the user's line of sight in a head-mounted display, the user's eyes are irradiated with infrared light so that the user does not feel glare, and the position of the pupil, etc. in the image of the eye irradiated with infrared light (infrared image) is detected. This is done by detecting the At this time, in order to accurately understand the relationship between the image obtained by irradiating infrared light and the line of sight direction, it is necessary to collect and analyze infrared light image data of a large number of eyes.
It is necessary to reflect the results of the analysis in the calculation of the line of sight direction. However, simulators that simulate eye movements target visible light images, and there are no simulators that analyze infrared images. For this reason, it has been difficult to acquire infrared light image data of a large number of eyes and use it for line-of-sight detection in a head-mounted display.

Japanese Patent Application Publication No. 2001-134371

The present invention provides a data generation device that facilitates the collection of infrared light image data of a large number of eyes, and an imaging system and data generation method that can utilize such data.

In order to solve the above-mentioned problems, a data generation device according to the present invention provides image pair acquisition for photographing an eye using visible light and infrared light to obtain a pair of a first visible light photographed image and a first infrared photographed image. and a machine learning unit that acquires a feature amount indicating a relationship between the first visible light photographed image and the first infrared photographed image and generates a learned model.

Further, the video system according to the present invention includes a line-of-sight detection unit that detects the direction of the user's line of sight by emitting infrared light, a feature amount of a visible-light photographed image of the eye captured using visible light, and a trained model storage unit that stores a trained model generated based on an infrared photographed image of the same eye taken by The device is configured to detect the direction of the line of sight based on the infrared image obtained by irradiation and the learned model.

Further, the data generation method according to the present invention includes the step of photographing the eye using visible light and infrared light to obtain a pair of a first visible light photographed image and a first infrared photographed image; a step of generating a learned model by acquiring a feature amount indicating a relationship between the image and the first infrared photographed image; a step of inputting a visible light image different from the first visible light photographed image; and a step of inputting a visible light image different from the first visible light photographed image; generating an infrared image based on the image and the learned model.

According to the present invention, it is possible to provide a data generation device that facilitates the collection of infrared light image data of a large number of eyes, and an imaging system that can utilize such data.

FIG. 1 is a schematic diagram illustrating a data generation device 1 according to a first embodiment. FIG. 2 is a conceptual diagram illustrating a procedure for generating a learned model in the machine learning unit 16 and a procedure for generating an infrared image Ro. 3 is a conceptual diagram illustrating an example of a procedure for calculating feature amounts in the machine learning unit 16. FIG. 3 is a conceptual diagram illustrating an example of a method of generating an infrared image Ro based on a visible light image Vo in an infrared image generation unit 18. FIG. FIG. 2 is a schematic diagram illustrating a data generation device 1 according to a second embodiment. FIG. 7 is a conceptual diagram illustrating a procedure for generating a trained model and a procedure for generating an infrared image Ro in the machine learning unit 16 of the second embodiment. 1 is a schematic diagram showing an overview of a video system 2. FIG. 2 is a perspective view schematically showing the configuration of an image presentation section 140 housed in a housing 110. FIG. 2 is a side view schematically showing the configuration of an image presentation section 140 housed in a housing 110. FIG. 2 is a block diagram showing the configuration of main parts of a head-mounted display 100 and a video playback device 200 that constitute a video system 2. FIG. It is a schematic diagram showing a modification.

This embodiment will be described below with reference to the accompanying drawings. In the accompanying drawings, functionally similar elements may be designated by the same number. Although the attached drawings show embodiments and implementation examples in accordance with the principles of the present disclosure, they are for the purpose of understanding the present disclosure, and should not be used to limit the present disclosure in any way. isn't it. The descriptions herein are merely typical examples and do not limit the scope of claims or applications of the present disclosure in any way.

Although the embodiments are described in sufficient detail for those skilled in the art to implement the present disclosure, other implementations and forms are possible without departing from the scope and spirit of the technical idea of the present disclosure. composition·
It is necessary to understand that it is possible to change the structure and replace various elements. Therefore, the following description should not be interpreted as being limited to this.

[First embodiment]
With reference to FIG. 1, a data generation device 1 according to a first embodiment will be described. This data generation device 1 includes an imaging device 11, a CPU 12, a visible light image storage section 13, an infrared image storage section 14, a visible light image input section 15, a machine learning section 16, and a learned model. Storage part 1
7 and an infrared image generation section 18.

The imaging device 11 photographs the eye E, which is the photographing target, using visible light and infrared light, respectively, and creates a visible light photographed image Vj (first visible light photographed image) and an infrared photographed image Rj (first infrared photographed image). ) is an image pair acquisition unit for acquiring pairs of images. As described later, by acquiring a visible light image and an infrared image of the same eye and comparing the feature amounts, useful data can be obtained in imaging systems such as head-mounted displays that detect line of sight using infrared light. It becomes possible to collect.

The imaging device 11 includes, for example, a visible light source 1001, an infrared light source 1002, an objective lens 1003, a dichroic mirror 1004, an imaging lens 1005, a visible image camera 1006, an imaging lens 1007, and an infrared light source 1002. An external camera 1008 is provided.

The visible light source 1001 is a light source that emits visible light for visible light photography, and for example,
Light containing as a component the entire wavelength range of ~780 nm or a part of the wavelength range is irradiated. For color photography, it is preferable to use light having wavelength components in all or most of the wavelength ranges mentioned above. Regarding the wavelength range, it may be limited to a specific wavelength range among the above wavelength ranges, as long as the desired imaging is possible. On the other hand, the infrared light source 1002 is a light source that emits infrared light for infrared photography, and emits light having a wavelength of about 780 to 2500 nm, for example.

The objective lens 1003 is placed in front of the eye E to be photographed, and has a function of condensing light from the eye E. Light exiting from the eye E and condensed by the objective lens 1003 is reflected or transmitted by the dichroic mirror 1004, enters the imaging lens 1005 or 1007, and then enters the visible image camera 1006 or the infrared camera 1008. Dichroic mirror 1004 has the property of reflecting infrared light and transmitting visible light.

The CPU 12 is an example of an arithmetic control unit that controls the entire data generation device 1 .
Instead of the CPU 12, it is also possible to employ a GPU (Graphics Processing Unit). Further, the visible light photographed image storage unit 13 and the infrared photographed image storage unit 14 are storage units that store visible light photographed images and infrared photographed images photographed by the imaging device 11. A visible light photographed image and an infrared photographed image of the same eye E photographed by the imaging device 11 are stored as a pair of image data in association with each other, for example, by a serial number. The method of association is not limited to serial numbers, and for example, association may be performed by making the addresses of storage locations the same.

The visible light image input unit 15 is a data input unit, separate from the imaging device 11, for inputting a visible light image of the eye. The visible light image input unit 15 is a part for inputting an image of the eye, but unlike the imaging device 11, it does not input a pair of a visible light photographed image and an infrared photographed image, but only a visible light photographed image. This is the part where you input. The input visible light photographed image is used as original data for generating an infrared image based on the infrared image generation section 18 as described later. The visible light image input unit 15 may be, for example, an interface that is simply connected to a network and can input an eye image from the outside, or may be a camera separate from the imaging device 11.

The machine learning unit 16, as shown in FIG.
An image pair EPj (j=1 to n) of j and an infrared photographed image Rj is analyzed using a predetermined image analysis method. Then, the machine learning unit 16 calculates the feature quantities Fvj and Frj for each of the visible light photographed image and the infrared photographed image based on the analysis results, and further calculates the difference ΔF (
=Fvj−Frj). The machine learning unit 16 collects data on the difference ΔF in feature amounts, generates a learned model regarding the difference in feature amounts between the visible light image and the infrared image, and stores it in the learned model storage unit 17.

Even if visible light photographed image Vj and infrared photographed image Rj are obtained from the same eye E, visible light photographed image Vj and infrared photographed image Rj differ due to the difference in wavelength between visible light and infrared light. It has features. The machine learning unit 16 collects this feature amount difference ΔF for each image pair EPj,
A learned model is generated based on this difference ΔF. The feature amount difference ΔF is the visible light photographed image V
This is an example of a feature amount indicating the relationship between Rj and an infrared photographed image Rj.

Human eyes differ in eyeball color, pupil size, eye length, height, aspect ratio, color of the white part of the eye, skin color, eyelash length, etc., and these also affect the feature amount. . Therefore, in this data generation device 1, the imaging device 11 images the eyes of at least several dozen people to obtain a pair of a visible light photographed image Vj and an infrared photographed image Rj. Data on the difference F in feature amount between the image taken outside and the image taken outside is collected. The collected data group of differences in feature amounts is subjected to machine learning in the machine learning unit 16.

An example of feature amount calculation will be described with reference to FIG. 3. Here, as an example, a case will be described in which the feature amounts of the visible light photographed image Vj and the infrared photographed image Rj are calculated using the well-known SIFT (Scaled In variance Feature Transform) as a method for calculating local image feature amounts. SIFT
This is just an example, and it goes without saying that other feature extraction methods may be used.

When the visible light photographed image Vj and the infrared photographed image Rj are obtained, feature points are first detected.
For example, as shown in FIG. 3, in a visible light photographed image Vj, feature points Pc1 to Pc4 on the edge of the pupil of the eye, feature points Bc1 to Bc6 on the boundary between the iris and white of the eye, and feature points on the boundary of the eyelid. Lc1 to Lc4, etc. are detected. Furthermore, in the infrared photographed image Rj, feature points Pi1 to Pi4 on the edge of the pupil of the eye, feature points Bi1 to Bi6 on the boundary between the iris and white of the eye, and feature points Li1 to Li4 on the boundary of the eyelid are detected. . For example, feature points can be detected by repeatedly performing Gaussian smoothing on the original image, obtaining a difference image thereof, and searching for a maximum change in the difference image.

When feature points are detected, gradient information around each feature point is acquired and calculated as a feature quantity. Then, the set of feature values of these feature points is defined as the feature values Fvj, Fr of each image.
j, and further calculate the difference ΔF between the two.

In this way, visible light photographed images Vj and infrared photographed images Rj are obtained for a plurality of eyes Ej (j=1 to n), and a data set of feature quantities Fvj, Frj, and difference ΔF is obtained for each. Then, using these data, it is possible to determine what kind of feature difference Δ is in what kind of eyes.
A trained model indicating whether F tends to be obtained is obtained as a result of machine learning in the machine learning unit 16. This trained model is stored in the trained model storage section 17.

The trained model can be made more accurate by obtaining visible light images Vj and infrared images Rj of a large number of eyes E and calculating feature amounts for each. However, in order to accommodate all potential users of the head-mounted display, thousands or tens of thousands of data pairs are required, but acquiring data from the imaging device 11 for such a large number of eyes is difficult. Although it is possible, it requires a long time and cost.

Therefore, in the data generation device of this first embodiment, as shown in FIG.
1, after obtaining an image pair EPj of a visible light photographed image Vj and an infrared photographed image Rj for a plurality of eyes E and generating a trained model based on this, a visible light image Vo obtained separately from the imaging device 11 is generated. is input from the visible light image input section 15. Furthermore, this visible light photographed image Vo
is input to the machine learning unit 16. The machine learning unit 16 uses this newly input visible light image V.
The infrared image generation unit 18 calculates the feature amount Fo of o, and generates the corresponding infrared image Ro based on the feature amount Fo and the learned model stored in the learned model storage unit 17. .
The input visible light image Vo and the generated infrared image Ro are treated in the same way as the image pair EPj of the visible light image Vj and the infrared image Rj captured by the imaging device 11, and are used as the learned image. It is also used as data for updating (reconfiguring) the model.

A method for generating an infrared image Vo based on a visible light image Vo will be described with reference to the conceptual diagram of FIG. 4. In FIG. 4, black dots and white dots indicate a visible light photographed image Vj and an infrared photographed image Rj obtained by the imaging device 11, respectively. As shown in the diagram on the left side of FIG. 4, a large number of image pairs EPj, which are paired data of a plurality of visible light photographed images Vj and infrared photographed images Rj, are acquired. When several tens to hundreds of image pairs EPj captured by the imaging device 11 are obtained, a learned model M1 is generated in the machine learning unit 16 based on the image pairs EPj.

Thereafter, when the visible light image Vo is separately obtained from the visible light image input section 15, the machine learning section 16
calculates the feature value Fo of the visible light image Vo, and converts the visible light photographed image Vjad having a feature value similar to the feature value Fo into the image pair EP used to generate the trained model M1.
j, and further extracts the corresponding infrared photographed image Rjad. Here, the visible light photographed image Vjad and the infrared photographed image Rjad to be extracted may be one set or a plurality of sets. Here, in order to simplify the explanation, the explanation will be given assuming that there is only one.

The infrared image generation unit 18 generates an infrared image Ro based on the difference ΔFad in the feature amount between the extracted visible light photographed image Vjad and the infrared photographed image Rjad. That is, visible light image V
The infrared image Ro is determined such that the feature amount difference ΔFo between the infrared image Ro and the infrared image Ro is the same as or approximates the feature amount ΔFad described above. In this way, the visible light image Vo and the infrared image Ro
Based on the combination, a new learned model M1' is reconstructed in the machine learning unit 16. The visible light image Vo and the infrared image Ro generated in this way are stored in the visible light image storage section 13 and the infrared image storage section 14, respectively. When storing both image data, data (serial number, etc.) for linking the two is added to both image data.

As explained above, according to the data generation device of the first embodiment, an image pair EPj of a visible light photographed image Vj and an infrared photographed image Rj is acquired for a plurality of eyes, and this image pair EP
Based on j, a difference ΔF in feature amounts indicating the relationship between the visible light photographed image Vj and the infrared photographed image Rj is obtained. A learned model is generated based on this set of differences ΔF.

Furthermore, according to the first embodiment, the image pair EP obtained by photographing with the imaging device 11
j, but also a visible light image Vo obtained separately from the imaging device 11, and an infrared image Ro corresponding to the input visible light image Vo can be estimated from the obtained trained model. can. The learned model can be reconstructed (corrected) using the image pair of the visible light image Vo and the infrared image Ro obtained in this way. As described later, the generated image pair of the visible light image Vo and the infrared image Ro can be used for line-of-sight detection in a video system such as a head-mounted display, similarly to the captured image pair EPi.

[Second embodiment]
With reference to FIG. 5, a data generation device 1 according to a second embodiment will be described. In FIG. 5, the same reference numerals are given to the same components as in FIG. 1, so redundant explanation will be omitted below. The data generation device 1 of this second embodiment further includes an attribute information storage section 19 and an attribute information input section 20 in addition to the components of the first embodiment.

The attribute information storage unit 19 stores attribute information (for example, eye color, skin color, eye width, height, difference,
This is a memory unit for storing corneal curvature, medical history, etc.). This attribute information may be supplied from the imaging device 11 or may be received from a communication unit (not shown), and any input method may be used. This attribute information is used by the machine learning unit 16 as original data for machine learning, together with the visible light photographed image Vj and the infrared photographed image Rj. By associating not only the visible light photographed image Vj but also the attribute information as learning data, it becomes possible to perform learning that more accurately reflects the characteristics of the eye E, which is the object of photographing.

Further, in this second embodiment, the visible light image Vo input from the visible light image input section 15
It also has an attribute information input section 20 for inputting attribute information. This attribute information input section 20
The attribute information input from is used together with the visible light image Vo in the infrared image generation unit 18 to generate an infrared image Ro to be paired with the visible light image Vo. By using the attribute information as input data in generating the infrared image Ro, it is possible to generate the infrared image Ro while more reflecting the characteristics of the visible light image Vo.

As shown in FIG. 6, the machine learning unit 16 uses the obtained visible light photographed image Vj and infrared photographed image R.
The image pair EPj (j=1 to n) with j is analyzed using a predetermined image analysis method, the feature quantities Fvj and Frj are calculated for each of the visible light photographed image and the infrared photographed image, and two further A difference ΔF (=Fvj−Frj) between feature amounts is calculated. The machine learning unit 16 collects the data of the feature difference ΔF in association with the corresponding attribute information, and creates a visible light photographed image Vj and an infrared photographed image R.
A trained model regarding the difference in feature amounts of j and attribute information is generated and stored in the trained model storage unit 17.

After the trained model is generated, the visible light image Vo obtained separately from the imaging device 11 is input from the visible light image input section 15, and its attribute information is also input from the attribute information input section 20. The machine learning unit 16 calculates the feature amount of this newly input visible light image Vo, and also analyzes its attribute information. The infrared image generation unit 18 generates a corresponding infrared image Ro based on the feature amount, the learned model, and the attribute information of the visible light image Vo. The input visible light image Vo and the generated infrared image Ro are treated in the same way as the image pair EPj of the visible light image Vj and the infrared image Rj captured by the imaging device 11, and are used as the learned image. It is also used as data to update the model.

As explained above, according to the second embodiment, it is possible to obtain the same effects as the first embodiment, and furthermore, by using attribute information to generate a trained model, it is more eye-catching. It is possible to generate a trained model that accurately reflects the characteristics.

[Usage form of trained model]
As described above, the data (trained model) generated by the data generation device 1 of the above embodiment can be effectively used in a video system such as a head-mounted display, particularly in a video system having a line of sight detection device. .

FIG. 7 is a diagram schematically showing an overview of the video system 2. As shown in FIG. Video system 2 according to the embodiment includes a head-mounted display 100 and a video playback device 200. As shown in FIG. 1, the head mounted display 100 may be a shielded head mounted display that is used by being attached to the user's head.

Video playback device 200 generates video data to be displayed by head-mounted display 100 and transmits it to head-mounted display 100 via wired or wireless communication. As an example, the video playback device 200 is a device capable of playing back video such as a stationary game machine, a portable game machine, a personal computer, a tablet terminal, a smartphone, a phablet, a video player, a television, or the like. Wireless connection between the video playback device 200 and the head-mounted display 100 can be realized using, for example, known wireless communication technologies such as Wi-Fi (registered trademark), Bluetooth (registered trademark), and infrared communication. In addition, head mounted display 10
0 and the video playback device 200 using Miracast (trademark), Wi
It is executed according to standards such as Gig (trademark), WHDI (trademark), or Ethernet (trademark). Is the video playback device 200 configured integrally with the head mounted display 100?
It may be built into the head mounted display 100.

The head mounted display 100 includes a housing 110, a head fixing section 120, and headphones 130. The housing 110 has a display, a transmission module,
Accommodates various sensors. The head fixing section 120 is a member that attaches the head mounted display 100 to the user's head. Headphones 130 output the audio of the video played by video playback device 200.

FIG. 8 is a perspective view schematically showing the configuration of the image presentation unit 140 housed in the housing 110.
FIG. 9 is a side view thereof. The presentation section 140 includes an infrared light source 141, an image display section 142, a hot mirror 143, a convex lens 144, a camera 145, and an image output section 146.

The infrared light source 141 is a light source for the line of sight detection device, and is a near infrared light source (780 nm to 250 nm).
This is a light source that can emit light in a wavelength band of about 0 nm). The image display unit 142 is a display that displays images to be presented to the user. The image displayed by the image display section 142 is
A GPU (Graphic Processing Unit, not shown) in the video playback device 200 generates the image. The image display section 142 can be realized using, for example, a known liquid crystal display or organic EL display.

The hot mirror 143 is arranged between the image display section 142 and the user's eyes EL and ER when the user wears the head-mounted display 100. The hot mirror 143 has a characteristic of transmitting visible light emitted by the image display section 142 while reflecting near-infrared light.

The convex lens 144 is arranged on the opposite side of the image display section 142 with respect to the hot mirror 143. In other words, the convex lens 144 is placed between the hot mirror 143 and the user's eyes EL and ER when the user wears the head-mounted display 100. The convex lens 144 collects the light transmitted through the hot mirror 143 for image display.

The infrared light that reaches the user's eyes EL and ER from the infrared light source 141 is reflected by the eyes EL and ER, and heads toward the convex lens 144 again. The infrared light transmitted through the convex lens 144 is reflected by the hot mirror 143. On the other hand, the camera 145 includes a filter (not shown) that blocks visible light, and images the near-infrared light reflected by the hot mirror 143. That is, camera 1
45 is a near-infrared camera that images near-infrared light emitted from the infrared light source 103 and reflected by the user's eyes EL and ER.

The image output unit 146 outputs the image captured by the camera 145 to the line-of-sight detection unit 203 (described later) provided in the video playback device 200. Specifically, the image output unit 146
5 transmits the captured image to the video playback device 200. The line of sight detection unit 203 detects the eyes EL and ER.
The direction of the user's line of sight can be detected from the positional relationship between the bright spot and the bright spot caused by near-infrared light. The line of sight detection unit 203 is realized by a line of sight detection program executed by a CPU (Central Processing Unit) of the video playback device 200. In addition, head mounted display 1
If the computer 00 has computational resources such as a CPU and memory, the CPU of the head-mounted display 100 may execute a program that implements the line-of-sight detection unit.

FIG. 10 is a block diagram showing the configuration of main parts of the head-mounted display 100 and the video playback device 200 that constitute the video system 2. As shown in FIG.
The head mounted display 100 includes a control section 101, a communication section 102, and an infrared light source 1.
41, an image display section 142, a camera 145, and an image output section 146. On the other hand, the video playback device 200 includes a control section 201, a communication section 202, a line of sight detection section 203, a video generation section 204, and a learned model storage section 205. Although not shown, the head-mounted display 100 may include a gyro sensor or the like for detecting the tilt direction of the head-mounted display 100.

The control unit 101 controls various signal processing, signal input/output, etc. in the head mounted display 100. The communication unit 102 and the communication unit 202 are connected to the head mounted display 1.
00 and the video playback device 200. The line of sight detection unit 203 is
According to the signal obtained from the head-mounted display 100, the line-of-sight directions of the eyes EL and ER are detected. When an image of the eye E irradiated with infrared light from the infrared light source 141 is captured by the camera 145, the image is outputted by the image output unit 146 and sent to the line of sight detection unit 203 via the communication units 102 and 202. is input. The line of sight detection unit 203 can detect the direction of the line of sight by determining the position of the irradiation spot in the image of the eye E, etc. The line-of-sight detection unit 203 calculates the line-of-sight direction according to a calculation formula determined based on the data X of the infrared image of the eye E illuminated with infrared light and the trained model stored in the trained model storage unit 205. do. The trained model here is generated in correspondence with the trained model generated by the data generation device 1 of the embodiment described above and stored in the trained model storage section 17. Based on the data on the difference in feature amounts between the visible light image and the infrared image of the image pair EPj generated by the data generation device 1, a trained model indicating the relationship between the infrared image and the line of sight detection direction is generated by an external computer. , or is calculated within the head-mounted display 100 and stored in the learned model storage unit 205.

As an example, the line-of-sight direction data G calculated by the line-of-sight detection unit 203 is calculated using the formula G=a1f1(X)+a2f2(X)+...+b using data X of an infrared image obtained by the camera 145. Consider the case where However, a1, a2, . ．． ．． , b are coefficients determined by the learned model. By acquiring the trained model, the coefficients a1, a2, . ．． ．． ,
When the value of b is determined and infrared image data X is obtained from the camera 145, the line-of-sight direction vector G is determined.

The video generation unit 204 generates the image display unit 1 according to a detection signal from a gyro sensor (not shown), etc.
It has a function of generating an image to be displayed at 42. Note that the video generation unit 204 may change the video to be generated according to the detection result of the line of sight detection unit 203 described above in addition to the output of a gyro sensor or the like.

[Modified example]
With reference to FIG. 11, a modification of the data generation device 1 of the embodiment will be shown. Note that in FIG. 11, components that are the same as or correspond to those in FIG. 1 are given the same or corresponding reference numerals in FIG. 11, and therefore, redundant explanation will be omitted below. In this modification, the data generation device 1 is divided into a first data generation device ML1 and a second data generation device ML2, and the two devices ML1 and ML2 are connected via a network NW.

The first data generation device ML1 acquires the image pair EPj, acquires its feature amount, and generates a trained model based on the feature amount. The second data generation device ML2 generates an infrared image Ro based on another visible light image Vo based on the learned model. First data generation device M
L1, second data generation device ML2, and video system 2 are connected via a network NW and perform data transmission and reception.

The first data generation device ML1 includes an imaging device 11, a CPU 12A, a visible light image storage section 13A, an infrared image storage section 14A, a machine learning section 16, and a learned model storage section 1.
7. The first data generation device ML1 acquires a pair of a visible light photographed image Vj and an infrared photographed image Rj captured by the imaging device 11, and stores them in a visible light photographed image storage section 13A and an infrared photographed image storage section 14A, respectively. let Thereafter, the machine learning unit 16 and the learned model storage unit 17A store the visible light photographed image Vj and the infrared photographed image R in the same manner as in the first embodiment.
The difference between the feature amounts of j is calculated, and based on this, a trained model is obtained and stored in the trained model storage section 17A.

The second data generation device ML2 receives the image pair EPj (visible light image Vj and infrared image Rj) and the above-mentioned learned model from the first data generation device ML1 via the network NW. The second data generation device ML2 includes a CPU 12B, a visible light image storage section 13B, an infrared image storage section 14B, a visible light image input section 15, a machine learning section 16,
It includes a trained model storage section 17B and an infrared image generation section 18.

The received visible light photographed image Vj and infrared photographed image Rj are stored in the visible light photographed image storage section 13B and infrared photographed image storage section 14B, respectively, and the learned model is stored in the learned model storage section 17B. .

Thereafter, when another visible light image Vo is input from the visible light image input section 15, an infrared image Ro is generated by the infrared image generation section 18 in the same manner as in the first embodiment, and the visible light image Vo and infrared image Ro are stored as a pair of images. The trained model is corrected based on this newly generated image pair. The obtained trained model and image pairs are directly or indirectly provided to the video system 2 and used for calculation of line of sight detection.

Although several embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

1...Data generation device, 2...Video system, 11...Imaging device, 12...CPU, 13
...Visible light image storage unit, 13...Visible light image storage unit, 14...Infrared image storage unit, 15...Visible light image input unit, 16...Machine learning unit, 17...Learned model storage unit,
18... Infrared image generation section, 19... Attribute information storage section, 20... Attribute information input section, 100...
Head mounted display, 101... Control unit, 102... Communication unit, 103... Infrared light source, 110... Housing, 120... Head fixing unit, 130... Headphones, 140... Image presentation unit, 141... Infrared light source, 142... Image display section, 143...Hot mirror, 14
4...Convex lens, 145...Camera, 146...Image output unit, 200...Video playback device,
201...Control unit, 202...Communication unit, 203...Line of sight detection unit, 204...Video generation unit,
205...Learned model storage unit, 1001...Visible light source, 1002...Infrared light source,
1003...Objective lens, 1004...Dichroic mirror, 1005...Imaging lens,
1005...Condensing lens, 1006...Visible image camera, 1007...Imaging lens, 1
008...Infrared camera.

Claims (11)

an image pair acquisition unit that photographs the eye with visible light and infrared light to acquire a pair of a first visible light photographed image and a first infrared photographed image;
A data generation device comprising: a machine learning unit that generates a trained model by acquiring feature amounts indicating a relationship between the first visible light photographed image and the first infrared photographed image. a visible light image input unit that inputs a visible light image different from the first visible light photographed image;
The data generation device according to claim 1, further comprising: an infrared image generation unit that generates an infrared image based on the visible light image and the learned model. The data generation device according to claim 2, wherein the machine learning unit reconstructs the learned model based on the pair of the visible light image and the infrared image. The infrared image generation unit acquires the feature amount of the visible light image, identifies the first visible light photographed image having a feature amount that approximates the feature amount of the visible light image, and The data generation device according to claim 3, wherein the infrared image is generated based on a difference in feature amount between a visible light photographed image and the paired first infrared photographed image. The data generation device according to claim 1, wherein the machine learning unit generates the learned model based on the feature amount and attribute information of the first visible light photographed image and the first infrared photographed image. a visible light image input unit that inputs a visible light image different from the first visible light photographed image;
an attribute information input unit for inputting attribute information of the visible light image;
The data generation device according to claim 5, further comprising: an infrared image generation unit that generates an infrared image based on the visible light image, the attribute information, and the learned model. a line-of-sight detection unit that detects the direction of the user's line-of-sight by emitting infrared light;
Store a trained model that is generated based on the feature values of a visible light image of an eye captured with visible light and the feature values of an infrared image of the same eye captured with infrared light. Equipped with a trained model storage unit and
The visual line detection unit is configured to detect the direction of the visual line according to an infrared image obtained by irradiating the infrared light and the learned model. photographing the eye with visible light and infrared light to obtain a pair of a first visible light photographed image and a first infrared photographed image;
generating a trained model by acquiring feature amounts indicating the relationship between the first visible light photographed image and the first infrared photographed image; and inputting a visible light image different from the first visible light photographed image. step and
generating an infrared image based on the visible light image and the learned model;
A data generation method with 9. The data generation method according to claim 8, further comprising the step of reconstructing the learned model based on the pair of the visible light image and the infrared image. The step of generating the infrared image includes acquiring the feature amount of the visible light image, identifying the first visible light photographed image having a feature amount that approximates the feature amount of the visible light image, and The data generation method according to claim 9, wherein the infrared image is generated based on a difference in feature amount between a first visible light photographed image and the paired first infrared photographed image. The data according to claim 8, wherein the step of generating the learned model generates the learned model based on the feature amount and attribute information of the first visible light photographed image and the first infrared photographed image. Generation method.