JP2022117780A - Image generation method and learned neural network model generated by image generation method thereof, image generator, image generation system, and image generation program - Google Patents

Abstract

To provide an image generation method, an image generator, etc. which output a correction image which is corrected according to a visual characteristic of each of refraction disorder persons, and can be seen clearly and normally even by a naked eye, with respect to various images and moving pictures, at a high speed.SOLUTION: An image generator 1 includes: a conversion unit 200 which applies image processing to an input image, by using a neural network; a blurring unit 300 which additionally applies image processing to the input image to which the image processing has been applied by the conversion unit 200, by using a blur reproduction mathematical model, and outputs a perception simulation image subjected to blurring processing according to a visual characteristic of a refraction disorder person; and an updating unit 400 which updates parameters of the neural network, by the comparison between the input image and the perception simulation image, and generates a learned neural network.SELECTED DRAWING: Figure 1

Description

In the present invention, even if a person with refractive error such as myopia, hyperopia, or astigmatism sees a blurry image with the naked eye, it can be clearly seen normally even with the naked eye by correcting it (from the eyes of the person with refractive error). This relates to an image generation method for generating a perceptual image (which is an image that can be clearly and normally perceived), a trained neural network model generated by the image generation method, an image generation device, an image generation system, an image generation program, etc. be.

In recent years, personal computers, smartphones, televisions, and the like have become widespread, and the amount of time people spend looking at these display screens has increased. In addition, with the aging of the population in recent years, the number of elderly people, that is, people with presbyopia is also increasing. People with refractive error, such as myopia, astigmatism, and presbyopia, who have difficulty adjusting the focus of their eyes when trying to see an object in front of them, see the display screens of computers and smartphones as blurry with the naked eye. end up

Here, for example, Patent Documents 1 and 2 disclose techniques for making the display screen easier to see. Patent Literature 1 describes a data editing device that can analyze the legibility of an image when a web page is displayed on the screen, and make appropriate corrections. Specifically, Patent Document 1 describes a rendering unit that renders an HTML document to generate an image, and processes the image generated by the rendering unit to determine how the image looks based on predetermined visual characteristics. An image processing unit that simulates and evaluates, and a result presenting unit that specifies HTML elements to be corrected in the HTML document to be processed based on the evaluation result of the image processing unit and presents them to the web page creator. A data editing device is described in which a result presenting unit retrieves a correction method for an HTML element to be corrected from a symptom model storage unit, and a document correction processing unit actually corrects the HTML document.

Further, Patent Literature 2 describes a super-resolution apparatus that performs iterative calculations and obtains a visually fine high-resolution image based on an input low-resolution image.

Japanese Patent Application Laid-Open No. 2004-348211 JP 2019-023798 A

However, the inventions according to Patent Documents 1 and 2 have high versatility because it is necessary to perform image processing for each image viewed by a person with refractive error each time, and there is a possibility that the images that can be handled may be limited. I can't say. In other words, it is difficult to apply image processing to various images and videos, such as images of natural objects and images of characters, by actually applying them to terminals such as personal computers and smartphones.

Therefore, the present invention provides an image generation method for quickly outputting (generating) perceptual images that are corrected according to the visual characteristics of each person with refractive error and that are clearly normal even to the naked eye, even for various images and moving images. and a trained neural network model generated by the image generation method, an image generation device, an image generation system, an image generation program, and the like.

The image generation method of the present invention uses a neural network to perform image processing on an input image, and uses a blur reproduction mathematical model to further perform image processing on the input image, so as to obtain an image that corresponds to the visual characteristics of a person with refractive error. A learning step including outputting a blurred perceptually simulated image, updating the parameters of the neural network by comparing the input image and the perceptually simulated image, and performing the neural network after the parameters have been updated. applying image processing to the input image using the inference step to output a perceptual image that is normally perceptible to the eye of the refractive error patient.

Further, the image generating apparatus of the present invention includes a conversion unit that performs image processing on an input image using a neural network, and further image processing on the input image that has been subjected to image processing by the conversion unit using a blur reproduction mathematical model. is applied, and the blurring unit outputs a simulated perceptual image that has undergone blurring processing according to the visual characteristics of a person with refractive error. By comparing the input image and the simulated perceptual image, the parameters of the neural network are updated and trained. and an updater that generates a neural network of .

According to the configuration of the image generating method and the image generating apparatus of the present invention, the input image is subjected to image processing by the neural network, and then the sensory simulated image is further subjected to the blur processing by the blur reproduction mathematical model. The parameters of the neural network are updated by comparing the image and the perceptually simulated image, ie, so that the error between the input image and the perceptually simulated image is minimized. After the parameters have been updated, the neural network outputs a perceived image, which is an image that can be clearly and normally perceived by the eyes of a person with refractive error.

Further, the blur reproduction mathematical model preferably outputs a perceptually simulated image that is perceived as blurred when a person with refractive error sees the input image. As a result, a simulated perceptual image corresponding to the blurring seen by the eyes of the refractive error person is output, so that the parameters of the neural network can be updated according to the blurring seen by the eyes of the refractive error patient.

Also, in the learning step, the parameters of the neural network are updated so as to minimize the error between the input image and the perceptually simulated image, and the error is obtained by a loss function based on the pixel values of the input image and the perceptually simulated image. and it is desirable to repeat the learning step until the error obtained by the loss function satisfies a predetermined condition. This makes it possible to further reduce the error between the input image and the perceptually simulated image.

In addition, the trained neural network model of the present invention (hereinafter sometimes simply referred to as "trained model") performs image processing on an input image using a neural network, uses a blur reproduction mathematical model, Further image processing is applied to the input image to output a simulated perceptual image that has undergone blurring processing according to the visual characteristics of a person with refractive error.By comparing the input image and the simulated perceptual image, the parameters of the neural network are updated. is a trained neural network model generated by an image generation method including:

Further, the image generation program of the present invention includes a conversion unit that performs image processing on an input image using a neural network, and further image processing on the input image that has been subjected to image processing by the conversion unit using a blur reproduction mathematical model. is applied, and the blurring unit outputs a simulated perceptual image that has undergone blurring processing according to the visual characteristics of a person with refractive error. By comparing the input image and the simulated perceptual image, the parameters of the neural network are updated and trained. and an updating unit that generates a neural network of .

Further, the image generation system of the present invention includes the image generation device and a terminal connected to the image generation device for transmitting an input image to the image generation device. a terminal including a display that receives and displays the
Furthermore, the terminal includes a perceptual image generator that performs image processing on the input image using the trained neural network generated by the updater and outputs a perceptual image that can be normally perceived by the eyes of a person with refractive error.

(1) According to the image generation method and the image generation device of the present invention, based on the above-described configuration, the input image is subjected to image processing by the neural network, and then the blur processing is further performed by the blur reproduction mathematical model. A simulated image is output, and the parameters of the neural network are updated by comparing the input image and the perceptually simulated image, that is, by minimizing the error between the input image and the perceptually simulated image. After the parameters have been updated, the neural network outputs a perceived image, which is an image that can be clearly and normally perceived by the eyes of a person with refractive error. Once the parameters have been updated (learned), the trained model after the parameters have been updated can be used to rapidly generate perceptual images based on the input image for various images and videos. Output can be done.

(2) In addition, the blur reproduction mathematical model is configured to output a perceptually simulated image that is perceived as being blurred when a person with refractive error sees an input image. Since the image is output, it is possible to update the parameters of the neural network according to the blurring seen from the eye of the person with refractive error, and the eye of the person with refractive error can be updated according to the type of refractive error and refractive power. It is possible to output an appropriate perceptual image according to how it is seen from.

(3) In the learning step, the parameters of the neural network are updated so that the error between the input image and the perceptually simulated image is minimized. By repeating the learning step until the error obtained by the loss function satisfies a predetermined condition, the error between the input image and the perceptual simulated image can be made smaller, and the accuracy is higher ( It can output a perceptual image that appears clearer and normal from an ametropic eye).

It should be noted that a terminal to which the learned model is deployed and a computer on which the image generation program is executed can provide the same effects as the image generation method and image generation device of the present invention.

Furthermore, an image generation device including a conversion unit that performs image processing on an input image using a trained neural network generated by the update unit and outputs a perceptual image that can be normally perceived by the eyes of a person with refractive error; a terminal that is connected to the image generation device and that transmits an input image to the image generation device, and that includes a display unit that receives and displays a perceived image corresponding to the input image output by the image generation device. According to the system, the user can receive the perceptual image output by the conversion unit and display it on their own equipment, or use a terminal that includes a display unit, improving convenience. do.

1 is a schematic configuration diagram showing an image generation device according to an embodiment of the present invention; FIG. 1 is an explanatory diagram for explaining an image generation device according to an embodiment of the present invention; FIG. It is an explanatory view explaining the function of a blurring part. It is a flow chart of the image generation method according to the embodiment of the present invention, (A) is a flow chart in the learning step, (B) is a flow chart in the inference step. It is explanatory drawing which shows the teacher image data set used by the Example of this invention, (A) is a natural image data set, (B) is an English image data set. 4 is a graph showing changes in loss values in the learning step of the embodiment of the present invention; It is explanatory drawing which shows the result of Example 1 of this invention. It is explanatory drawing which shows the result of Example 1 of this invention. It is explanatory drawing which shows the result of Example 2 of this invention. It is explanatory drawing which shows the result of Example 2 of this invention. 1 is a schematic configuration diagram showing an image generation system according to an embodiment of the present invention; FIG.

Embodiments of the present invention will be described in detail below. is not limited to the contents of

[Image generation device]
FIG. 1 is a schematic configuration diagram showing an image generation device according to an embodiment of the present invention. Also, FIG. 2 is an explanatory diagram for explaining the image generation device according to the embodiment of the present invention. As shown in FIG. 2, the image P201 is blurred like the image P202 to the naked eye of the refractive error person (perceived by the eyes of the refractive error person to be blurred). Therefore, the image generating apparatus 1 converts the image P201 into the image P201 so that the person with refractive error can clearly see normal images like the image P212 even when the person is unaided (perceived clearly and normally by the eyes of the person with refractive error). A processed and corrected image P211 is generated.

As shown in FIG. 1, the image generation device 1 includes a storage unit 100, a conversion unit 200, a blurring unit 300, and an updating unit 400. FIG.

The storage unit 100 is an area that stores input images, trained models, and other temporary information.
Specifically, in the learning step of the image generation method of the present invention,
(1) Storage of image data (input image, sensory simulation image, etc.).
(2) Storage of the blur reproduction mathematical model.
(3) storage of neural network models;
(4) Memorization of learned models (including those in the process of learning).
and so on. Also, in the inference step of the image generation method of the present invention,
(1) Storage of trained models.
and so on.

The conversion unit 200 performs image processing on an input image using a neural network (hereinafter sometimes referred to as “NN”). The NN is, for example, a CNN (Convolutional Neural Network), an autoencoder, or the like.

A blurring unit 300 blurs and outputs an image using a blur reproduction mathematical model. Specifically, blurring processing is applied to the image according to the visual characteristics of the refractive error person, and a blurred image that can be seen through the eyes of the refractive error person is output (see FIG. 3).

The update unit 400 compares the input image P101 input to the image generation device 1 with the sensory simulated image P103 image-processed by the conversion unit 200 and the blurring unit 300 to obtain an error. Then, the NN parameters (weights and biases) are updated so that the error is minimized (that is, the perceptually simulated image P103 approaches the input image P101).

Further, the conversion unit 200 performs image processing on the input image using the NN whose parameters have been updated by the update unit 400, and outputs the corrected image. For example, image processing is performed on the input image P101 to output a corrected image P111. When the corrected image P111 is viewed by a person with refractive error, it does not appear blurred like the blurred image P102 illustrated in FIG. 3, but appears clearly normal like the input image P101. Hereinafter, such a corrected image that has undergone image processing so that it looks clearly normal to a person with refractive error may be referred to as a "perceived image".

[Image generation method]
Embodiments of the image generating method of the present invention will be described below together with embodiments of the image generating apparatus of the present invention with reference to the drawings. An embodiment of the image generation method of the present invention includes a learning step and an inference step.

(1) Learning Step FIG. 4A is a flow diagram of the learning step of the image generation method according to the embodiment of the present invention. In this embodiment, NN is CNN. In the learning step, learning is performed based on the input image P101, and the parameters of the CNN are updated to generate a trained model (flow F1 in FIG. 1). Also, the input image P101 in the learning step is a set of a plurality of teacher images (teacher image data set).

The input image P101 is stored in the storage unit 100, and is first subjected to image processing by the CNN of the conversion unit 200 (step S101). CNN is a convolutional neural network that performs image processing, and super-resolution convolutional neural network (SRCNN: Super-Resolution Convolutional Neural Network), FSRCNN (Fast SRCNN), VDSR (Very Deep Super Resolution), etc. can be used. .

The input image P101 subjected to image processing by the CNN is then subjected to blurring processing by the blur reproduction mathematical model of the blurring unit 300 (step S102). Using a blur reproduction mathematical model, the corrected image that has undergone image processing by CNN is further subjected to image processing (blurring) according to the visual characteristics of the person with refractive error, and the image after this blurring is processed by the person with refractive error. A blurred image that can be seen from the eyes of the person with refractive error when viewed is output as a sensory simulated image P103. In other words, the blur reproduction mathematical model performs image processing to reproduce a blurred image seen through the eyes of a person with refractive error. A simulated perceptual image is a reproduction of an “image seen through the eyes of a person with refractive error” by performing blurring processing using a blur reproduction mathematical model of the blurring unit 300 .

In this way, after the resolution correction processing is performed by the CNN, the sensory simulated image P103 subjected to the blurring processing by the blur reproduction mathematical model is output. After that, the update unit 400 compares the output sensory simulation image P103 with the input image P101 stored in the storage unit 100 before image processing is performed by the conversion unit 200 and the blurring unit 300 .

Based on this comparison, the updating unit 400 obtains the error between the input image P101 and the sensory simulated image P103 (step S103). Also, the updating unit 400 updates the parameters of the NN so that the obtained error is minimized by the method of least squares or the like.
This error is, for example, the mean square error between the pixel values of the input image P101 and the sensory simulated image P103. Specifically, it is a value calculated by squaring the error of each pixel value between the input image P101 and the sensory simulated image P103 and dividing the sum by the number of data (the number of pixels). Then, the update unit 400 determines whether or not the mean square error satisfies a predetermined condition, and if not, repeats the learning steps (steps S101 to S103) until the condition is satisfied (step S104).

Here, until a predetermined condition is satisfied is, for example, the case where the mean square error becomes equal to or less than a certain value, or the case where the mean square error does not change a predetermined number of times. In short, this is the case when it is determined that the mean squared error has sufficiently converged.

If it is determined that the mean squared error satisfies the predetermined condition, the learning step ends (step S105). That is, the learning step ends when it is determined that the sensory simulation image P103 is sufficiently close to the input image P101. At this time, the learned model is stored (saved) in the storage unit 100 . Such determination and parameter update of the NN can be realized, for example, by using the mean square error as the loss function in the learning step.

Here, the point of bringing the perceptual simulation image P103 sufficiently close to the input image P101 will be described.
The sensory simulated image P103 is a blurred image that can be seen through the eyes of the person with refractive error reproduced according to the visual characteristics of the person with refractive error. Further, the image distinguishes clearly visible parts and difficult-to-see parts from the eyes of the refractive error patient.

The updating unit 400 in the learning step takes into account the state of the perceptual simulation image P103, and, in addition to the clearly visible part, particularly the part that is difficult to see with the eyes of a person with refractive error, is adjusted to the nearsightedness, astigmatism, presbyopia, or the like. A process of approximating the input image P101, which is usually clearly visible to non-persons, is performed.

In other words, according to the present invention, an image that a user (a user of the present invention) wants to perceive can be clearly seen, such as an image that is normally seen clearly by a person who does not have myopia, astigmatism, or presbyopia as described above. The image is processed as follows. Therefore, in the present invention, compared with the conventional image processing that uniformly makes the display screen easier to see, it is possible to obtain an image that is further processed according to the visual characteristics while meeting the perception needs of the user.

The input image processed by the present invention will be exemplified later. It may be the one subjected to sharpening or image enhancement. Needless to say, if the input image is not sharp, it is necessary to generate a sharpened image in advance and apply it to the learning step.
Such findings have been found by the present inventors through the present intensive studies of the present invention.

Through the above procedure, a trained model is generated in the learning step. Here, the loss function may be mean absolute error or SSIM (Structural Similarity), and can be implemented using a deep learning library such as Keras or Chainer. Also, the error may be an index for measuring the difference between images other than the specific examples described above, and the update unit 400 can appropriately adopt a method for obtaining this index.

(2) Inference Step FIG. 4B is a flow diagram of the inference step of the image generation method according to the embodiment of the present invention. The inference step performs image processing on the input image P101 using the trained model generated in the learning step, and outputs a perceptual image P111 (flow F2 in FIG. 1).
During the inference step, the learned model is stored in the storage unit 100 of the image generation device 1 . By storing a trained model generated by the image generation device 1 in the storage unit 100 of another image generation device 1 , another image generation device 1 can perform inference. In addition, it is also possible to deploy trained models to terminals (edge devices), store them in a cloud server, and perform inference on the terminals.

When an image is input to the image generation device 1, first, using the trained model stored in the storage unit 100, the conversion unit 200 performs image processing on the input image P101 (step S201). Then, it is output as a perceived image P111 (step S202). The input image in the inference step is, for example, an image that is displayed on the screen of a personal computer, smartphone, or the like owned by the person with refractive error and viewed.
When viewed by a person with refractive error with the naked eye, the input image P101 without any image processing is blurred (blurred image P102 in FIG. 1). P111 appears clearly normal.

A plurality of trained models can be stored in the storage unit 100, and a trained model to be used in the inference step can be appropriately selected according to the visual characteristics of the user (refractive person). For example, when one image generation device 1 is used by a plurality of people such as a family member, a trained model to be used can be selected according to the visual characteristics of the person using it.

[Blur part]
As described above, the blurring unit 300 performs blurring processing using a blur reproduction mathematical model to reproduce a blurred image seen through the eyes of a person with refractive error. Here, the blur reproduction mathematical model can be prepared in accordance with the appearance from the eyes of each refractive error patient. The way the eyes of each refractive error person see is the difference in the type of refractive error such as myopia, hyperopia, astigmatism, and presbyopia, and the refractive power (for example, weak myopia, moderate myopia, strong myopia, maximum myopia). ) is the difference.
In addition, the blurring unit 300 can reproduce not only the refractive error described above, but also an image that can be seen through the eyes of a person with color blindness. For example, the blurring unit 300 can perform color vision compensation processing on an image by using a color vision deterioration model.
In other words, the blurring part 300 is a visual deterioration model that can reproduce visual deterioration such as refractive error and color blindness.

The blur reproduction mathematical model applies a point spread function (PSF), which indicates how a point is blurred when viewed through a certain optical system, to its model structure and parameters. It is possible to reproduce the appearance from the eyes of each refractive error person. The point intensity distribution of the eye can be determined based on the pupil function and pupil radius.

The pupil radius is 1.5 mm to 3.0 mm for the human eye. The pupil function can be derived from the wavefront aberration, which indicates how much the human eye is distorted from the ideal state (the state in which the retina can be properly focused) when the human eye is considered as a lens (optical system).
Wavefront aberration can be represented by a combination of Zernike polynomials and associated coefficients. By using the coefficients related to the Zernike polynomials as unique parameters indicating the degree of refraction of each refractive error patient's eye, it is possible to obtain the point intensity distribution of each refractive error patient's eye.

Then, by applying the point intensity distribution of each refractive error person's eye to the blur reproduction mathematical model, it is possible to generate a blur reproduction mathematical model that reproduces the appearance from the eyes of each refractive error person. That is, the blur reproduction mathematical model can be generated in the image generation device 1 according to the visual characteristics of the user (refractive person). The method of generating a mathematical model for reproducing blur in operation is, for example, a strict generation method based on data that can be obtained from measuring instruments in an ophthalmology department, or a simple generation method based on average data assumed from the visual acuity of a person with refractive error. There are methods.
The reproduction mathematical model generated in this way is stored in the storage unit 100 and can be used as appropriate in the learning step, so that a perceptual simulation image corresponding to various visual characteristics of a person with refractive error can be output more accurately. be able to.

[Image generation system]
The image generation device 1 has been described so far, and the image generation system will now be described. FIG. 11 is a schematic configuration diagram showing an image generation system according to an embodiment of the invention. The image generation system 10 includes an image generation device 1 and a terminal T connected to the image generation device 1 .

The image generation device 1 and the terminal T are connected via a network N and can communicate with each other. The network network N is a network line such as a WAN (Wide Area Network) or a LAN (Local Area Network). They can communicate with each other.

In the image generation system 10, the image generation device 1 can be configured as, for example, a cloud server or an on-premises server.
A terminal T is owned by a user of the image generation system 10 (for example, a person with refractive error). The terminal T is, for example, a personal computer, a smartphone, or the like, and has a function (communication function) of transmitting an input image such as a still image or a moving image to the image generating device 1 and receiving a perceptual image from the image generating device 1. have. The computer can be of any type, such as a desktop computer or a laptop computer. Note that the terminal T may be a tablet, a TV, or the like as long as it has a communication function capable of communicating with the image generating apparatus 1 via the network N described above.

The terminal T includes an imaging unit (not shown) and a display unit (not shown). The imaging unit has a function of capturing a still image, a moving image, or the like to be transmitted to the image generation device 1, and is, for example, a camera of a smartphone. In other words, the terminal T can acquire an input image such as a still image or a moving image by the imaging unit. Further, still images and moving images to be input images may be obtained from the outside via the Internet network N, or may be stored in the terminal T via a storage medium such as a USB memory or an SD card. . On the other hand, the display unit has a function of receiving a perceptual image output from the image generation device 1 and displaying the perceptual image, and is, for example, a display of a personal computer, a smartphone, or a TV.

Hereinafter, an example of the operation of the image generation system 10 will be described assuming that the user of the image generation system 10 is a person with refractive error, and the terminal T is a smartphone owned by the person with refractive error. Here, it is assumed that the terminal T stores photographs (still images) and movies (moving images) taken by the imaging unit of the terminal T when the user traveled in the past. The memory unit 100 of the image generation device 1 also stores a learned model that has been learned according to the visual characteristics of the person with refractive error who is the current user.

The user transmits pictures and movie images stored in the terminal T to the image generation device 1 via the network N. FIG. Then, the conversion unit 200 of the image generation device 1 uses the learned model stored in the storage unit 100 to convert the received image into a perceptual image that can be normally perceived by the eyes of the person with refractive error who is the current user. to output The output perceptual image is transmitted to the terminal T via the network N. The terminal T that has received the perceptual image transmitted from the image generation device 1 causes the display unit to display the perceptual image.

In this way, the terminal T displays a perceptual image that can be seen clearly and normally from the eyes of the person with refractive error who is the current user. The user can clearly and normally browse the photos and movies stored in the terminal T.

Further, in this case, since the processing of outputting the input image as the perceived image is performed by the image generation device 1, the terminal T includes hardware such as memory, disk capacity, processor (for example, GPU (Graphics Processing Unit)). High performance requirements are not required. In other words, in this case, since the terminal T owned by the user may include a display unit and the like, convenience and versatility for the user are improved.

On the other hand, the trained model may be stored in the terminal T, the conversion unit 200 may be provided in the terminal T, and the terminal T may output the input image as the perceived image. In this case, the sensory image can be output and displayed on the display unit without communicating with the image generating apparatus 1 via the network N. FIG. Therefore, the user can use the image generating system 10 even in an offline environment or when a failure occurs in the network N, which improves convenience.

Also, the image generation device 1 may be divided into a device that generates a trained model and a device that stores the generated trained model, is provided with a conversion unit 200, and outputs an input image as a perceived image. . In other words, the function of the conversion unit 200 can be separated and provided to separate hardware according to its use. In the above description, for example, the terminal T is provided with a conversion unit having a function of outputting an input image cut from the conversion unit 200 as a perceptual image, and processing is performed. That is, the conversion unit performs image processing on the input image using the trained neural network generated by the update unit 400, and outputs a perceptual image that can be normally perceived by the eyes of the person with refractive error. It has a function as A device that outputs the input image as a perceived image may be the terminal T as described above, or may be another piece of hardware.
Furthermore, as an example, the above describes that the device (hardware) that generates the trained model and the device (hardware) that stores the generated trained model and outputs the input image as a perceptual image can be separated. However, if necessary, the hardware can be divided into more units, or the functions other than the conversion unit 200 can be distributed to each piece of hardware.

Note that the learned model stored in the terminal T may be downloaded from the image generation device 1 via the network N. For example, if the terminal T is a personal computer and the personal computer is shared by a family, a trained model that has been learned according to the visual characteristics of the user (father, mother, child) is adapted to the user. You may make it download suitably. Alternatively, a trained model learned according to the visual characteristics of the user (father, mother, child) is stored in the terminal T in advance, and the trained model to be used is changed according to the timing when the user uses the terminal T. may be selected as appropriate.

The operation of the image generation system 10 described above is merely an example, and the number of hardware constituting the image generation system 10, the form of the network N, and the like can be changed as appropriate without departing from the gist of the present invention. is. Also, the input image may be not only a photograph (still image) or movie (moving image), but also an electronic book such as a comic book or a novel, a web page, or a streaming image.

Hereinafter, as Examples 1 and 2, results of image generation using the image generation apparatus and the image generation method of the present invention are shown. In Examples 1 and 2, SRCNN was adopted as NN. Here, the generated perceptual image is subjected to blurring processing by the blurring unit 300 (flow F3 in FIG. 1) to reproduce the image seen by the eyes of the person with refractive error, so that the perceived image is clearly normal to the eyes of the person with refractive error. confirmed that it can be seen

(Example 1)
In Example 1, a natural image data set of 200×200 pixels and a total of 4996 natural images was used as the teacher image (teacher image data set) (see FIG. 5A).

In the learning step, the natural image data set is randomly divided into a training data set (Train): 3996 images and an evaluation data set (Validation): 1000 images, and one training data set is learned (one epoch ), the loss value was calculated using the verification data set. The loss function was the mean squared error, and if the loss value calculated for each epoch did not improve 15 times in succession, it was determined that the learning had converged, and the learning was terminated.

FIG. 6 is a graph showing changes in loss values in the learning step. Referring to FIG. 6, the loss value drops to around 0.0025 when the number of epochs exceeds 400 for both the training data set and the validation data set. When the loss value is around 0.0025, the predetermined condition set in this embodiment (no improvement for 15 times in a row) is satisfied, and learning is completed.

FIG. 7 is an explanatory diagram showing the results of Example 1, and is a perceptual image and the like output using the trained model generated by the learning step. The input image was a natural image containing an animal (zebra).
Here, the simulated sensory image P302 is obtained by subjecting the input image P301 to blurring processing using the blur reproduction mathematical model of the blurring unit 300 in order to reproduce a blurred image that can be seen through the eyes of a person with refractive error. The sensory simulated image P312 is obtained by subjecting the sensory image P311 to blurring processing using the blurring reproduction mathematical model of the blurring section 300 in order to reproduce a blurred image seen by the eyes of a person with refractive error.

In this way, when the person with refractive error sees the input image P301, it appears blurred like the sensory simulated image P302. However, when the perceptual image P311 generated in the inference step is viewed by a refractive error person, it appears as clearly normal as the perceptual simulated image P312.

For easy understanding, a part of the perceptual simulation image P302 and the perceptual simulation image P312, which is clearly visible to the eyes of the person with refractive error, is enlarged and compared (see FIG. 8). From this, it can be seen that the perceptual simulated image P312 is more suppressed in blurring than the perceptual simulated image P302, and looks clearly normal like the input image P301.

(Example 2)
Also, the natural image data set (see FIG. 5A) and the English image data set (see FIG. 5B) were used for learning to generate trained models. The conditions of the natural image data set used are the same as in the first embodiment. On the other hand, an English image data set of 250×250 pixels and a total of 3000 images (training: 2400 images, evaluation: 600 images) was used.

Then, as Example 2, each trained model was used to perform image correction of an English image including alphabets. 9 and 10 are explanatory diagrams showing the results of Example 2. Using the respective trained models, two types of images (1) in the upper row and (2) in the lower row are output as input images. perceptual images, etc. FIG. 9 shows the results when a trained model trained on a natural image data set is used, and FIG. 10 shows the results when a trained model trained on an English language image data set is used.

Comparing the perceived image in FIG. 9 and the perceived image in FIG. 10, there is almost no difference. In other words, it can be seen that a perceptual image with sufficient visibility could be output even when image correction was performed on an English image using a trained model that was trained using a natural image dataset as a teacher image.

The present invention is an image generation method for generating a perceptual image that is corrected according to the visual characteristics of each person with refractive error and that can be seen clearly and normally even with the naked eye. It is industrially useful because images can be displayed on the screens of personal computers, smartphones, and the like.

1 Image generation device 10 Image generation system 100 Storage unit 200 Conversion unit 300 Blur unit 400 Update unit P101, P301 Input image P102, P103, P302, P312 Perceptual simulated image P111, P311 Corrected image (perceptual image)
P201, P202, P211, P212 Image F1-F3 Flow

Claims (11)

Applying image processing to the input image using a neural network;
further subjecting the input image to image processing using a blur reproduction mathematical model, and outputting a sensory simulated image subjected to blurring processing according to the visual characteristics of a person with refractive error;
a learning step comprising updating parameters of the neural network by comparing the input image and the sensory simulated image;
an inference step of performing image processing on the input image using the neural network whose parameters have been updated and outputting a perceptual image that is normally perceptible from the eyes of the person with refractive error;
Image generation method including. 2. The image generation method according to claim 1, wherein said blur reproduction mathematical model outputs a perceptually simulated image that is perceived as blurred when a person with refractive error sees said input image. The learning step updates the parameters of the neural network so that an error between the input image and the sensory simulated image is minimized;
2. The error is determined by a loss function based on pixel values of the input image and the sensory simulated image, and the learning step is repeated until the error determined by the loss function satisfies a predetermined condition. 2. The image generating method according to 2. The blur reproduction mathematical model calculates a pupil function based on a wavefront aberration in which a coefficient related to a Zernike polynomial is a characteristic parameter indicating the degree of refraction of the eye of a refractive error patient, and the pupil function and the pupil radius of the refractive error patient are calculated. The image generation method according to any one of claims 1 to 3, wherein the point intensity distribution calculated based on is applied. The image generation method according to any one of claims 1 to 4, wherein the neural network is a super-resolution convolutional neural network. Applying image processing to the input image using a neural network;
further subjecting the input image to image processing using a blur reproduction mathematical model, and outputting a sensory simulated image subjected to blurring processing according to the visual characteristics of a person with refractive error;
updating parameters of the neural network by comparing the input image and the sensory simulated image;
A trained neural network model generated by an image generation method including A conversion unit that performs image processing on an input image using a neural network;
Blur for further performing image processing on the input image that has undergone image processing by the conversion unit using a blur reproduction mathematical model, and outputting a sensory simulated image that has been subjected to blur processing according to the visual characteristics of a person with refractive error. Department and
an updating unit that updates parameters of the neural network by comparing the input image and the sensory simulated image to generate a trained neural network;
an image generating device comprising: 8. A terminal connected to the image generation device according to claim 7 and configured to transmit an input image to the image generation device, comprising a display unit for receiving and displaying a perceived image corresponding to the input image output by the image generation device. containing terminals. A perceptual image generator that performs image processing on an input image using the trained neural network generated by the updating unit and outputs a perceptual image that can be normally perceived by the eyes of the person with refractive error. 8. The terminal according to 8. An image generation system comprising the image generation device according to claim 7 and the terminal according to claim 8 or 9. A conversion unit that performs image processing on an input image using a neural network;
Blur for further performing image processing on the input image that has undergone image processing by the conversion unit using a blur reproduction mathematical model, and outputting a sensory simulated image that has been subjected to blur processing according to the visual characteristics of a person with refractive error. Department and
An image generation program that causes a computer to function as an image generation device, including an updating unit that updates parameters of the neural network by comparing the input image and the sensory simulated image to generate a trained neural network.

Images