JP7325775B2 - Image processing system, method, and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Law 1. (1) Publication date of the website September 4, 2020 ((2) (a)), September 17, 2020 ((2) (a)) (2) Website address (a) ) https://cedec. cesa. or. jp/2020/session/detail/s5e8329edf1425. html (a) https://www. silicon studio. co. jp/rd/ (3) Publisher Silicon Studio, Inc. (4) Details of the disclosed invention As shown in the attachment. (2) On the website of (a), an online session was held using the attached materials. In addition, on the website of (2) (b), we have linked so that the attached materials can be downloaded.

The present invention relates generally to a system for performing image processing, and more specifically to a system for efficiently performing graphic processing such as rendering using a neural network model.

In recent years, AI has made it possible to generate high-quality images and videos, and it has become possible to express human faces without feeling unnatural. For example, non-real-time applications have the ability to produce images that are indistinguishable from real photographs. There are also attempts to apply AI technology in graphics processing.

For example, a technique has been proposed for performing efficient distributed denoising of graphics frames (Patent Document 1).

That is, in US Pat. No. 6,300,000, there are a plurality of nodes that perform ray tracing operations, and a dispatcher node that dispatches graphics work to the plurality of nodes, each node for an image frame specified by the graphics work. and at least a first node of the plurality of nodes, performing ray tracing to render a first region of the image frame. a ray tracing renderer that denoises the first region using a combination of data associated with the first region and data associated with a region outside the first region; and at least a first node having a denoiser, wherein at least some of said data associated with said regions outside said first region is retrieved from at least one other node; there is

Also proposed is a cloud-based real-time rendering technology that performs more efficient ray tracing calculations (Patent Document 2).

That is, in US Pat. No. 6,200,008, a system is a first graphics processing node that executes a first set of graphics processing operations to render a graphics scene, wherein the first set of graphics processing operations is a ray tracing independent operation and an interconnection or network interface coupling said first graphics processing node to a second graphics processing node, said second graphics processing node being connected to said first graphics processing node. receiving an indication of a user's current view; receiving or constructing a view independent surface generated by view independent ray traversal and intersection operations; performing a transformation based on the current view of the user to generate a view dependent surface and providing the view dependent surface to the first graphics processing node, the first graphics processing node performing a second graphics processing A system is disclosed that executes a set of operations to complete the rendering of the graphic scene using the view dependent surface.

A technique has also been proposed that employs machine learning in rendering processing to reduce the occurrence of significant artifacts (Patent Document 3).

That is, Patent Document 3 discloses an image processing apparatus for generating an image from volume data, which acquires a volume data set, acquires a non-uniformity map based on the volume data set, determining locations of a set of aperiodic sampling points using a map; generating from the volume data set a set of sampled data values based on the determined locations of the aperiodic sampling points; An image processing apparatus is disclosed that includes processing circuitry configured to generate an image data set by performing an aggregation process to generate a set of image data points from a set of derived data values.

JP 2020-102195 A Japanese Patent Application Laid-Open No. 2020-109620 JP 2020-191061 A

However, since real-time rendering requires a large amount of calculations to generate high-quality images and videos, it is still difficult to generate images and videos with AI in real time. In addition, AI research on images and videos at the time of filing this application focuses on "improving quality", and there is still room for improvement in applying AI to real-time rendering. That is, there is room for introducing a new architecture when trying to apply AI to real-time rendering.

Present problems are shown with concrete examples. A neural network may be implemented in a GPU when generating high-quality images and videos in real time with AI. A conventional neural network consists of many layers, and since there are data dependencies between these layers, synchronization processing for each layer is required. Also, data output by one layer is output to memory, and this output data must be read from memory again to be input to the next layer. And each layer has a large number of channels without branches, and it is necessary to process all input channels, including unnecessary channels. Therefore, redundant calculation processing and large consumption of memory are generated.

Also, existing neural network models do not perform well for real-time rendering. For example, not only is the execution speed slow due to the above reasons, but also the face shape, direction, lighting, etc. cannot be specified directly. Therefore, even if the existing rendering method is replaced with AI as it is, a high effect cannot be expected.

SUMMARY OF THE INVENTION An object of the present invention is to enable real-time rendering of natural human face images and real-time video with AI by eliminating the above-described inefficient processing.

Therefore, an image processing system according to an embodiment of the present invention is an image processing system that includes a CPU and a GPU and performs rendering processing using a neural network. and a processing unit that performs hash division of labor for properly using the neural network trained for each tile, and the hash division of labor includes: a calculator for hashing pixel values in the screen frame to calculate a hash value; and a reader that uses the hash value as a key to select and read weights in the neural network for corresponding tiles from a lookup table. and the output from the hash division of labor is an estimate estimated by the weights of the neural network corresponding to the tiles in the screen frame.

The computer also includes a plurality of partial processors (synchronization groups, or SMs, hereinafter also simply referred to as "synchronization groups") that can be efficiently synchronized with each other in processing units, and an on-chip memory that the synchronization groups can share. wherein the hash division is characterized in that for each tile in the screen frame, the sync group is assigned and the neural network weights are loaded onto the on-chip memory.

The computer also has the synchronization group and a register (an on-chip memory of an instruction scheduler), and the hash division assigns the synchronization group to each tile in the screen frame, and the neural network is stored in the register.

According to the image processing system and the like according to one embodiment of the present invention, a special effect of realizing a new application of AI to real-time rendering and enabling rendering of images and videos of human faces without unnaturalness. play.

1 is an explanatory diagram illustrating an example of the overall configuration of an image processing system according to an embodiment of the present invention; FIG. FIG. 4 is an explanatory diagram illustrating variations of the information processing server configuration in the image processing system according to the embodiment of the present invention; 1 is an explanatory diagram illustrating an external configuration of an information processing device in an image processing system according to an embodiment of the present invention; FIG. 1 is an explanatory diagram illustrating functional blocks of an information processing device in an image processing system according to an embodiment of the present invention; FIG. 4 is a flowchart for explaining an outline of operations of an image processing system, etc., according to an embodiment of the present invention; 4 is a flow chart explaining the details of the operation of the image processing system etc. according to the embodiment of the present invention; FIG. 2 is an explanatory diagram for explaining a data configuration example that is a prerequisite for the operation of the image processing system and the like according to the embodiment of the present invention; FIG. 2 is an explanatory diagram for explaining a data configuration example that is a prerequisite for the operation of the image processing system and the like according to the embodiment of the present invention; FIG. 4 is an explanatory diagram illustrating an example of a hash function employed in the image processing system etc. according to the embodiment of the present invention; 4 is a flowchart for explaining detailed operations of the image processing system and the like according to one embodiment of the present invention; 4 is a flowchart illustrating detailed operations (preprocessing for training) of the image processing system, etc., according to an embodiment of the present invention. 4 is a flowchart illustrating detailed operations (preprocessing for training) of the image processing system, etc., according to an embodiment of the present invention. 4 is a flowchart illustrating detailed operations (preprocessing for training) of the image processing system, etc., according to an embodiment of the present invention. FIG. 4 is an explanatory diagram illustrating a specific example of the operation of the image processing system etc. according to the embodiment of the present invention; FIG. 4 is an explanatory diagram illustrating a specific example of the operation of the image processing system etc. according to the embodiment of the present invention; FIG. 4 is an explanatory diagram illustrating a specific example of the operation of the image processing system etc. according to the embodiment of the present invention; FIG. 4 is an explanatory diagram illustrating a specific example of the operation of the image processing system etc. according to the embodiment of the present invention; FIG. 10 is an explanatory diagram for explaining an operation example in a conventional image processing system or the like; FIG. 10 is an explanatory diagram for explaining an operation example in a conventional image processing system or the like;

(Definition of terms)
First, terms used in this embodiment are defined.
[Rasterize]
In general, in image processing, it refers to converting non-raster format data (e.g., vector format data) into raster format and creating an image, but in 3D computer graphics, shape data such as polygons to pixel data (also called fragments). In this embodiment, it refers to storing the intermediate state of each pixel in a buffer such as a G-Buffer in relation to 3D computer graphics processing. Although the present invention is not limited to these, examples of rasterization output in one embodiment of the present invention include the two-dimensional position (or UV) of the object surface for each pixel, the ID of the surface type (classification ID ), the amount of light (luminous intensity) on the same surface. Note that UV is a value (UV value) in a coordinate system (UV coordinate system) for each texture.
[tile]
This is a unit for treating one pixel (one pixel) collectively to some extent. As an example, a tile can be 16×16 pixels. Also, each pixel can consist of a data set such as UV value, classification ID, and luminous intensity.
[Global scale]
Scope at the scale of the world (global) coordinate system.
[Tile size]
It refers to the scope on the scale of the local coordinate system (Part 1). The local coordinate system here is a local coordinate that leaves room for reduction to the pixel scale.
[Pixel Scale]
It refers to the scope on the scale of the local coordinate system (part 2). The local coordinate system here is the minimum local coordinate in this embodiment.
[SM (Streaming Multiprocessor)]
It is a processor unit of GPU. As an example, for the GPU architecture of NVIDIA Corporation, [https://images.nvidia.com/aem-dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture -Whitepaper-V1.pdf].

Although the present invention is not limited to this, one SM has four independent instruction schedulers (referred to as "partial processors"), and one instruction scheduler includes registers ( As an example, 64 KB). A plurality (for example, four) of partial processors (which can also be referred to as a "synchronization group" in one embodiment) constitutes the SM. Thus, in one embodiment, "effectively synchronized with each other in a processing unit" means that sub-processors within a single SM can be efficiently synchronized with each other.

Also, in one embodiment, the processing unit is provided with on-chip memory (eg, 128 KB) that can be shared by a sync group within one SM.

Although the invention is not so limited, in one embodiment it can be envisaged to utilize an SM with the following configuration. First, the number of SMs in a GPU ranges from tens to 100 or more. Also, four instruction schedulers are implemented per SM, and each instruction scheduler has 64 KB of registers. In one embodiment of the present invention, these SMs may process a rasterized and tiled input image (input tiles) in parallel, and as an example, allow one scheduler to process one tile. . The output of these processes is the output tile, and the input and output tiles occupy corresponding positions in the image.
[Hash division of labor]
1 is a novel architecture employed in the present invention; This approach can reduce or eliminate inefficiencies in CG processing using conventional neural networks. Hash division of labor is roughly a process of hashing (calculating a hash value) the content (numerical value) of each tile in a screen frame, and using the hash value as a key, extracting the corresponding tile from a lookup table. and selecting and loading weights in the neural network (model). Hash division of labor can be said to be a process that separates and executes subdomains of a neural network.

Also, the output from the hash division of labor is an estimate estimated by the neural network weights corresponding to the tiles in the screen frame.
(Characteristics of the present invention)
Next, features of the present invention will be described. A novel feature of the present invention is that a neural network is used to directly draw the CG real-time drawing process.

In general, the results of image processing by AI are often performed by comparing screenshots, but the results in one embodiment of the present invention are not based solely on the measurement results, but the correct approach from the basic principle. It can be derived logically.

In addition, the present invention focuses on dependence of data to be handled and expansion of dependence. Therefore, one of the conditions under which the present invention is effective is the presence of global data dependency. For example, in the real-time drawing envisioned by the present invention, data input is generated from binary data such as cameras and meshes, rather than from photographs whose content is unknown. Therefore, the information to be put into the pixel is free and can be added if necessary. This means that there is room for the innovative approach proposed by the present invention.
(Basic concept of the present invention)
The basic concept of the present invention is a set of processes to solve the inefficiencies specific to real-time rendering. As an example, in one embodiment of the present invention, mesh information and/or lighting information is converted into a highly structured input format so that the mesh information and/or lighting information can be applied to a neural network.

Further, in one embodiment of the present invention, a number of neural networks specialized for specific drawing are trained, and hash division of labor is performed in which each tile is selectively used by hash.

In other words, the present invention is supported by the following basic processing groups.
(1) Mesh information and/or lighting information is converted into a highly structured input format for neural networks.
(2) A hash division of labor process is employed in which a large number of neural networks specializing in specific drawing are trained and each tile is selectively used by hash.

The neural networks in (1) and (2) above are designed to fit within the constraints of hardware performance.
(Example of application scene of the present invention - when drawing a human face)
Next, an example of an application scene of the present invention will be given for easy understanding of the present invention. In general, humans can perceive faces with very high sensitivity. For example, humans can immediately notice that the facial expression of the same person is even slightly unnatural. This fact means that if a computer can correctly draw subtle changes in a human face, the attractiveness of an image to humans can be greatly increased. In recent years, real-time ray tracing by hardware has been realized, and for example, it has become possible to accurately calculate the specular reflection of an object, but it is still difficult to accurately calculate the surface scattering of human skin. Have difficulty.

On the other hand, it is relatively easy to obtain a large amount of actual images of human faces, so it is promising to use the large amounts of available human face image data as training data and draw with AI. Conceivable. Although the invention is not so limited, in one embodiment of the invention there are significant advantages in real-time rendering of human facial images.

An image processing system, method, and program according to one embodiment of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows an example of the overall configuration of an image processing system according to one embodiment of the present invention. The present invention can be operated stand-alone on a computer such as a PC, but can also be implemented in a group of information processing servers in a network configuration as shown in FIG. Further, even with the network configuration as shown in FIG. 1, at least some or all of the processing routines may be executed in another information processing server or an information processing device such as a PC or a tablet terminal. Hereinafter, one embodiment of the present invention will be described with reference to FIG. 1 for easy understanding of the present invention.

As shown in FIG. 1, an image processing system 10 includes, as an embodiment, an information processing server group 11 and various information processing devices used by users (in the figure, PCs 12 and 13, a mobile phone 14, A smartphone, a mobile information terminal, or a tablet terminal 15 is shown, hereinafter collectively referred to as "various terminals", "user terminals", or simply "terminals"), and an information processing server The group 11 and various terminals are connected so as to be mutually communicable by a dedicated line or a public line such as the Internet (in FIG. 1, 16 to 19 are shown as examples of wired lines), as shown in FIG. It is Also, the line may be wired or wireless. When the line is wireless, the mobile phone 14 and the terminal 15 enter the Internet 19 via a base station, wireless router, etc. (not shown), and are connected to the information processing server group 11 via the line 18 so as to be able to communicate with each other. .

Many of the mobile phones 14, smart phones, mobile information terminals, or tablets 15 at the time of filing this application have processing capabilities (communication processing speed, image processing capability, etc.) equivalent to those of personal computers (PCs). It should be called a small computer.

Programs or software necessary for carrying out the present invention are usually stored in a storage unit in an information processing server group, and if necessary, in a HDD (Hard Disk Drive) or SSD (Solid State Storage) in a storage unit of a PC or a portable information terminal. Drive) or the like, and when the program or software is executed, all or part of it is read as a software module into the memory in the storage unit as required, and the CPU or the like executes calculations.

It should be noted that the execution of calculations does not necessarily have to be performed only by a central processing unit such as a CPU, and a processor such as a graphical processing unit (GPU) or a digital signal processor (DSP) (not shown) can also be used.

Furthermore, the hardware configuration of the information processing server group 11 can basically employ a PC. Although the present invention is not limited to this, the information processing server group 11 may include a plurality of PCs (for example, several tens to tens of thousands of PCs) in order to improve the hardware specifications as necessary. A configuration suitable for processing large-scale data can also be obtained by operating in parallel. In addition, a cloud configuration that is available at the time of filing of this application can also be adopted.

Although the image processing system 10 according to one embodiment of the present invention has been described above with reference to FIG. 1, the configuration of the present invention is not necessarily limited to this. is concentrated in the information processing server group 11, the information processing server group 11 may be used as an image processing system as another embodiment of the present invention (the same applies hereinafter).

Further, as already explained, in the image processing system according to another embodiment of the present invention, it is possible to adopt a stand-alone configuration mainly composed of a single server or a single terminal without adopting a network configuration.

FIG. 2 shows a variation of the information processing server configuration in the image processing system according to one embodiment of the present invention. The operations of the information processing server group 11 are realized by individual operations of hardware described below and cooperative operations between software and these hardware.

In FIG. 2, the information processing server group 11 accessed from the user terminals 15a to 15c is illustratively configured as a cluster system so that a plurality of server systems are linked and operated as one system. With such a cluster configuration, for example, even if one server fails, other servers can continue processing. , the processing can be distributed to other server(s) and the stability of the overall system can be improved. Such a cluster configuration is one of the advantageous configurations especially when constructing a real-time multi-game play system.

Although the present invention is not limited to this, for ease of understanding of the present invention, hereinafter, the information processing server group 11 is assumed to be a server group that provides a real-time multiplayer game including real-time rendering. do.

In FIG. 2 , the information processing server group 11 is roughly divided into a real-time cluster (group) 111 , a load balancer (group) 112 (group), and an API server 23 . Also, other embodiments of the present invention may be configured to have cache cluster(s) not shown.

In one embodiment of the invention, real-time cluster(s) 111 includes Lobby cluster(s) and Game cluster(s).

In one embodiment of the invention, load balancer(s) 112 also include Lobby load balancer(s) and Game load balancer(s).

In one embodiment, the Lobby cluster(s) can be configured to handle user matching operations in lobbies to establish real-time multiplayer games. Also, the Game cluster (group) can be configured to be in charge of real-time communication processing and the like in the action portion in progressing the real-time multi-game.

In one embodiment, in the load balancer(s) 112, a Lobby load balancer is responsible for load balancing and autoscaling for the Lobby cluster(s) and a Game load balancer is responsible for load balancing and autoscaling for the Game cluster(s). can coexist and operate while coordinating with each other such as process monitoring.

FIG. 3 shows an external configuration of a tablet terminal as an information processing device in an image processing system according to one embodiment of the present invention. In FIG. 3 , an information processing device (tablet terminal) 15 includes a housing 151 , a display 152 , and hardware buttons 153 provided in the lower central portion of the housing 151 . The display 152 is typically composed of a liquid crystal display (LCD) or the like, and can display various information such as characters, still images, and moving images. Also, a menu button or a software keyboard can be displayed on the display 152 and can be used as an instruction (command) to the tablet terminal 15 by touching it with a finger or a touch pen (not shown). In this respect, the hardware button 153 is not an essential component, but for convenience of explanation of the present invention, it is implemented as a button that performs a certain function. Of course, it is also possible to substitute the hardware button 153 with a menu button displayed on a part of the display 152 .

Further, the display 152 includes a multi-touch input panel, and touch input position coordinates on the touch input panel are transmitted to the processing system (CPU) of the tablet terminal 15 via an input device interface (not shown). It is processed. And, the multi-touch input panel is configured to be able to simultaneously sense multiple touch points on the panel. This detection (sensor) can be realized by various methods, and is not necessarily limited to a contact sensor. For example, an optical sensor can be used to extract an indication point on the panel. Further, the sensor may be a contact sensor, an optical sensor, or a capacitive sensor that senses contact with human skin.

Moreover, although not shown in FIG. 3, the tablet terminal 15 can also include a microphone and a speaker. In this case, it is also possible to discriminate the voice of the user picked up by the microphone and use it as an input command. Furthermore, although not shown in FIG. 3, a camera device such as CMOS is mounted on the back surface of the tablet terminal 15 or the like.

FIG. 4 illustrates a functional block diagram of hardware configuring the tablet terminal 15 according to one embodiment of the present invention. The operation of the tablet terminal 15 is realized by individual operations of hardware and cooperative operations between software and these hardware, which will be described below.

In FIG. 4, the tablet terminal 400 as a hardware block as a whole is roughly divided into an input unit 401 composed of hardware buttons 153 in FIG. 3, a multi-touch input panel provided on the display 152, a microphone, etc. A storage unit 402 composed of a hard disk, RAM and/or ROM for storing data, etc., a central processing unit 403 composed of a CPU that performs various numerical calculations and logical operations according to programs, and a display 152 etc. a display unit 404, a control unit 405 for controlling chips, electrical systems, etc., a slot for accessing the Internet, a port for performing optical communication, and a communication interface composed of a communication interface. unit 406, an output unit 407 such as a speaker and vibration, a timer unit 408 for measuring time, a sensor unit 409 including an image sensor such as a CMOS, and a power supply for supplying power to each module in the device. These modules are appropriately connected by wiring such as a communication bus and a power supply line as required (in FIG. 4, they are collectively represented by a connection 411).

Note that the sensor unit 409 may include a GPS sensor module for specifying the position of the tablet terminal 400 (15). A signal detected by an image sensor such as a CMOS that constitutes the sensor unit 409 can be processed as input information in the input unit 401 .

Programs or software necessary for carrying out the present invention are usually installed or stored in a hard disk or the like constituting the storage unit 402, and when the programs or software are executed, all of them are stored in the memory in the storage unit 402 as necessary. Alternatively, it is read as a part of the software module, and the CPU 403 performs the calculation.

It should be noted that arithmetic execution does not necessarily have to be performed only by a central processing unit such as a CPU, and in a gaming tablet, a processor such as a graphical processing unit (GPU) or a digital signal processor (DSP) (not shown) can also be used.

Next, the outline of the operation of the image processing system or image processing program according to one embodiment of the present invention will be described with reference to the operation flows or flow charts of FIGS. 5 and 6. FIG.

As already described, the characteristic operation of the present invention can be performed mainly in the information processing server group 11, but at least part of it can also be performed by an information processing apparatus or the like.

FIG. 5 shows a flowchart for explaining the outline of the operation of the image processing system etc. according to one embodiment of the present invention. In the flowchart of FIG. 5, the basic operations of the image processing system according to one embodiment of the present invention are three processes for image data: (1) rasterization, (2) hash division of labor, and (3) neural network model. It is shown to consist of an execution process based on

When the process is started in step S501 of FIG. 5, the process proceeds to step S502 and rasterization is performed. Next, in step S503, hash division of labor is performed, and in step S504, arithmetic processing based on the neural network model is performed.

Then, in step S505, output to the frame buffer is performed based on the output result in the previous step, and in step S506, image data to be output to the display of the user terminal or the like is output. Next, the process proceeds to step S507, and the process ends as the main flow for explanation.

In FIG. 5, the means of "rasterization" and "hash division of labor" are used for easy understanding of the invention, but the invention is not limited to these. In particular, rasterization has been described as a specific example using structured data, and the present invention can employ various "structured data" instead of rasterization.

Next, details of rasterization, hash division of labor, and execution processing based on a neural network model will be described as examples of data structuring.
(1) Rasterization In one embodiment, the rasterization process is initiated by a request from an API running on the API server 23 . The inputs for this rasterization process are prototypes of 3D graphics data, which in one embodiment include polygon information (world scale) such as triangle meshes, light sources in the scene (world scale), and camera information (world scale). .

The content of the rasterizing process in one embodiment of the present invention is to rasterize the mesh in the polygon information and write it into the buffer memory. Various methods in 3D computer graphics rendering methods can be adopted as the rasterization method.

In the rasterization process according to one embodiment of the present invention, processing is performed so as to retain texture UV, mesh classification ID, luminosity of irradiated light, and the like. On the other hand, it is not necessary to retain color information in the rasterization process according to one embodiment of the present invention.

The outputs of the rasterization process in one embodiment of the present invention are UV (per pixel), classification information (per pixel), and luminosity information (per pixel).
(2) Hash Division In one embodiment, the inputs in the hash division are UV (per pixel), classification information (per pixel), and circuit weight arrays on a global scale.

The content of the hash division of labor in one embodiment of the present invention is to divide the area in the screen frame into tiles and to entrust the processing of each divided tile to the instruction scheduler of the GPU. In one embodiment of the present invention, there are no intra-tile data dependencies, so multiple SMs can be processed in parallel.

The contents of the tile pixels are then converted to hash values. The transformed hash values are employed as weight array keys in the neural network, and these values are written to on-chip memory.

The output from the hash division of labor in one embodiment of the present invention is the result (estimated value) estimated by the weight of the neural network per tile in the screen frame (hereinafter also referred to as "weight information").

In addition, although the present invention is not limited to this, in one embodiment, a GPU is used to perform hash division of labor. SM) and on-chip memory that the sync groups can share. In this case, the hash division assigns the sync group for each tile in the screen frame and loads the neural network weights onto the sharable on-chip memory. The per-tile neural network is small enough to be loaded entirely into the sharable on-chip memory. Since there is no need for weights outside the tile, no synchronization outside the sync group is needed, and multiple pixels using the same neural network can be processed simultaneously. Therefore, the efficiency can be further improved.

Furthermore, although the invention is not so limited, in one embodiment, utilizing a GPU to implement hash division of labor involves multiple partial processors (synchronization groups, or SM) and registers can be employed. In this case, the hash division of labor assigns the synchronization group to each tile in the screen frame and stores the output of the intermediate layer of the neural network in the above register, so that each layer can be used as a RAM or cache on the GPU. Processing for each layer can be executed without synchronizing with output to "slow memory", and processing can be made more efficient.
(3) Neural Network Execution Process In one embodiment, the inputs in the neural network execution process are UV (per pixel), luminous intensity information (per pixel), and weight information (per tile in the screen frame).

The content of the execution processing of the neural network in one embodiment of the present invention is to read the weight information written on the on-chip memory and execute the calculation up to the output nerve of the neural network. Since the calculation is performed for each tile in the screen frame, the input domain is not widened and the number of neural channels can be reduced. Therefore, there is an advantage that it is not necessary to send intermediate results of calculations to a cache memory or the like each time.

Note that nerve-to-nerve data dependencies in the same tile are not a big problem since the computations are executed by the same instruction scheduler for the same tile.

The output of the neural network execution process in one embodiment of the present invention is RGBA data (per pixel). This RGBA data is handled as screen output data.
(Training process and pretreatment process)
Before executing the processing shown in FIG. 5, the embodiment of the present invention also has a neural network training step. One of the features of one embodiment of the present invention is that various optimizations are performed as preprocessing for this training process. These pretreatment steps will be described later with reference to FIGS.

FIG. 6 shows a flow chart explaining the details of the operation of the image processing system etc. according to one embodiment of the present invention. The operation flow shown in FIG. 6, like the operation flow shown in FIG. 5, shows from rasterization processing to color information output for each pixel, but is more specific than the flow shown in FIG. is explained in Further, descriptions of operations that overlap with the operations described in FIG. 5 are omitted as appropriate. In other words, the operation flow shown in FIG. 6 can employ the contents described with reference to FIG. 5, but includes variations not shown in FIG.

When the process starts in step S601 of FIG. 6, the process proceeds to step S602, where rasterization processing is performed on mesh information, light source information, and camera information (all of which are on a global scale) for each screen frame.

In one embodiment, this rasterization process can employ many conventional rasterization techniques. The intermediate state of each pixel is saved in a buffer such as a G-Buffer. The output contents of rasterization are, for example, the two-dimensional position (UV) of the surface, the ID of the surface type (classification ID), and the amount of each type of light (luminous intensity).

In step S603, division processing is performed on UV, classification ID, and luminous intensity information (all of which are on a global scale) for each pixel.

In step S605, hash calculation processing is performed on the UV and classification ID (both of which are on a global scale) for each pixel. In this step, UV and classification ID for each pixel are taken up, but the present invention is not limited to this, and hash calculation processing may be performed on luminous intensity information (world scale).

For the hash calculation here, any hash formula can be adopted, but importantly, if the tile-sized surface areas have sufficiently similar properties (e.g. UV), the same The feature of obtaining a hash value is maintained. In one embodiment of the present invention, a bitfield approach is employed. Also, U and V are individually managed in sections. In one embodiment, one bit represents each section.

Here, if the tile contains any pixel in the section, the corresponding bit is set to 1, otherwise the bit remains 0. It is desirable that the pixel information be read into a register and held in a register or the like.

In step S606, the neural network weights for each tile are generated (tile scale) by applying the value retrieved from the lookup table to the hash value (tile scale) for each tile calculated in the previous step.

Here, since lookup table search is a memory read, all memory reads are a factor in degrading performance. Therefore, at some point the neural network weights need to be loaded into the local on-chip memory of the GPU's instruction scheduler (sometimes multiple instruction schedulers are combined), which may be performed in this step. . Since we have already computed a hash value that fully represents the current tile's content, we can use that hash value to select a neural network that has been rigorously trained to handle this kind of content.

Various methods of obtaining data based on hash values are conceivable, but in one embodiment of the present invention, the hash generated in this flow can be used as a key to the off-chip memory array.

In step S607, the weights of the neural network generated in the previous step and the UV and luminous intensity information (both on a global scale) for each tile processed in step S603 are input, and arithmetic processing based on the neural network model is performed. done.

Here, in one embodiment of the present invention, the logic that executes the neural network includes a processing system that takes UV, classification ID, and luminous intensity as inputs and pixel color as output. It is also possible to correctly combine the outputs of multiple neural networks at low cost by having the neural network output light instead of color. Also, various optimizations can be applied to the logic here.

In step S608, color information is output (pixel scale).

Then, the process proceeds to step 609 and ends the process as this flow.

Next, detailed operations of the image processing system or image processing program according to one embodiment of the present invention will be described with reference to FIGS. 7 to 12. FIG. The description here partially overlaps with what was described with reference to FIGS. 5-6, but includes more detailed descriptions and variations on hashing.

As already described, the characteristic operation of the present invention can be mainly performed by the information processing server group 11, but at least a part of it can also be performed by an information processing apparatus or the like.

7 and 8A show examples of data structures that are prerequisites for the operation of the image processing system and the like according to one embodiment of the present invention.

Also, in the screen frame 70 shown in FIG. 7, the tile information is projected into UV space. In UV space, a tile is represented by a location and shape that reflects what it is, rather than what it looks like. Therefore, by approximating this shape with two adjacent bit-fields, it can be associated with a well-specialized neural network.

FIG. 7 shows the UV space with the U hash sequence (00000111) on the horizontal axis and the V hash sequence (00011110) on the vertical axis. It can be seen that 71 appears as a tile 81 in the screen frame 80 (a total of 64 tiles, 8 in length and width) shown in FIG. 8A. Note that in one embodiment, each tile consists of 16×16 pixels.

Also, keeping in mind the projection of tiles into UV space described with reference to FIGS. 7 and 8A, in one embodiment of the present invention, the hash function defined in FIG. 8B is used. The present invention is not limited to this, and any hash function may be used as long as specific conditions ((A) and (B) below, and preferably (C) below) are satisfied. do not have.
(A) Tiles with the same hash value constitute a narrower input domain than all types of tiles taken as a whole.

In one embodiment of the present invention, the narrower the input domain, the better the results. In practice, this input domain is narrowly configured so that the speed can be sufficiently increased to the extent that processing can be performed substantially in real time.
(B) Calculation of hash values should be fast (computer hardware specifications at the time of filing this application fully satisfy this requirement).
(C) Only target tiles are used as inputs in hash division, although the invention is not so limited.

9 to 12 show flowcharts for explaining detailed operations of the image processing system and the like according to one embodiment of the present invention. More specifically, FIG. 9 shows an overview of the hashing flow based on the assumptions explained with reference to FIGS. 7 and 8A. FIG. 9 also describes in more detail assuming the flow shown in FIG. As a premise of the operation flow shown in FIG. 9, it is assumed that the scene data handled by the image processing system according to the embodiment of the present invention is rasterized on the frame buffer.

10 to 12 show an example processing flow for neural network training. Various preprocessing is performed from the viewpoint of optimization toward the training shown in step S1210 in FIG.

FIG. 10 shows the stage of generating a binary file of training data, showing the flow of dividing the pixel buffer into tiles, assigning hash values to each tile, and adding hash values to the hash map. ing. Also, FIG. 11 shows a flow including output of content to a binary file, and FIG. 12 shows a flow including mapping of the binary file content to memory space.

When the process starts in step S901 of FIG. 9, the process advances to step S902 to read data including UV, mesh ID, and illuminance information from the screen frame buffer.

In step S903, the data on the screen frame buffer is divided into 16x16 pixel tiles.

In step S905, a hash value is calculated using all UVs in the tile. Also, a classification ID may be used as needed.

In step S906, the neural network weights are read from the lookup table using the hash value calculated in the previous step as a key.

In step S908, the weight read in step S906 and the UV and luminosity information for each tile processed in step S903 are input, and neural network interface data is output. This output data is RGB data in one embodiment.

Then, the process proceeds to step 909 and ends the process as this flow.

(Training pretreatment process)
The flows shown in FIGS. 10 to 12 show preprocessing and training flows for neural network training for executing the processing described with reference to FIGS. 6 and 9. FIG.

In the process of learning the neural network for each tile, the pixel values in the screen frame are hashed to calculate the hash value, only the training data of the tile corresponding to this hash value is used, and the weight of the neural network dedicated to this hash value can be learned.

The avatar generator in this flow is an application that artificially generates a human face mesh. In one embodiment of the present invention, the purpose is to collect face image samples by CG generation rather than collecting them from live-action video.

Also, the mesh is rendered by ray tracing, and hash_map.txt stores a list of hash values that appear in the training data.

In one embodiment, although the invention is not so limited, fbx files are used to provide interoperability between digital content creation applications (and so on). An fbx file is an open framework for 3D data transfer.

Then, when the process starts in step S1001 of FIG. 10, the process advances to step S1002, and the output file of the avatar generator is input. Next, proceeding to S1003, all fbx files to be processed stored in the memory are read and listed.

In step S1004, it is determined whether or not any fbx file to be processed remains in the read fbx file list.

In step S1005, the fbx file to be processed in this routine is read from the list of fbx files, and in step S1006, the mesh is rendered in the pixel buffer.

Next, proceeding to S1007, the pixel buffer is divided into tiles and a hash value is assigned to each tile.

In step S1008, it is determined whether or not the hash value exists in the hash map on the current memory. , the process proceeds to step S1009.

In step S1009, the hash value determined in the previous step is added to the hash map in memory.

Also, in step S1010, since all the fbx files to be processed have been processed, the hash values in the hash map on the memory are output to the hash map file (hash_map.txt).

Then, the process proceeds to step S1011, and the process as this flow ends.

FIG. 11 shows the stage of generating a binary file of teacher data. As one embodiment following the flow shown in FIG. 10, fbx files are used that are used to provide interoperability between digital content creation applications.

In this flow, the tile data must be sorted by hash values, but it is difficult to store all of them in a memory such as RAM at the same time. Therefore, in one embodiment of the present invention, the binary file is written out incrementally. This measure can have significant speed benefits, depending on the operating system and SSD configuration.

Also, the inputs (U, V, luminosity, classification ID) and outputs (R, G, B, α) per pixel can be adopted as the dataset of the training data tile in one embodiment of the present invention. Although the present invention is not limited to this, in consideration of RAM problems, it is written like an AOS (Array of structs). A later training step requires a SOA (Structure of arrays). Different hash values are stored in different binary files.

In one embodiment, although the invention is not so limited, hash_map.txt only contains the tile information for the contained hash values. Rendering is performed by a graphics API for real time. Then, when the process starts in step S1101 of FIG. 11, the process proceeds to step S1102, and the output file of the avatar generator is input. Next, proceeding to S1103, all fbx files to be processed stored in the memory are read and listed.

In step S1104, the hash map file (hash_map.txt) to which the hash value was output in step S1010 of FIG. 10 is read.

Next, in step S1105, the hash map on the memory is filled with hash values from the hash map file (hash_map.txt) in each array.

In step S1106, it is determined whether or not fbx files to be processed remain in the fbx file list read out in step S1104.

In step S1107, the fbx file to be processed in this routine is read from the list of fbx files, and in step S1108, the mesh is rendered in the pixel buffer.

In step S1109, the png file, which in one embodiment is the target file, is read.

Next, proceeding to step S1110, the pixel buffer is divided into tiles and a hash value is assigned to each tile. In step S1111, each tile is added to an array associated with its hash value.

Next, in step S1112, the contents of the hashmap array are output to the appropriate binary file, and in step S1113, the contents are cleared from the array in the hashmap.

In step S1114, the contents of the hashmap array are output to the appropriate binary file.

Then, the process advances to step S1115, and the process as this flow ends.

FIG. 12 shows the stage of actually training a neural network for one hash value, and shows the process flow of mapping the binary file contents to the memory space as an embodiment following the flow shown in FIG. there is

In this flow, the data of the binary file can be used as it is by adjusting the stride of the array. Therefore, unlike the general case, there is no need to convert the loaded training data into a format suitable for training input and output (for example, converting the image format from int8 to float32, modifying the content, etc.). etc.).

The final step in FIG. 12 is the training of the neural network (for this training itself, a conventional method can be adopted, so detailed explanation is omitted except for matters specific to the present invention).

In one embodiment of the present invention, the neural network training framework (PyTorch) can be utilized via a Python language GUI. As an aside, the Python language is a computationally slow language, so it is preferable not to actually read the data. In one embodiment of the present invention, the operating system and CPU configuration allows file contents to be specified directly through pointers in virtual memory. Also, if the pointer pointing to the variable in the file and the stride of the array in the file are sent to the GUI of PyTorch in the Numpy array format, the teacher data can be transferred quickly to the GPU outside PCIe. Numpy is another Python language library.

In one embodiment, although the invention is not so limited, the neural network architecture is based on Gao Huang et al. ) can be implemented based on

Also, in the flow shown in FIG. 12, a 1×1 convolution kernel can be used in place of the 3×3 convolution kernel.

When the process starts in step S1201 of FIG. 12, the process advances to step S1202, and the hash value generated up to the flow shown in FIG. 11 is input. Next, the flow advances to step S1203 to secure file_list={}, which is a list of binary files.

In step S1204, it is determined whether or not any binary file to be processed remains in the list of reserved binary files.

Next, proceeding to step S1206,
(binary_file.name.u &hash.u)>0
If Yes, the process proceeds to step S1207, and if No, the process returns to step S1204.

Next, the process proceeds to step S1207,
(binary_file.name.v & hash.v) > 0
If Yes, the process proceeds to step S1208, and if No, the process returns to step S1204.

In step S1208, the binary file is added to file_list.

In step S1209, once for each file in file_list, the contents of the file are mapped directly into the pytorch tensor virtual memory space.

In step S1210, a neural network is trained on this hash.

Then, the process advances to step S1211, and the process as this flow ends.

Note that in one embodiment of the present invention, the training data of a tile corresponding to a hash value can include training data of a plurality of other tiles that are close to the input area, depending on the hash value. As a result, it is possible to learn the tile boundary portion so as to obtain a more natural result.

Binary files are partitioned by hash values that specify input areas. Training of the neural network is performed using its own hash value or a hash value that approximates its own hash value. Since the hash value is originally a bit field representing the area of the UV space, it is possible to estimate the similarity between values by bitwise AND operation (logical product).

FIG. 13 shows the operating principle by adopting a 1×1 convolution kernel among specific examples of the operation of the image processing system according to the embodiment of the present invention.

Calculation processing of convolution kernels in an image processing system according to an embodiment of the present invention will be described below in comparison with an example of calculation processing of convolution kernels in a conventional image processing system (FIGS. 17 and 18).

First, in the 1×1 convolution kernel according to one embodiment of the present invention shown in FIG. It is assumed that information (1) is embedded in 1311 . It is also assumed that information (0) is embedded in the convolution kernel 1320 shown in FIG.

Then, the output on the output layer 1330 by the operation of the convolution kernel target region 1311 and the convolution kernel 1320 is obtained as follows by the product (and sum) of each element in the convolution kernel target region 1311 and the convolution kernel 1320 .

(1 x 0) = 0
Although the present invention is not limited to this, in an image processing system or the like according to an embodiment of the present invention, by adopting the 1×1 convolution kernel as described above, the data dependence problem can be further resolved. Further, in other embodiments of the present invention, the same effect can be achieved by a convolution kernel with a simple configuration other than the 1×1 convolution kernel.

Next, for comparison with the 1×1 convolution kernel, after explaining the calculation cost of the conventional convolution kernel, the hash division of labor, which is another feature of one embodiment of the present invention, will be explained.
(conventional convolutional neural network)
The conventional method of designing a convolutional neural network has a problem that the utilization efficiency of hardware decreases when it is implemented in the latest GPU at the time of filing of the present application. On the other hand, in the special case of real-time rendering, as described above, these inefficiencies can be improved by using unusual and advantageous data-dependent characteristics. A specific example of "hash division of labor" will be described in more detail with reference to FIGS. 14 to 16).

17 and 18 show examples of calculation processing of convolution kernels in a conventional image processing system or the like. More specifically, FIG. 17 shows the calculations generated by the convolution kernel, and FIG. 18 shows how the calculations shown in FIG. 17 change with the number of input channels. It is shown.

A large number of channels are prepared in order to give expressiveness to the neural network. An example is 512 channels (or more). In general, it is necessary to increase the number of channels in order to improve image quality and expressive power. However, in each pixel, most of the channels are often irrelevant, which is inefficient. In other words, inefficiency due to the number of channels becomes a scalability bottleneck.

FIG. 17 shows at least part of the input image in the input image area 1710, and the convolution kernel target area 1711 in the input image area 1710 includes (0, 0, 0, 0, 1 , 1, 0, 1, 2) are packed. Also, the convolution kernel 1720 shown in FIG. 17 is packed with information (4, 0, 0, 0, 0, 0, 0, 0, -4) in order from the upper left.

Then, the output on the output layer 1730 by the operation of the convolution kernel target region 1711 and the convolution kernel 1720 is obtained by the product and sum of each element in the convolution kernel target region 1711 and the convolution kernel 1720 as follows.

(0×4)+(0×0)+(0×0)+
(0×0)+(1×0)+(1×0)+
(0×0)+(1×0)+(2×(−4))
=-8
FIG. 18 shows how the calculation shown in FIG. 17 changes according to the number of input channels, and is an example of calculation when the number of channels is three. As shown in FIG. 18, the output on the output layer 1830 by operating the convolution kernel target regions 1811a-1811c in the input image regions 1810a-1810c and the convolution kernels 1820a-1820c is obtained as follows.

(0×1)+(0×0)+(0×4)+
(0×1)+(0×0)+(0×0)+
(1×0)+(0×4)+(0×0)+
(1×0)+(0×0)+(0×0)+
(1×0)+(1×1)+(1×0)+
(2 x 1) + (1 x 1) + (1 x 0) +
(0x0) + (0x0) + (0x0) +
(0×0)+(1×0)+(1×0)+
(0×1)+(0×0)+(2×(−4))
=-4
Thus, the computational cost of the convolutional kernel shown in FIGS. 17 and 18 is calculated by multiplying the number of input pixels by the number of output channels.

Considering this calculation cost in terms of the utilization rate of the GPU, the following problems (A) to (C) can be considered.
(A) As the number of layers increases and each layer is composed of multiple shader calls, overhead increases due to starting and terminating a large number of small shaders.
(B) A convolutional neural network with a moderate number of channels may suffer from a shortage of operation instructions for memory instructions. More channels means better balance for certain operations, but ultimately longer runtimes. Neural networks can be said to be efficient from the point of view of hardware utilization, but from the point of view of real-time performance, conventional methods are inadequate. Therefore, even as the number of channels increases, some operations remain imbalanced.
(C) Operations within a layer and operations between layers are implemented as shader calls that depend on data, so the intermediate output is written to off-chip memory before executing the next operation in another shader call. There is redundancy to be improved in that it needs to be reread as input.

Next, considering the clues to solving the above problems (A) to (C), first of all, regarding (A), if we consider why the neural network is subdivided and each shader is called, it is because is due to a combination of the data dependencies of , and the data dependencies between elements/pixels. If there are no dependencies between operations, it suggests that layers can be executed in parallel. It also suggests that output pixels can be executed in parallel if there are no dependencies between elements. Either way, it means that the neural network can be solved with a single shader call.

Regarding (B), it is suggested that the problem is not that the neural network as a whole lacks computational instructions, but that the number of computational instructions included in each shader call is too small. That is, each thread has a semi-fixed wake-up time while waiting for its first memory read to complete, resulting in a high percentage of idle clock cycles and low GPU utilization.

Regarding (C), the problem is that there are a large number of small shader calls. Repeatedly rewriting and rereading intermediate outputs causes a lot of memory traffic and latency. In order to reduce the number of shader calls, it is necessary to solve the data dependency problem.

Based on the above considerations and observations, it can be seen that in the world of computer graphics, if the data dependency between elements can be resolved in advance, the problems pointed out so far can be resolved.

The solution is the adoption of the 1×1 convolution kernel in one embodiment of the present invention described with reference to FIG. As for the convolution kernel of (1), depending on the situation, the effect of the present invention can also be achieved by convolution kernels with other simple configurations). A specific example of hash division of labor in one embodiment of the present invention will be described below with reference to the drawings.

14 to 16 show the operation principle by adopting the hash division of labor among the specific examples of the operation of the image processing system etc. according to the embodiment of the present invention.

In implementing hash division of labor according to one embodiment of the present invention, tiles in a screen frame are assigned to individual GPU schedulers. A dedicated neural network can then be run locally on on-chip memory by having the GPU calculate a hash value based on the contents of each tile. This makes it possible to specialize in a smaller neural network, and keeps intermediate data and data dependencies within the same scheduler, thus theoretically improving processing efficiency.

An example of the processing flow of the hash division of labor has already been described with reference to FIG.

The hash division illustrated in FIGS. 14-16 illustrates the process of computing a hash value from the pixel contents of a tile, and in one embodiment employs the following pseudocode.

for (pixel in tile) {hash_u|=(1<<pixel.u)}
for (pixel in tile) {hash_v|=(1<<pixel.v)}
The texture UV of the 3D model is ideal for applying the hash division of labor in one embodiment of the present invention. Further, desirably, further scaling such as eight times is employed.

With such a configuration, weights can be efficiently loaded with the relationship of "arrangement key=hash value".

The hash division of labor in FIG. 14 shows how the value 0.8 in the pixel 1411 of the input image area 1410 is output based on the following formula based on the above premise.

output |=
1<<((static_cast<int>(0.8f*8.0f))%8)
Also, in the hash division of labor in FIG. 15, a state in which the value 0.2 in the pixel 1511 of the input image area 1510 is output based on the following formula is shown.

output |=
1<<((static_cast<int>(0.2f*8.0f))%8)
Also, in the hash division of labor in FIG. 16, a state in which the value 1.0 in the pixel 1611 of the input image area 1610 is output based on the following formula is shown.

output |=
1<<((static_cast<int>(1.0f*8.0f))%8)
Thereafter, similarly, the hash division of labor is successively performed on the values in the pixels within the input image area.
[Theoretical effect]
The theoretical effects of one embodiment of the present invention derived from the examples described above are as follows.
(Theoretical effect 1: reduction in the number of neurons)
Fewer neurons provide sufficient performance in constrained input regions. This is reflected notably in terms of performance. Furthermore, theoretically, a small enough neural network that we are dealing with can live in the registers of a single instruction scheduler (or cluster of schedulers) for its lifetime, so this also has the effect of It can be expected (see theoretical effects 2-4 below).
(Theoretical effect 2: reduction in the number of times shaders are used)
All neural networks can be solved simultaneously in a single shader call, thus avoiding a myriad of platform-induced overheads.
(Theoretical effect 3: ALU usage rate improvement)
Since both inputs and outputs are registered, the ALU is not latency bound by memory dependencies. Allowing the ALU to run at its own pace leads to improved performance in all respects.
(Theoretical effect 4: Reduction of memory traffic)
Memory traffic is reduced because there is no need to write intermediate values to and read back from external memory. This reduces the burden on the memory subsystem and improves performance.
(Theoretical effect 5: Improved scalability of large-capacity systems)
As the breadth and quality of rendered output produced by a neural network increases, so does the size of the neural network, and as the size of the neural network increases, the performance decreases. Hash division in one embodiment of the present invention can circumvent this problem through structured subdomain decoupling.
(Theoretical Effect 6: Streaming Friendly Assets)
When deployed as an art asset, the neural network needs to be streamed from SSD in real time. So by making each neural network small and labeling it in a structured way, it can be moved in and out of memory as needed based on the current screen content.
(Theoretical effect 7: Improved controllability and stability)
Limited memory dependencies make it easier to infer the inputs used to arrive at a particular conclusion. This reduces the risk that the neural network will fail under sufficiently different circumstances.
(Theoretical Effect 8: Raster Resistant Zooming)
The logic required for low-resolution output can be very different from the logic required for high-resolution output, and without the application of the present invention raster artifacts occur at wider zoom levels in the input domain. Therefore, by adopting the hash algorithm according to an embodiment of the present invention, it is advantageous that different neural networks are learned according to the zoom level, and this process is performed automatically.
(Theoretical Effect 9: Render Engine Compatibility)
Hash division according to an embodiment of the present invention uses the same 3D models, animations and lighting as other engines and can also be added as shaders for some (or all) scene objects.
(Theoretical effect 10: Neural network image quality)
GAN-based neural networks offline have already achieved higher quality output than current real-time rendering systems. Doing this quickly and consistently makes it an attractive option for many kinds of real-time engines and products.
(Theoretical effect 11: Asset formation)
Training neural networks is in many ways more forgiving than static 3D scans of assets and PBR constants used in shaders. It can provide realistic options for maintaining a consistent photorealistic quality level throughout the scene.

The embodiments of the image processing system and the image processing program have been described above based on specific examples. It can also be implemented as a storage medium (for example, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a CD-RW, a magnetic tape, a hard disk, a memory card) or the like.

Moreover, the implementation form of the program is not limited to an application program such as an object code compiled by a compiler or a program code executed by an interpreter. good.

Furthermore, the program does not necessarily have to be executed entirely by the CPU on the control board. ) can also be configured to implement a part or all of it.

For all elements described in this specification (including the claims, abstract, and drawings) and/or for any method or process step disclosed, these features are mutually exclusive. Any combination can be used except for the combination of

Moreover, each of the features disclosed in this specification (including the claims, abstract, and drawings) may serve the same, equivalent, or similar purpose unless expressly contradicted Alternate features may be substituted. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic series of identical or equivalent features.

Furthermore, the invention is not limited to any specific configuration of the embodiments described above. The present invention resides in any novel feature or combination thereof, or any novel method or process step, or method described in this specification (including claims, abstract, and drawings). can be extended to combinations of

10 image processing system 11 information processing server group 111 real-time cluster 112 load balancers 12, 13 PC (one form of information processing apparatus)
14 Mobile phones (a form of information processing equipment)
15, 15a to 15c tablet terminal (a form of information processing device)
19 Public line (dedicated line, Internet, etc.)
23 API server

Claims (14)

a conversion step for converting the mesh and lighting information in the screen frame into a structured input format adaptable to the neural network;
performing a hash division of labor for properly using the neural network trained for each tile,
The hash division of labor includes:
hashing pixel values in the screen frame to calculate a hash value;
using the hash value as a key to select and load a weight in the neural network for the corresponding tile from a lookup table;
An image processing method by a computer, wherein the output by the hash division of labor is an estimated value estimated by the weight of the neural network corresponding to the tile in the screen frame. The computer has a plurality of partial processors (hereinafter referred to as "synchronization groups") that can be efficiently synchronized with each other in processing units, and an on-chip memory that can be shared by the synchronization groups,
2. The method of claim 1, wherein the hash division assigns the synchronization groups and loads the neural network weights onto the on-chip memory for each tile in the screen frame. The computer has a plurality of sub-processors (hereinafter referred to as "synchronization groups") that can be efficiently synchronized with each other in the processing units, and registers,
2. The method of claim 1, wherein the hash division assigns the synchronization group for each tile in the screen frame and stores the outputs of intermediate layers of the neural network in the registers. In the step of learning the neural network for each tile, the pixel values in the screen frame are hashed to calculate a hash value, and only the training data of the tile corresponding to the hash value is used to create a neural network dedicated to the hash value. 2. The method of claim 1, wherein the weights of are learned. 5. The method of claim 4, wherein the training data of the tile corresponding to the hash value includes training data of other tiles that are close as input regions according to the hash value. An image processing system comprising a CPU and a GPU and performing rendering processing using a neural network,
a conversion unit for converting mesh information and lighting information in a screen frame into a structured input format adaptable to a neural network;
a processing unit that performs hash division of labor for properly using the neural network learned for each tile,
The hash division of labor includes:
a calculation unit for hashing pixel values in the screen frame to calculate a hash value;
a reading unit that selects and reads weights in the neural network for corresponding tiles from a lookup table using the hash value as a key;
The image processing system, wherein the output from the hash division of labor is an estimated value estimated by the weight of the neural network corresponding to the tile in the screen frame. The image processing system comprises a plurality of sub-processors (hereinafter referred to as "synchronization groups") that can be efficiently synchronized with each other in processing units, and an on-chip memory that can be shared by the synchronization groups,
7. The system of claim 6, wherein the hash division assigns the synchronization groups and loads the neural network weights onto the on-chip memory for each tile in the screen frame. The image processing system comprises a plurality of sub-processors (hereinafter referred to as "synchronization groups") that can be efficiently synchronized with each other in the processing units, and registers,
7. The system of claim 6, wherein the hash division assigns the synchronization group for each tile in the screen frame and stores the outputs of intermediate layers of the neural network in the registers. In the step of learning the neural network for each tile, the pixel values in the screen frame are hashed to calculate a hash value, and only the training data of the tile corresponding to the hash value is used to create a neural network dedicated to the hash value. 7. The system of claim 6, wherein the weights of are learned. A program that is executed on an image processing system that includes a CPU and a GPU and performs rendering processing using a neural network, wherein when executed on the system,
to the CPU or the GPU,
converting the mesh and lighting information in the screen frame into a structured input format adaptable to the neural network;
performing a hash division of labor for properly using the neural network trained for each tile,
The hash division of labor includes:
hashing pixel values in the screen frame to calculate a hash value;
using the hash value as a key to select and load a weight in the neural network for the corresponding tile from a lookup table,
The program, wherein the output from the hash division of labor is an estimated value estimated by the weights of the neural network corresponding to the tiles in the screen frame. The system includes a plurality of partial processors (hereinafter referred to as "synchronization groups") that can be effectively synchronized with each other in processing units, and an on-chip memory that can be shared by the synchronization groups;
11. The program of claim 10, wherein the hash division includes assigning the synchronization group and loading the neural network weights onto the on-chip memory for each tile in the screen frame. The system comprises a plurality of partial processors (hereinafter referred to as "synchronization groups") that can be efficiently synchronized with each other in the processing units, and registers,
11. The program product of claim 10, wherein the hash division includes assigning the synchronization group for each tile in the screen frame and storing the outputs of intermediate layers of the neural network in the registers. In the step of learning the neural network for each tile, the pixel values in the screen frame are hashed to calculate a hash value, and only the training data of the tile corresponding to the hash value is used to create a neural network dedicated to the hash value. 11. The program according to claim 10, which learns the weight of . 14. The program according to claim 13, wherein said teacher data of said tile corresponding to said hash value includes teacher data of a plurality of other tiles that are close as an input area according to said hash value.

Images